sshd_config is the ssh daemon (or ssh server process) configuration file. As you’ve already stated, this is the file you’ll need to modify to change the server port.
ssh_config file is the ssh client configuration file. The client configuration file only has bearing on when you use the
ssh command to connect to another ssh host. So, in this case, you don’t need to modify it. It will be other client machines connecting to your server.