本地scp已正常安装运行，执行远程拷贝时却提示'command not found'的问题排查与求助

我最近碰到了一个非常奇怪的scp问题，明明本地的scp看起来一切正常，但执行远程拷贝时却总是弹出bash: scp: command not found的错误，下面是我的整个排查过程，想求助大家帮我看看问题到底出在哪：

最开始我执行了这条scp命令：

$ scp -i <key> -P <port> <user>@<host>:<path> .
bash: scp: command not found

我第一反应是检查scp是否存在，结果显示它就在标准路径里：

$ which scp
/usr/bin/scp

不死心的我直接用绝对路径执行，结果还是一样的错误：

/usr/bin/scp -i <key> -P <port> <user>@<host>:<path> .
bash: scp: command not found

更奇怪的是，单独执行scp或者绝对路径的scp，都能正常显示用法说明，说明程序本身完全没问题：

$ scp
usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config]
           [-i identity_file] [-J destination] [-l limit]
           [-o ssh_option] [-P port] [-S program] source ... target
$ /usr/bin/scp
usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config]
           [-i identity_file] [-J destination] [-l limit]
           [-o ssh_option] [-P port] [-S program] source ... target

目前我能确定的几点：

• scp已经成功安装
• 它在系统的$PATH路径中
• 程序本身可以正常运行

我还尝试了卸载重装相关依赖包，结果还是没解决：

sudo apt-get remove openssh-client openssl
sudo apt-get install openssh-client openssl

之前scp在我机器上一直好好的，突然就出了这个问题，实在摸不着头脑。另外补充一句，远程服务器上也已经安装了scp。

补充排查1：加-v参数调试 #

按照评论区的建议，我给scp命令加了-v参数调试，输出如下：

$ /usr/bin/scp -v -i <key> -P <port> <user>@<host>:<path> .
Executing: program /usr/bin/ssh host <host>, user <user>, command scp -v -f <path>
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data <path>
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to <host> [<address>] port <port>.
debug1: Connection established.
debug1: identity file <path> type 0
debug1: identity file <path> type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Raspbian-10+deb10u4
debug1: compat_banner: match: OpenSSH_7.9p1 Raspbian-10+deb10u4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to <host>:<port> as '<user>'
debug1: load_hostkeys: fopen <path>/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:dBz2B5K8e3bkRhl7S995urLdbvHm34eYlwCUwFyVWNs
debug1: load_hostkeys: fopen <path>/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[<host>]:<port>' is known and matches the ED25519 host key.
debug1: Found key in <path>/.ssh/known_hosts:54
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: <path> RSA SHA256:8c7N6ZX/gpWvY2qQPG4dkTy3LDY51UWaQ2NoaMLhXwE explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: <path> RSA SHA256:8c7N6ZX/gpWvY2qQPG4dkTy3LDY51UWaQ2NoaMLhXwE explicit
debug1: Server accepts key: <path> RSA SHA256:8c7N6ZX/gpWvY2qQPG4dkTy3LDY51UWaQ2NoaMLhXwE explicit
Authenticated to <host> ([<address>]:<port>) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching <path>/.ssh/known_hosts for [<host>]:<port> / (none)
debug1: client_input_hostkeys: searching <path>/.ssh/known_hosts2 for [<host>]:<port> / (none)
debug1: client_input_hostkeys: hostkeys file <path>/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug1: Remote: <path>/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: <path>/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "C.UTF-8"
debug1: channel 0: setting env LC_CTYPE = "en_US.UTF-8"
debug1: Sending command: scp -v -f <path>
bash: scp: command not found
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3024, received 2840 bytes, in 0.1 seconds
Bytes per second: sent 28190.5, received 26475.2
debug1: Exit status 127

从输出里看到，在debug1: Sending command: scp -v -f <path>之后就报错了，这会不会是远程机器的问题？

补充排查2：测试远程执行基础命令 #

我试着通过ssh远程执行一些基础命令，结果发现更诡异的情况：

$ ssh -i <key> -p <port> <user>@<host> "cat <file>"
bash: cat: command not found

部分命令能正常执行，但很多基础命令都找不到：

$ ssh -i <key> -p <port> <user>@<host> "pwd"
/home/<user>
$ ssh -i <key> -p <port> <user>@<host> "echo foo"
foo
$ ssh -i <key> -p <port> <user>@<host> "ls"
bash: ls: command not found
$ ssh -i <key> -p <port> <user>@<host> "whoami"
bash: ls: command not found
$ ssh -i <key> -p <port> <user>@<host> "cd"
$

像pwd、echo能正常跑，但cat、ls、whoami这些都提示找不到命令，这完全不符合常理啊。

补充排查3：远程机器的PATH环境变量问题 #

我在远程机器上直接登录后执行echo $PATH，得到的结果是正常的：

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games

但通过ssh远程执行echo $PATH时，结果却变成了：

${PATH}:/usr/local/ssl/bin

我检查了远程服务器上的/etc/environment文件，发现内容是：

PATH="${PATH}:/usr/local/ssl/bin"

这应该就是问题的根源了吧？因为这个配置导致ssh非交互式登录时，PATH变量没有被正确解析，变成了${PATH}的字符串，而不是实际的系统路径，所以系统找不到那些在标准路径里的命令，包括scp。

备注：内容来源于stack exchange，提问作者nullromo