在Amazon EMR运行自定义脚本遇文件不存在错误求助

我在AWS EMR中搭建JupyterHub环境，此前按照官方文档操作无异常。希望在集群部署阶段添加步骤批量创建用户，官方提供的示例Bash脚本可通过script-runner.jar成功运行。但替换为自定义的add_users_ERM.sh脚本（实际为Python脚本）后，运行时出现"error=2, No such file or directory"错误，错误日志及脚本内容如下：

错误日志 #

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/tez/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "/mnt/var/lib/hadoop/steps/s-23QLFU7JXPPM7/./add_users_ERM.sh" (in directory "."): error=2, No such file or directory
    at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:143)
    at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.io.IOException: Cannot run program "/mnt/var/lib/hadoop/steps/s-23QLFU7JXPPM7/./add_users_ERM_Linuxx.sh" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at com.amazon.elasticmapreduce.scriptrunner.ProcessRunner.exec(ProcessRunner.java:96)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 8 more

自定义脚本add_users_ERM.sh #

#!/opt/conda/bin/python

import os
import subprocess
import traceback
import sys

TOKEN="$(sudo docker exec jupyterhub /opt/conda/bin/jupyterhub token jovyan | tail -1)"

def users_from_text(file):
        
    #Now we have users and their team, we can create a user account and assign them to a team
    for user in file:
                username = user
 
                print(f"Adding {user}")
                cmd = ["sudo", "docker", "exec", "jupyterhub","useradd", "-m" ,"-s" ,"/bin/bash" ,"-N" ,username]
                p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                output, error = p.communicate()
                output = output.strip().decode("utf-8")
                error = error.decode("utf-8")
                if p.returncode != 0:
                    print(f"Error adding user: {error}")
                else:
                    print(F"{user} was added")

                cmd = ["sudo", "docker", "exec", "jupyterhub","bash", "-c" ,f"echo {username}:{username} | chpasswd"]
                p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                output, error = p.communicate()
                output = output.strip().decode("utf-8")
                error = error.decode("utf-8")
                if p.returncode != 0:
                    print(f"Error adding password: {error}")
                else:
                    print(F"{user} password was added")

                cmd = ["curl", "-XPOST", "--silent", "-k",f"https://$(hostname):9443/hub/api/users/{username}", "-H" ,f"Authorization: token {TOKEN}", "|", "jq"]
                p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                output, error = p.communicate()
                output = output.strip().decode("utf-8")
                error = error.decode("utf-8")
                if p.returncode != 0:
                    print(f"Error adding user to JH: {error}")
                else:
                    print(F"{user}  was added to JH")

                #To do: Convert api call to subprocess request
    return output
    

test_data = ["worker_1", "worker_2", "worker_3", "worker_4", "worker_5", "worker_6"]
txt_file = test_data  
print("Attempting add_user.sh script")
output = users_from_text(txt_file)

1. 脚本文件名不匹配 #

错误日志中明确提到找不到add_users_ERM_Linuxx.sh，但你实际使用的脚本是add_users_ERM.sh，文件名存在拼写差异（多了Linuxx后缀）。检查EMR步骤中指定的脚本文件名是否与实际上传的完全一致，包括大小写、拼写和后缀。

2. Python解释器路径有效性问题 #

脚本开头的shebang行#!/opt/conda/bin/python指定了Python解释器路径，需确认EMR主节点上该路径真实存在。可登录主节点执行ls /opt/conda/bin/python验证，若路径不存在，建议改为#!/usr/bin/env python3，该写法会自动查找系统环境中的Python3解释器，兼容性更强。

3. 脚本缺少可执行权限 #

确保脚本上传到EMR集群后拥有可执行权限。可在EMR步骤中添加前置命令chmod +x /mnt/var/lib/hadoop/steps/s-23QLFU7JXPPM7/add_users_ERM.sh，或在上传脚本时提前设置好权限（比如本地执行chmod +x add_users_ERM.sh再上传）。

4. Python脚本中混用Shell语法 #

你的脚本是Python文件，但直接使用了Shell语法，导致Python无法解析：

• 获取TOKEN的Shell命令：TOKEN="$(sudo docker exec ...)"是Shell语法，Python无法识别，需改用subprocess执行命令并提取结果：

  import subprocess
  # 执行命令获取token
  token_result = subprocess.run(
      ["sudo", "docker", "exec", "jupyterhub", "/opt/conda/bin/jupyterhub", "token", "jovyan"],
      capture_output=True,
      text=True
  )
  # 提取最后一行作为token
  TOKEN = token_result.stdout.strip().split('\n')[-1]
  

• 主机名获取：$(hostname)是Shell语法，Python中需导入socket模块获取主机名：

  import socket
  hostname = socket.gethostname()
  # 替换原URL中的$(hostname)
  api_url = f"https://{hostname}:9443/hub/api/users/{username}"
  

• curl命令中的管道| jq：subprocess.Popen无法直接处理管道，需通过Shell执行整个命令，将shell=True参数传入：

  cmd = f"curl -XPOST --silent -k {api_url} -H 'Authorization: token {TOKEN}' | jq"
  p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
  

内容的提问来源于stack exchange，提问作者Zizi96