SSH远程执行Python脚本时K8s Pod日志未生成问题排查
2026-7-14
问题：远程执行K8s运维脚本无法生成日志文件
我是一名想转型DevOps的系统管理员，编写了kube_reload.py脚本用于检查Kubernetes Pod，当Pod挂起时记录其日志并删除该Pod，计划将监控脚本check_radio加入Cron自动处理挂起Pod。本地在Kube Master执行kube_reload.py时功能完全正常（可生成日志并删除Pod）；但从Nginx服务器通过SSH远程执行该脚本时，仅Pod被成功删除，却未生成对应的日志文件，且无任何报错信息。我使用Python 3.6，脚本及所在目录权限均为777，目前暂未掌握K8s API的使用方法，希望先解决当前问题，同时欢迎针对运维工作的优化建议。
相关脚本
kube_reload.py（Kube Master上的脚本）
#!/bin/python3 import sys import os import re import subprocess as sp from contextlib import redirect_stdout as r_stdout marker = False pod_nums = '' pod_nums_int = [] for_reload = [] get_pods = sp.check_output(['kubectl', 'get', 'pods']) get_pods = str(get_pods) get_pods = get_pods.split() pods = [] '''write garbage pod names in array''' for word in get_pods: if 'radio-hls' in word: pods.append(word) '''make usual pod names from garbage names''' for num in range(len(pods)): while marker == False: if pods[num][:5] != 'radio': pods[num] = pods[num][1:] if pods[num][:5] == 'radio': break '''Function that lists all pods''' def pod_list(): sp.run('clear') print('PODS LIST:\n') for num in range(len(pods)): print(num+1, '.\t', pods[num]) print(len(pods)+1, '.\t Reload all') '''Function for recursion in try-except statement in 'input_nums()' function''' def return_for_except(): pod_list() print('Error input. Try again\n\n') input_nums() '''Function that asks which pods you want to reload''' def input_nums(): pod_nums = str(input('Select which pods you want to reload (type all numbers with a space):\nExample: 1 2 3 10\n\n')) pod_nums = pod_nums.split() try: global pod_nums_int pod_nums_int = [eval(i) for i in pod_nums] except: return return_for_except() '''Function to write pod logs to a file''' def write_logs(): global for_reload if len(pods)+1 in pod_nums_int: all_reload = input('You selected "Reload all". To continue, press Enter\n') if all_reload == '': for i in range(len(pods)): with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log: sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log) print(f'{pods[i-1]} logged successfully') for_reload.append(i) else: print('Something went wrong') quit() else: for i in pod_nums_int: with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log: sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log) print(f'{pods[i-1]} logged successfully') for_reload.append(i) '''Function that reloads streams''' def reload_pods(for_reload, pods): for i in for_reload: sp.run(['kubectl', 'delete', 'pod', f'{pods[i-1]}']) print(f'{pods[i-1]}', 'deleted') '''Start''' '''manual (with no arguments)''' if len(sys.argv) == 1: pod_list() input_nums() write_logs() reload_pods(for_reload, pods) sp.run(['kubectl', 'get', 'pods']) print() sp.run(['ls', '-lah']) '''auto (from nginx srver)''' if len(sys.argv) > 1: for arg in sys.argv: if arg == './kube_reload.py': continue else: for pod in pods: if arg in pod: #write logs with open (f'{arg}.log', 'w') as log: sp.run(['kubectl', 'logs', f'{pod}'], stdout=log) #reload pods sp.run(['kubectl', 'delete', 'pod', f'{pod}']) else: continue
check_radio（Nginx服务器上的脚本）
#!/bin/python3 import requests as r import subprocess as sp import sys '''IN CASE OF ADDING ADDITIONAL STREAM ADD IT TO "streams" ''' streams = [ 'gold', 'tophits', 'worldchart', 'ukraine', 'rock', 'chill', 'rap', 'retromusic', 'elektro', 'sport', 'radionv', 'edyninov', 'hromadske' ] '''IF IF THERE IS NO NEED TO CHECK SOME STREAMS YOU CAN ADD IT TO "streams_not_to_check" ''' streams_not_to_check = [ 'radionv', 'edyninov', 'hromadske' ] streams_to_delete = [] #CLUSTER_API = 'https://host_ip:6443' #auth_header = 'Authorization: ' for stream in streams: check_stream = r.get(f'https://host.host/stream/{stream}/status/health').json() if check_stream['metadata'] == 'UNHEALTHY': streams_to_delete.append(stream) for stream in streams_not_to_check: if stream in streams_to_delete: streams_to_delete.remove(stream) print(streams_to_delete) if len(streams_to_delete) >= 1: for stream in streams_to_delete: sp.Popen(f'ssh developer@radio1 python3 ~/deekly/kube_reload.py {stream}', shell=True).communicate()
执行结果对比
1. Nginx服务器远程执行
[developer@radio-lb1 deekly]$ ./check_radio ['rap', 'rock'] pod "radio-hls-rap-f4b86bd77-jpmr4" deleted pod "radio-hls-rock-57fc8fcd64-m54k5" deleted [developer@radio-lb1 deekly]$
Kube Master上无日志文件生成
2. Kube Master本地执行
[developer@radio1 deekly]$ ./kube_reload.py rap rock pod "radio-hls-rap-f4b86bd77-4bvcs" deleted pod "radio-hls-rock-57fc8fcd64-g5j96" deleted [developer@radio1 deekly]$ ls -la total 32 drwxrwxrwx 2 developer developer 59 Aug 5 16:38 . drwx------ 14 developer developer 4096 Aug 4 20:26 .. -rwxrwxrwx 1 developer developer 2850 Aug 4 19:51 kube_reload.py -rw-rw-r-- 1 developer developer 8303 Aug 5 16:38 rap.log -rw-rw-r-- 1 developer developer 8345 Aug 5 16:38 rock.log
成功生成对应日志文件
问题原因及解决方法
核心原因
远程执行时，脚本的工作目录不是你预期的~/deekly/。SSH远程执行命令时，默认工作目录是用户的home目录（/home/developer），而不是脚本所在的~/deekly/。所以日志文件其实生成在了/home/developer/目录下，而不是~/deekly/里，你没找到而已。
解决方法
有两种简单的修复方式：
修改远程执行命令，先切换到脚本目录再执行
修改check_radio里的SSH命令：sp.Popen(f'ssh developer@radio1 "cd ~/deekly && python3 kube_reload.py {stream}"', shell=True).communicate()用双引号把命令包起来，先
cd到脚本所在目录再执行。修改
kube_reload.py，指定日志文件的绝对路径
在脚本开头定义日志目录的绝对路径，比如：LOG_DIR = '/home/developer/deekly/'然后在写日志的地方改成：
with open (f'{LOG_DIR}{arg}.log', 'w') as log:这样不管工作目录在哪，都会把日志写到指定的绝对路径下。
运维优化建议
- 避免使用
eval()：input_nums()里的eval(i)存在安全风险，换成int(i)即可，因为输入的是数字。 - 日志文件按时间命名：避免同名日志被覆盖，比如用
f"{arg}_{datetime.datetime.now().strftime('%Y%m%d_%H%M%S')}.log"来命名。 - 加入错误处理：当前脚本里
sp.run()没有处理异常，比如kubectl logs可能因为Pod不存在失败，应该添加try-except块捕获并记录错误。 - 用systemd定时器替代Cron（可选）：如果是Linux系统，systemd定时器比Cron更灵活，支持秒级调度，还能记录执行日志。
- 日志集中管理：把Pod日志和脚本执行日志收集到ELK或Promtail+Loki里，方便后续排查问题，不用登录服务器找日志。
- Pod健康检查交给K8s自身：可以给Pod配置
livenessProbe和readinessProbe，让K8s自动重启不健康的Pod，减少自定义脚本的维护成本。 - 学习Kubernetes API：长期来看，直接调用K8s API比执行
kubectl子进程更可靠、高效，也更容易捕获错误，是DevOps转型的必备技能。
内容的提问来源于stack exchange，提问作者Deekly
相关产品推荐
相关产品推荐
