远程LB日志恢复脚本stdout为空问题排查求助

脚本预期逻辑 #

• 连接至LB -> 执行指定命令 -> 检查stdout是否非空
• 若stdout为空，更新网页为ALL_GOOD；若存在持久化会话，更新网页为NOT_GOOD并发送带脚本输出附件的邮件
• 更新statusEmail文件记录邮件发送日期，避免单日重复发送

当前问题现象 #

执行脚本后返回空结果[]，输出如下：

% ./lb-logs.py  
SSH connection established successfully.
[]
Command executed.

相关代码 #

主脚本 lb-logs.py #

#! /usr/bin/env python3

import logging
import os
from src.utils import run_command, get_results
from datetime import datetime

lbUser = "lb_user"
lbIp = "lb_ip"
lbPass = "lb_pass"
attachment = "p-sessions.txt"
date = datetime.now().strftime("%d/%m/%Y")
mailList = "mail_list"

logging.basicConfig(
    filename="logs.log",
    encoding="utf-8",
    level=logging.DEBUG,
    format="%(asctime)s:%(levelname)s:%(message)s",
)

result = get_results(lbIp, lbUser, lbPass)

status = 'UNKNOWN'

if result:
    run_command(
        "sudo sed -i 's/^<h1>.*/<h1>NOT_GOOD-TEST<\/h1>/' /var/www/html/index1.html"
    )
    status = "NOT_GOOD"
    logging.info(f"Status: {status}")
else:
    run_command(
        "sudo sed -i 's/^<h1>.*/<h1>ALL_GOOD-TEST<\/h1>/' /var/www/html/index1.html"
    )
    status = "ALL_GOOD"
    logging.info(f"Status: {status}")

if status == "NOT_GOOD":
    if os.path.exists(attachment):
        logging.info(f"File {attachment} exists.")
        print("File exists.")
    else:
        # Send email if date in statusEmail is different from today's date
        with open("src/statusEmail", "r") as statusEmailFileHandle:
            if date == statusEmailFileHandle.read():
                logging.info("Email already sent.")
            else:
                # Create file
                for line in result:
                    print(line)
                    with open(attachment, "a") as attachmentFileHandle:
                        attachmentFileHandle.write(line + "\n")
                logging.info(f"File {attachment} created.")

                result1 = run_command(
                    f"echo 'Generated from vm: ' | mailx -r 'vm' 'mail_relay' -s 'Persistent sessions found on LB - {date}' -t {mailList} -A {attachment}"
                )

                print(result1)

                with open("src/statusEmail", "w") as f:
                    f.write(f"NOT_GOOD: {date}")
                logging.info("Email sent.")

工具模块 utils.py #

import paramiko
import subprocess
import logging


def connect(lb_ip, lb_user, lb_passwd):
    try:
        _ssh = paramiko.SSHClient()
        _ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        _ssh.load_system_host_keys()
        _ssh.connect(hostname=lb_ip, username=lb_user, password=lb_passwd)
        return _ssh
        print(_ssh)
    except Exception as exceptHandler:
        logging.error(f"Failed to connect to {ip}: {exceptHandler}")
        raise


def run_command(command):
    _process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
    _output, _error = _process.communicate()
    print(_output.decode())
    return _output.decode().strip()


def get_results(lb_ip, lb_user, lb_pass):
    conn = connect(lb_ip, lb_user, lb_pass)
    if conn is not None:
        print("SSH connection established successfully.")
        stdin, stderr, stdout = conn.exec_command(
            'shell nsconmsg -K /var/nslog/newnslog -g dht_ns_tot_max_limit_exceeds -d current -s disptime=1 | grep dht_ns_tot_max_limit_exceeds'
        )
        print(stdin, stdout, stderr)
        logging.info(f"stdin={stdin}, stdout={stdout}, stderr={stderr}")
        conn.close()
        return stdout.readlines()
    else:
        print("Failed to establish SSH connection.")
        return None

排查与解决建议 #

1. 验证命令本身的有效性 #

手动通过SSH登录LB，执行以下命令，确认是否有输出：

shell nsconmsg -K /var/nslog/newnslog -g dht_ns_tot_max_limit_exceeds -d current -s disptime=1 | grep dht_ns_tot_max_limit_exceeds

• 若手动执行无输出：排查LB的日志文件/var/nslog/newnslog是否存在、权限是否对lb_user开放，或dht_ns_tot_max_limit_exceeds指标是否有数据。
• 若手动执行有输出：继续排查脚本执行环境问题。

2. 修复paramiko命令执行的环境问题 #

paramiko的exec_command默认使用非交互式shell，可能无法正确解析管道或特定命令。修改get_results函数中的命令，用bash -c包裹以确保shell解析：

stdin, stderr, stdout = conn.exec_command(
    'bash -c "shell nsconmsg -K /var/nslog/newnslog -g dht_ns_tot_max_limit_exceeds -d current -s disptime=1 | grep dht_ns_tot_max_limit_exceeds"'
)

3. 捕获命令执行错误信息 #

在读取stdout前先读取stderr，排查命令执行是否有错误：

def get_results(lb_ip, lb_user, lb_pass):
    conn = connect(lb_ip, lb_user, lb_pass)
    if conn is not None:
        print("SSH connection established successfully.")
        stdin, stderr, stdout = conn.exec_command(
            'bash -c "shell nsconmsg -K /var/nslog/newnslog -g dht_ns_tot_max_limit_exceeds -d current -s disptime=1 | grep dht_ns_tot_max_limit_exceeds"'
        )
        # 先读取错误输出
        error_msg = stderr.read().decode().strip()
        if error_msg:
            print(f"Command execution error: {error_msg}")
            logging.error(f"Command execution error: {error_msg}")
        # 再读取标准输出
        output_lines = stdout.readlines()
        conn.close()
        return output_lines
    else:
        print("Failed to establish SSH connection.")
        return None

4. 修复connect函数的错误 #

• print(_ssh)在return之后永远不会执行，可直接删除。
• 异常捕获中的{ip}应为{lb_ip}，参数名不匹配：

def connect(lb_ip, lb_user, lb_passwd):
    try:
        _ssh = paramiko.SSHClient()
        _ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        _ssh.load_system_host_keys()
        _ssh.connect(hostname=lb_ip, username=lb_user, password=lb_passwd)
        return _ssh
    except Exception as exceptHandler:
        logging.error(f"Failed to connect to {lb_ip}: {exceptHandler}")
        raise

5. 修复statusEmail的日期匹配逻辑 #

当前写入statusEmail的是NOT_GOOD: {date}，但读取时用date == statusEmailFileHandle.read()匹配，导致永远不相等。可修改写入逻辑为仅保存日期：

with open("src/statusEmail", "w") as f:
    f.write(date)

或修改读取逻辑提取日期：

with open("src/statusEmail", "r") as statusEmailFileHandle:
    content = statusEmailFileHandle.read().strip()
    saved_date = content.split(": ")[-1] if ": " in content else content
    if date == saved_date:
        logging.info("Email already sent.")
    else:
        # 发送邮件逻辑

内容的提问来源于stack exchange，提问作者Dh3va