ELK新手求助：Docker部署下Logstash无法向Elasticsearch传输日志

Docker部署ELK栈后，Python脚本生成的日志无法传输到Logstash，Logstash容器显示已正常监听5959端口，请求排查故障。

Logstash配置（logstash.conf） #

input {
    udp {
        port => 5959
        codec => json 
    }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => "elasticsearch:9200"
        index => "logdb" 
    }
}

Python日志生成脚本 #

import logging
import logstash

class CustomLogStashLogger:
    def __init__(self, logger_name='demo-py-logger', host='0.0.0.0', udp=5959):
        self.logger_name = logger_name
        self.host = host
        self.udp_port = udp

    def get(self):
        logging.basicConfig(
            filename="logfile",
            filemode="a",
            format="%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s",
            datefmt="%H:%M:%S",
            level=logging.INFO,
        )

        self.stderrLogger = logging.StreamHandler()
        logging.getLogger().addHandler(self.stderrLogger)
        self.logger = logging.getLogger(self.logger_name)
        self.logger.addHandler(logstash.LogstashHandler(self.host,
                                                        self.udp_port,
                                                        version=1))
        return self.logger

logger_instance = CustomLogStashLogger(udp=5959, host='localhost', logger_name='akhil')
logger = logger_instance.get()

count = 0
#from time import sleep
for i in range(0, 10):
    count = count + 1
    if count % 2 == 0:
        logger.error('Error Message Code Failed :{} '.format(count))
    else:
        logger.info('python-logstash: test logstash info message:{} '.format(count))

Logstash容器日志 #

[2022-08-31T08:00:31,019][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-08-31T08:00:31,039][INFO ][logstash.inputs.udp ][main][ba85faa6d462179e21c318cb3447f81446e77e02b6d4d9382e62aa12b138b91a] Starting UDP listener {:address=>"0.0.0.0:5959"}
[2022-08-31T08:00:31,087][INFO ][logstash.inputs.udp ][main][ba85faa6d462179e21c318cb3447f81446e77e02b6d4d9382e62aa12b138b91a] UDP listener started {:address=>"0.0.0.0:5959", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2022-08-31T08:00:31,328][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

排查与解决方案 #

1. 网络连通性问题 #

• Python脚本运行在宿主机，Logstash在Docker容器内，脚本中host='localhost'仅指向宿主机端口，需确认Logstash容器是否做了UDP端口映射：
  • 执行docker ps查看Logstash容器端口映射，确保5959/udp已映射到宿主机；若未映射，启动容器时添加参数-p 5959:5959/udp。
  • 或直接使用Logstash容器IP作为脚本中的host值，容器IP可通过docker inspect <logstash容器ID>获取。

2. 日志格式不匹配 #

• Logstash输入配置指定codec => json，但logstash.LogstashHandler（version=1）默认发送msgpack格式，导致Logstash无法解析日志并直接丢弃：
  • 方案一：修改Logstash输入配置，去掉codec => json，或替换为对应格式的codec：

    input {
        udp {
            port => 5959
            codec => msgpack
        }
    }
    

  • 方案二：调整Python脚本，使用支持JSON输出的handler，或修改handler参数确保输出JSON格式。

3. Python日志级别配置 #

• 脚本中logging.basicConfig设置了level=logging.INFO，但需显式给自定义logger实例设置级别，避免因级别过滤导致日志无法发送：
  在self.logger = logging.getLogger(self.logger_name)后添加self.logger.setLevel(logging.INFO)。

4. 防火墙/安全组限制 #

• 检查宿主机防火墙是否允许UDP 5959端口的入站、出站流量，确保日志传输未被拦截。

内容的提问来源于stack exchange，提问作者AkhilTC