基于Vector VRL的YAML配置解析NGINX通用访问日志

解析策略 #

• 优先用内置parse_nginx_log函数：你的日志是标准NGINX通用访问格式，Vector内置函数已适配该格式，之前失败大概率是字段映射或语法细节问题，用它能避免正则编写错误。
• 嵌套nginx子对象：通过VRL的对象构造语法，把解析出的字段规整到nginx键下。
• 补充自定义字段：按要求添加file（从Vector源元数据取日志文件路径）、host（用get_hostname()获取当前主机名）、server（复用解析出的客户端IP）。

如果内置函数仍不生效，可使用自定义正则匹配，适配你日志格式的正则如下：

^(?P<client_ip>\S+) - (?P<remote_user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<http_version>\S+)" (?P<status>\d+) (?P<body_bytes_sent>\d+) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)" "(?P<request_time>[^"]*)"$

YAML配置示例 #

以下是完整的Vector配置，包含日志读取、解析、输出到标准输出的流程：

sources:
  nginx_logs:
    type: file
    path: "/var/log/nginx/access.log" # 替换为你的实际日志路径
    read_from: beginning

transforms:
  parse_nginx:
    type: remap
    inputs: ["nginx_logs"]
    source: |
      # 内置函数解析NGINX日志
      parsed = parse_nginx_log!(.message)
      
      # 构造嵌套的nginx子对象
      .nginx = {
        "client": parsed.client_ip,
        "method": parsed.method,
        "path": parsed.path,
        "http_version": parsed.http_version,
        "status": parsed.status,
        "body_bytes_sent": parsed.body_bytes_sent,
        "referer": parsed.referer,
        "user_agent": parsed.user_agent,
        "request_time": parsed.request_time,
        "timestamp": parsed.timestamp,
        "remote_user": parsed.remote_user
      }
      
      # 添加要求的其他字段
      .file = .source.path
      .host = get_hostname!()
      .server = parsed.client_ip
      
      # 清理冗余原始字段（可选）
      del(.message)
      del(.source)

sinks:
  stdout:
    type: console
    inputs: ["parse_nginx"]
    encoding:
      codec: json

若内置函数失效，替换transform的source部分为正则解析：

source: |
  # 自定义正则解析
  regex = r'^(?P<client_ip>\S+) - (?P<remote_user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<http_version>\S+)" (?P<status>\d+) (?P<body_bytes_sent>\d+) "(?P<referer>[^"]*)" "(?P<user_agent>[^"]*)" "(?P<request_time>[^"]*)"$'
  parsed = parse_regex!(.message, regex)
  
  # 后续逻辑与上面一致
  .nginx = {
    "client": parsed.client_ip,
    "method": parsed.method,
    "path": parsed.path,
    "http_version": parsed.http_version,
    "status": parsed.status,
    "body_bytes_sent": parsed.body_bytes_sent,
    "referer": parsed.referer,
    "user_agent": parsed.user_agent,
    "request_time": parsed.request_time,
    "timestamp": parsed.timestamp,
    "remote_user": parsed.remote_user
  }
  .file = .source.path
  .host = get_hostname!()
  .server = parsed.client_ip
  del(.message)
  del(.source)

测试方法建议 #

• 本地单日志测试：
  1. 创建test_access.log，写入你的示例日志内容。
  2. 修改配置文件的path为./test_access.log。
  3. 运行vector --config vector_config.yaml，查看标准输出的JSON是否符合要求。
• Vector测试框架验证：
  创建测试文件parse_nginx_test.vrl，写入测试用例后用vector test parse_nginx_test.vrl执行，示例用例：

  test "parse nginx log" {
    input = {"message": '47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain1.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"'}
    expected = {
      "file": "./test_access.log",
      "host": "your-hostname",
      "server": "47.29.201.179",
      "nginx": {
        "client": "47.29.201.179",
        "method": "GET",
        "path": "/?p=1",
        "http_version": "HTTP/2.0",
        "status": "200",
        "body_bytes_sent": "5316",
        "referer": "https://domain1.com/?p=1",
        "user_agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
        "request_time": "2.75",
        "timestamp": "28/Feb/2019:13:17:10 +0000",
        "remote_user": "-"
      }
    }
    output = parse_nginx_log!(input.message)
    # 补充字段构造逻辑后断言
    assert output.client_ip == expected.nginx.client
  }
  

• 字段类型转换（可选）：若需将状态码、字节数转为数字类型，在VRL中添加：

  .nginx.status = to_int!(.nginx.status)
  .nginx.body_bytes_sent = to_int!(.nginx.body_bytes_sent)
  .nginx.request_time = to_float!(.nginx.request_time)
  

目标JSON输出示例 #

{
  "file": "/var/log/nginx/access.log",
  "host": "your-server-hostname",
  "server": "47.29.201.179",
  "nginx": {
    "client": "47.29.201.179",
    "method": "GET",
    "path": "/?p=1",
    "http_version": "HTTP/2.0",
    "status": 200,
    "body_bytes_sent": 5316,
    "referer": "https://domain1.com/?p=1",
    "user_agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
    "request_time": 2.75,
    "timestamp": "28/Feb/2019:13:17:10 +0000",
    "remote_user": "-"
  }
}

内容的提问来源于stack exchange，提问作者GangGreenTemperTatum