使用Logstash导入CSV至Elasticsearch报错及配置求助

Hey there, let's work through your Logstash to Elasticsearch CSV import issue together. I’ve spotted a few potential problems in your config, plus some missing pieces that are likely causing errors. Let’s break this down step by step.

1.1 Incomplete CSV Columns #

Your columns list cuts off at "L..." — this is a big red flag. Logstash needs a full, exact list of columns that matches your CSV file (including the final column). If even one column is missing or misnamed, the CSV filter will fail to parse rows correctly. Double-check your CSV’s header row and fill in all column names completely.

1.2 Windows Path Escaping #

On Windows, Logstash interprets single backslashes (\) as escape characters, so your file path won’t be recognized properly. Fix this in one of two ways:

• Use double backslashes: "C:\\Users\\welcome\\Dropbox\\IT_Department\\BigDataProjects\\StudentWithdraElastic\\student_withdraw.csv"
• Switch to forward slashes: "C:/Users/welcome/Dropbox/IT_Department/BigDataProjects/StudentWithdraElastic/student_withdraw.csv"

1.3 Sincedb Path Misconfiguration #

Your sincedb_path => "j:\null" has two issues: the unescaped backslash, and using null instead of the Windows-specific null device. To force Logstash to read the CSV from the start every time (useful for testing), set this to:
sincedb_path => "NUL"

You only shared the input and filter blocks, but Logstash requires an output block to send data to Elasticsearch. Here’s a standard, configurable output setup:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"] # Update this to your Elasticsearch address
    index => "student_withdraw" # Name your Elasticsearch index here
    document_id => "%{STUDENT_NO}" # Optional: Use student number to avoid duplicate entries
  }
  stdout { codec => rubydebug } # Optional: Print parsed data to console for debugging
}

• File Permissions: Make sure the user running Logstash has read access to your CSV file. On Windows, this might mean adjusting file properties to grant read permissions to the Logstash service account.
• CSV File Integrity: Open your CSV in a text editor (not just Excel) to check for formatting errors: unclosed quotes, inconsistent column counts per row, or malformed special characters. These will break the CSV filter.
• Logstash Logs: Check Logstash’s log files (usually in logstash-<version>/logs) for specific error messages. The logs will tell you exactly what’s failing (e.g., "column mismatch" or "file not found").

Here’s a complete, tested config incorporating all the fixes above (fill in your full column list):

input {
  file {
    path => "C:/Users/welcome/Dropbox/IT_Department/BigDataProjects/StudentWithdraElastic/student_withdraw.csv"
    start_position => "beginning"
    sincedb_path => "NUL"
  }
}

filter {
  csv {
    separator => ","
    columns => ["DEPT_NAME", "CERT_NAME", "SPEC_NAME", "STUDENT_NO", "STUD_NAME", "GENDER", "ADVISORS_NAME", "ACADEMIC_YEAR", "REQUEST_NO", "withdraw_reason_category", "STATUS", "WITHDRAW_DATE", "LAST_UPDATED"] # Replace with your full column list
    skip_header => true # Add this if your CSV has a header row to skip parsing it as data
  }
  # Optional: Convert date fields to Elasticsearch-compatible format
  date {
    match => ["WITHDRAW_DATE", "yyyy-MM-dd"] # Adjust the date format to match your CSV
    target => "@timestamp" # Optional: Overwrite Logstash's default timestamp with your withdraw date
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "student_withdraw"
  }
  stdout { codec => rubydebug }
}

1. Save the corrected config as student_withdraw.conf
2. Open a command prompt, navigate to your Logstash bin directory, and run:

   logstash -f student_withdraw.conf
   

3. Check the console output — if you see formatted data from your CSV, parsing is working. Then verify the student_withdraw index exists in Elasticsearch and contains your data.

内容的提问来源于stack exchange，提问作者Muhammad Idrees