Pentaho按唯一Plant生成CSV：字段过滤与文件数据异常问题咨询

Let's break down how to fix the data anomalies you're seeing with your Pentaho workflow. From what you described, you've got a solid foundation—extracting unique plant values in one transformation, then using those as parameters to generate individual CSVs in a second. Here are the most common culprits and how to debug them:

1. Verify Parameter Passing #

• First, make sure the plant parameter's data type matches exactly what's in your tb_rawcsvdata table. If the source plant is a string, don't accidentally pass it as a number—this can cause silent filtering failures.
• In your second transformation, enable the Show parameter values option in the run configuration. This lets you see exactly what plant value is being passed during execution.
• Add a Write to log step at the start of the second transformation to print the received plant parameter. This confirms the value isn't getting lost or modified mid-flow.

2. Validate Your Data Query Logic #

• Double-check the SQL query in your second transformation. It should use parameter binding (not string concatenation) to avoid escaping issues, like this:

  SELECT plant, employeenumber, term_dt 
  FROM tb_rawcsvdata 
  WHERE plant = ?
  

• If your plant values include special characters (spaces, single quotes, etc.), parameter binding ensures they're handled correctly. Test the query directly in your database with a known plant value to confirm it returns the expected rows.
• Make sure there's no unintended filtering (like an extra WHERE clause) or missing joins that would truncate or alter your data.

3. Check CSV Output Configuration #

• Confirm your CSV output step's field mappings are correct. It's easy to accidentally map employeenumber to term_dt or skip a field entirely, which will mess up your data structure.
• Review delimiter and quoting settings: If term_dt uses a non-standard date format, set the correct format in the field configuration. If any fields contain your CSV delimiter, enable Quote all strings to prevent data from being split incorrectly.
• Check the Error handling tab in the CSV output step. If you're ignoring error rows, enable logging for skipped rows to see why they're being excluded (e.g., invalid data types, missing values).

4. Audit the Unique Plant Result Set #

• In your first transformation, verify the Unique rows step is actually returning distinct plant values. Add a Write to log step to print all unique plants—this will catch duplicates or NULL values that might be causing unexpected CSV outputs.
• If you're using a Job to orchestrate the transformations, confirm the Copy results to parameters setting in the Execute a transformation job entry is correctly mapping the result set's plant field to the second transformation's parameter.
• NULL plant values in the result set will generate a CSV with all NULL plant rows—make sure you filter those out in the first transformation if they're not intended.

5. Test with a Single Parameter #

• Run the second transformation manually with a single, known plant value. If the generated CSV is correct, the problem lies in how parameters are being passed in bulk. If it's still broken, focus on fixing the second transformation's internal logic and configuration first.

内容的提问来源于stack exchange，提问作者Balaji