使用TFX CsvExampleGen读取分号CSV报错，如何传参给CsvCoder？

Great question! The issue you’re hitting happens because CsvExampleGen defaults to using commas as the CSV delimiter, but your file uses semicolons. To fix this, you can pass custom configuration parameters to CsvExampleGen that get forwarded to tft.coders.CsvCoder under the hood.

Here’s how to adjust your code:

1. Create a custom configuration dictionary specifying your delimiter:

   custom_config = {
       'csv_delimiter': ';'
   }
   

2. Pass this config to CsvExampleGen:

   from tfx.components import CsvExampleGen
   from tfx.utils.dsl_utils import external_input
   
   data_path = './data'
   intro_component = CsvExampleGen(
       input=external_input(data_path),
       custom_config=custom_config
   )
   

Why this works: #

The CsvExampleGen executor checks for the csv_delimiter key in the custom_config parameter. If found, it uses this value to initialize tft.coders.CsvCoder with your specified delimiter instead of the default comma. This ensures your semicolon-separated CSV is parsed correctly, resolving the ValueError: Columns do not match specified csv headers error.

For newer TFX versions (using TFXIO): #

If you’re on TFX v1.5 or later, you can use CsvIoConfig for more granular control over CSV parsing (like quote characters, skipped rows, etc.):

from tfx.components import CsvExampleGen
from tfx.utils.dsl_utils import external_input
from tfx.extensions.google_cloud_big_query import csv_utils

data_path = './data'
csv_io_config = csv_utils.CsvIoConfig(delimiter=';')
intro_component = CsvExampleGen(
    input=external_input(data_path),
    custom_config={'csv_io_config': csv_io_config.to_json()}
)

内容的提问来源于stack exchange，提问作者Oleks