TensorFlow中基于队列加载CSV构建RNN输入管线报错求助

Hey there! I get it—building input pipelines in TensorFlow can be tricky, especially when your CSVs have a special structure that doesn't play nice with the default TextLineReader. Let's break down how to fix that genfromtxt error and get your data flowing into your RNN smoothly.

First, let's understand the core issue #

When you're using TensorFlow's queue-based APIs (like string_input_producer), you're working within the TensorFlow computation graph. Numpy functions like genfromtxt run outside this graph, so mixing them directly can cause compatibility issues—this is probably why you're seeing errors. Plus, the old queue APIs are a bit outdated; TensorFlow now recommends using tf.data.Dataset for input pipelines, which is more flexible and easier to debug.

Solution 1: Use tf.data.Dataset with TensorFlow-native column parsing #

This approach avoids mixing Numpy and TensorFlow graph operations entirely. You can read the CSV files, split them into lines, decode the columns, and then pick the ones you need—all within TensorFlow:

import tensorflow as tf

def load_selected_columns(file_path):
    # Read the entire CSV file as text
    raw_text = tf.io.read_file(file_path)
    # Split text into individual lines, filter out empty lines
    lines = tf.strings.split(raw_text, sep="\n")
    non_empty_lines = tf.boolean_mask(lines, tf.strings.length(lines) > 0)
    
    # Define default values for each column (adjust based on your CSV structure)
    # Example: if your CSV has 5 columns, all floats
    record_defaults = [tf.float32] * 5
    # Decode each line into columns
    columns = tf.io.decode_csv(non_empty_lines, record_defaults=record_defaults)
    
    # Pick the columns you need (e.g., columns 1 and 3, 0-indexed)
    selected_cols = tf.stack([columns[1], columns[3]], axis=1)
    return selected_cols

# Create a dataset of your CSV file paths
file_dataset = tf.data.Dataset.list_files("/path/to/your/csvs/*.csv")
# Map the loading function to each file
csv_dataset = file_dataset.map(load_selected_columns)

# Prepare the dataset for your RNN: batch, shuffle, repeat as needed
rnn_dataset = csv_dataset.shuffle(buffer_size=1000).batch(32).repeat()

# Iterate over the dataset to feed your RNN
for batch in rnn_dataset.take(5):
    print(f"Batch shape: {batch.shape}")  # Should be (32, num_timesteps, 2) if your data is sequential

Solution 2: Wrap Numpy's genfromtxt in tf.py_function (if you need to stick with Numpy) #

If you really need to use genfromtxt (e.g., for complex parsing logic TensorFlow can't handle), you can wrap it in tf.py_function to make it compatible with the TensorFlow graph. Here's how to do it with the old queue API (though again, tf.data is better):

import tensorflow as tf
import numpy as np

def read_csv_with_numpy(file_path_tensor):
    # Convert the TensorFlow string path to a Python string
    file_path = file_path_tensor.numpy().decode("utf-8")
    # Use genfromtxt to read specific columns (usecols defines which columns to pick)
    # Adjust delimiter, dtype, and usecols to match your CSV
    data = np.genfromtxt(file_path, delimiter=",", usecols=(1, 3), dtype=np.float32)
    return data

# Create a queue of CSV filenames
file_queue = tf.train.string_input_producer(["file1.csv", "file2.csv"])
reader = tf.WholeFileReader()
_, file_path = reader.read(file_queue)

# Wrap the Numpy function in tf.py_function to integrate with the graph
csv_data = tf.py_function(
    func=read_csv_with_numpy,
    inp=[file_path],
    Tout=tf.float32
)
# Set a shape for the tensor (adjust based on your data's dimensions)
csv_data.set_shape((None, 2))  # None for variable-length rows, 2 for the two columns we picked

# Create batches for your RNN
batch_data = tf.train.batch([csv_data], batch_size=32)

# Run the pipeline (note: old queue APIs require a session and coordinator)
with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)
    
    try:
        while not coord.should_stop():
            batch = sess.run(batch_data)
            print(f"Batch shape: {batch.shape}")
    except tf.errors.OutOfRangeError:
        print("Finished reading all CSV files")
    finally:
        coord.request_stop()
        coord.join(threads)

Common fixes for genfromtxt errors #

• Double-check your delimiter parameter—if your CSV uses tabs or semicolons instead of commas, genfromtxt will fail to parse correctly.
• Make sure the usecols indices match your actual CSV columns (remember, it's 0-indexed).
• Verify that your file paths are correct—if the path is a TensorFlow string tensor, you need to decode it to a Python string first (like in Solution 2).

内容的提问来源于stack exchange，提问作者Ploo