Ruby实现百万级数组数据转指定格式并导出文本文件求助

Hey there! Since you're new to Ruby and working with potentially millions of records, let's walk through a reliable, efficient way to convert your data and export it to output.txt.

First, Let's Clarify the Requirement #

You have a collection of data_array elements, each structured as:
[hash_value, tx_array, time_value]

We need to expand this so each element in tx_array gets its own line in the output file, with each line containing:
hash_value, tx_element, time_value (or your preferred separator)

Solution 1: Basic (For Small/Test Data) #

If you're starting with test data (like your 8 data_array examples), this simple approach works perfectly:

# Example test data (replace with your actual dataset)
test_data = [
  ["abc123", ["tx_001", "tx_002", "tx_003"], "2024-05-20 14:30"],
  ["def456", ["tx_004"], "2024-05-20 14:35"],
  # Add more data_array entries here
]

# Open output file (block mode auto-closes the file when done—no resource leaks!)
File.open("output.txt", "w") do |file|
  test_data.each do |data|
    # Destructure the data_array into individual variables (super readable for beginners!)
    hash_val, tx_list, time_val = data
    
    # Loop through each tx in the tx_array and write a line
    tx_list.each do |tx|
      # Use comma separator here—swap with "\t" for tabs, if needed
      line = "#{hash_val},#{tx},#{time_val}\n"
      file.write(line)
    end
  end
end

Solution 2: Robust & Efficient (For Million-Scale Data) #

When dealing with millions of records, we need to avoid loading all data into memory at once. Also, using Ruby's built-in CSV library ensures we handle special characters (like commas in tx values) correctly, so your output stays clean.

require 'csv'

# If your data comes from a file (e.g., one data_array per line in JSON format)
File.open("your_input_data.txt", "r") do |input_file|
  File.open("output.txt", "w") do |output_file|
    # Initialize CSV writer with your preferred separator
    csv_writer = CSV.new(output_file, col_sep: ',')
    
    # Process input line by line (no massive memory usage!)
    input_file.each_line do |line|
      # Parse the line into a data_array (adjust parsing based on your input format)
      data_array = JSON.parse(line) # Use this if input is JSON; tweak for other formats
      
      hash_val, tx_list, time_val = data_array
      
      tx_list.each do |tx|
        # CSV writer automatically escapes special characters (e.g., commas in tx text)
        csv_writer << [hash_val, tx, time_val]
      end
    end
  end
end

Key Tips for Success #

• Avoid loading all data into memory: Process records one (or a small batch) at a time instead of storing everything in an array. This prevents your program from crashing with large datasets.
• Use the CSV library for structured output: If you need a standard CSV file (e.g., to open in Excel), the CSV gem takes care of escaping rules and formatting automatically.
• Adjust parsing logic: If your input data isn't JSON, replace JSON.parse(line) with the appropriate parsing method (e.g., YAML.load for YAML, or custom string splitting for plain text).

If you share snippets of your original code, I can help pinpoint exactly where it went wrong too!

内容的提问来源于stack exchange，提问作者Rubz