如何使用GNU Parallel传递数组参数运行Python脚本test.py？

Let's break down exactly how to use GNU Parallel to pass array-based parameters to your test.py script, depending on your specific use case. I’ll cover the most common scenarios and fix typical mistakes people make.

Scenario 1: You have a bash array of argument sets #

Suppose you’ve defined a bash array where each element is a complete set of arguments you want to pass to test.py:

# Example array: each element is a space-separated list of args for test.py
job_args=(
    "input1.txt output1.csv --verbose"
    "input2.txt output2.csv"
    "input3.txt output3.csv --debug"
)

To run test.py in parallel for each element in this array, use:

parallel python test.py {} ::: "${job_args[@]}"

The ::: tells Parallel to take the following items as input values. The "${job_args[@]}" expands each array element as a separate input, and {} gets replaced with the full element for each job. This is exactly equivalent to running each of these commands in parallel:

python test.py input1.txt output1.csv --verbose
python test.py input2.txt output2.csv
python test.py input3.txt output3.csv --debug

Scenario 2: You want to generate combinations of parameters #

If you have separate arrays for different parameter positions (e.g., input files and output files), and want all possible combinations, you can use multiple ::: clauses:

input_files=(input1.txt input2.txt input3.txt)
output_files=(output1.csv output2.csv output3.csv)
flags=(--verbose --debug "")

parallel python test.py {} {} {} ::: "${input_files[@]}" ::: "${output_files[@]}" ::: "${flags[@]}"

This will run every combination of input, output, and flag. For example, one job would be python test.py input1.txt output2.csv --debug.

Scenario 3: Parameters are stored in a text file #

If your argument sets are in a newline-separated file (each line is a full set of args for test.py), use the :::: syntax to read from the file:

# params.txt content:
# input1.txt output1.csv --verbose
# input2.txt output2.csv
# input3.txt output3.csv --debug

parallel python test.py {} :::: params.txt

This works exactly like the bash array scenario, but pulls input from a file instead.

Common Mistakes to Avoid #

• Forgetting to quote the array: If you write parallel python test.py {} ::: ${job_args[@]} (without quotes), bash will split each array element into separate words, breaking arguments with spaces. Always use "${job_args[@]}".
• Passing the entire array as a single argument: Using ::: "${job_args}" (without [@]) passes the entire array as one long string, which will be treated as a single job with all args mashed together.
• Not escaping special characters: If your arguments contain spaces, quotes, or special shell characters, make sure to escape them properly in your array or text file, or use parallel -q to quote the arguments automatically.

For example, if an argument has a space like "my input.txt", you can define it in the array as:

job_args=(
    "\"my input.txt\" output1.csv"
)

Or use parallel -q to handle quoting:

parallel -q python test.py {} ::: "${job_args[@]}"

内容的提问来源于stack exchange，提问作者ajohnrobertson