如何用GNU parallel读取单个txt文件并并行执行其中的Linux命令？

Got it, let's break this down simply—running commands from a single text file with GNU Parallel is straightforward once you know the right syntax. Here's what you need to do:

Step 1: Prepare Your Command File #

First, make sure your .txt file has one command per line. No extra line breaks between commands, and each entry should be exactly how you'd run it directly in the shell. For example, your commands.txt might look like this:

echo "Processing file A" && gzip /data/fileA.csv
ls -lh /backup | grep "2024-"
sleep 3 && curl -s https://example.com > temp.html

Step 2: Basic Execution #

The core command to run all lines in parallel is:

parallel < commands.txt

That's it! GNU Parallel will read each line from the file as a separate job and execute them in parallel (by default, it uses one job per CPU core).

Useful Options to Customize Behavior #

• Limit parallel jobs: Use -j to set how many commands run at once. For example, to cap it at 3 simultaneous jobs:

  parallel -j 3 < commands.txt
  

• Test without running: Use --dry-run to preview commands without executing them—perfect for debugging typos:

  parallel --dry-run < commands.txt
  

• Log job results: Track which jobs succeeded or failed with --joblog:

  parallel --joblog parallel_log.txt < commands.txt
  

• Handle special characters: If your commands include quotes, pipes, or shell variables, explicitly specify the shell to ensure proper interpretation (though Parallel does this by default for complex lines):

  parallel -S /bin/bash < commands.txt
  

Troubleshooting Common Issues #

• Commands misbehaving: Double-check that each command is on a single line. If a command needs to span lines, use a backslash (\) at the end of each line in your text file to continue it.
• Permission errors: Ensure your user has access to run each command, and that the text file is readable (run chmod 644 commands.txt if needed).
• Variables not expanding: If your commands use shell variables like $HOME, make sure they're either expanded in the text file or that Parallel runs the command in a shell with access to those variables (the default setup should handle this, but -S /bin/bash can help if not).

Content of the question comes from Stack Exchange, asked by SG Kwon