如何使用Linux「at」功能调度Docker中的PySpark脚本夜间运行

Got it, let's get your 2-hour PySpark while-loop job scheduled to run overnight using Linux's at command. Here's a tailored step-by-step guide that fits your current workflow:

First, we'll package your Docker and Spark commands into a shell script—this avoids parsing issues with complex commands in at and makes maintenance easier.

Create a file named run_spark_plog.sh (you can use any name you prefer) with the following content:

#!/bin/bash
# Run the PySpark job inside the Docker container (non-interactive mode)
docker exec r-hadoop-spark-yarn-spark-master-1-085ae410 \
  nohup /opt/spark-2.0.2/bin/spark-submit \
    --master spark://hadoop-spark-yarn-spark-master-1:7077 \
    --executor-memory 8G \
    bm_plog_hdfs.py &> /path/to/your/logs/SparkPLOG.out &

• Critical tweak: I removed -it from the docker exec command. Since at runs jobs in the background without an interactive terminal, -it would throw errors—non-interactive execution is required here.
• Log path: Replace /path/to/your/logs/ with an absolute path (either inside the Docker container or a host-mounted directory, so you can easily access logs later). If you want to keep the log in the container's working directory, use a path like /your/container/working/dir/SparkPLOG.out.

Grant execution permissions to the script with this command:

chmod +x /path/to/run_spark_plog.sh

Replace /path/to/ with the actual directory where you saved the script.

Use at to schedule the script to run at your desired overnight time. Here are common examples:

• Run tomorrow at 2 AM:

  echo "/path/to/run_spark_plog.sh" | at 02:00 tomorrow
  

• Run tonight at 10 PM:

  echo "/path/to/run_spark_plog.sh" | at 22:00
  

• Run on a specific date (e.g., Oct 15th at 1 AM):

  echo "/path/to/run_spark_plog.sh" | at 01:00 10/15/2024
  

• Check pending jobs: Use atq to list all active at tasks—you'll see a job number, scheduled time, and your username.
• Cancel a job: If you need to abort a scheduled task, run atrm <job-number> (replace <job-number> with the value from atq).

• Ensure the atd service is running on your server. If not, start and enable it with:

  sudo systemctl start atd
  sudo systemctl enable atd  # Keeps the service running after reboot
  

• Make sure the user running at has permission to execute docker exec commands. If needed, add sudo before docker exec in the script (or run at with sudo privileges).
• Remember: at is for one-time tasks. If you need the job to run every night, you'll want to use cron instead—just let me know if you need help setting that up!

内容的提问来源于stack exchange，提问作者Bruce Okallau