如何获取Qsub/Torque中已完成/终止作业状态并生成状态文件

Great question—dealing with disconnected job emails and scattered output files is a common headache when working with Torque. Let’s break this down into two actionable solutions for your needs:

1. Save Job Status Info (From Emails) to the Same Directory as .o/.e Files #

Instead of relying on emails to track job status, you can automatically capture that detailed resource and exit information directly into a file alongside your file.oXXXX and file.eXXXX outputs. Here are two reliable methods:

Method 1: Use qacct for Full Mail-like Details #

Torque’s qacct command pulls the exact same information sent in job completion emails (resource allocation, exit status, CPU/memory usage, etc.). Add this snippet to the end of your job script to save it to a linked file:

# Capture full job status and save to the submission directory
qacct -j $PBS_JOBID > "${PBS_O_WORKDIR}/job_status_${PBS_JOBID}.txt"

• $PBS_JOBID: Automatically set by Torque for your running job (matches the XXXX in .oXXXX/.eXXXX)
• $PBS_O_WORKDIR: The directory you submitted the job from (where your .o/.e files are stored)
• The output file job_status_XXXX.txt will have identical content to the email, and is directly linked to your job outputs via the job ID.

Method 2: Basic Status Capture (No qacct Required) #

If you don’t have access to qacct (e.g., permission restrictions), you can capture core status details using environment variables and basic commands:

# Write essential job status to a file
cat > "${PBS_O_WORKDIR}/job_status_${PBS_JOBID}.txt" << EOF
Job ID: $PBS_JOBID
Exit Code: $?
Submission Directory: $PBS_O_WORKDIR
Start Time: $(date -d "@$PBS_STARTTIME")
Completion Time: $(date)
EOF

This gives you the key info without needing extra tools.

2. Retrieve Status of Completed/Terminated Torque Jobs #

To check past job statuses anytime, use these Torque built-in commands:

• qstat -f <job-id>: Shows full details for a specific job, even after completion (as long as Torque hasn’t purged the job history). Add -x to list all terminated jobs, or -a to view all jobs (active + completed).
  Example:

  qstat -f 12345.cluster-headnode
  

• qacct -j <job-id>: Pulls the complete audit trail for a job—this is the same data used in completion emails. Use -u <your-username> to get a full history of all your completed jobs:
  Example:

  qacct -j 12345.cluster-headnode
  qacct -u your_username
  

• qhist (if available): A user-friendly tool that displays job history in a formatted table, with filters for time ranges, queues, or users. Example:

  qhist -u your_username -d 7  # Show jobs from the last 7 days
  

Note: If jobs have been removed from Torque’s active history (after the cluster’s retention period), you’ll need to ask your admin to check the cluster’s log files for older records.

内容的提问来源于stack exchange，提问作者Jim Newton