如何断开本地连接后在服务器持续运行深度学习训练任务？

Great question—this is a super common pain point when running long training jobs (10+ hours) on remote servers. I’ve dealt with this dozens of times, so here are the most reliable, practical methods I recommend:

1. nohup (Quick & Minimal Setup) #

This is the simplest option if you don’t need to reattach to the training session later—it just keeps the process running even after you log out, and saves output to a log file.

• Usage: Prefix your training command with nohup and append & to send it to the background:

  nohup python train.py > training_$(date +%Y%m%d).out 2>&1 &
  

  • nohup: Ignores the "hangup" signal sent when you disconnect your local device.
  • > training_$(date +%Y%m%d).out: Redirects all regular output to a timestamped log file (so you can check progress later).
  • 2>&1: Sends error messages to the same log file (instead of losing them).
  • &: Runs the process in the background so you can keep using the terminal.

• Check progress:

  • View the log file in real-time with tail -f training_20240520.out.
  • Find the process ID with ps aux | grep train.py.

• Stop the job: Use kill <process_id> (replace <process_id> with the number from ps aux).

Pros: No extra software needed (pre-installed on almost all Linux servers).
Cons: Can’t reattach to the live terminal session—you’ll only see output in the log file.

2. screen (Resumable Terminal Sessions) #

screen lets you create persistent terminal sessions that you can detach from, log out, and reattach to later. It’s more flexible than nohup because you can interact with the running job again.

• Basic workflow:

  1. Connect to your server and create a named session (easier to find later):

     screen -s my_training_session
     

  2. Run your training command normally (e.g., python train.py).
  3. Detach from the session without stopping it: Press Ctrl+A then D (you’ll see a message like [detached from 1234.my_training_session]).
  4. Disconnect your local device—your model keeps training on the server.
  5. When you reconnect later, list all active sessions:

     screen -ls
     

  6. Reattach to your training session:

     screen -r my_training_session
     

• Troubleshooting: If the session says "attached" (e.g., if you accidentally disconnected without detaching), use screen -x my_training_session to force reattach.

Pros: Easy to use, lets you resume the live terminal session.
Cons: Less feature-rich than tmux, interface is basic.

3. tmux (Powerful Terminal Multiplexing) #

tmux is the upgraded version of screen—it’s more stable, has better window management, and is the go-to tool for most remote server users these days.

• Setup: First install it if your server doesn’t have it (most do, but just in case):

  # Debian/Ubuntu
  sudo apt install tmux
  # RHEL/CentOS
  sudo yum install tmux
  

• Basic workflow:

  1. Create a named session:

     tmux new -s training_job
     

  2. Run your training command.
  3. Detach the session: Press Ctrl+B then D (you’ll get a detached confirmation).
  4. Reconnect later, list sessions:

     tmux ls
     

  5. Reattach to your session:

     tmux attach -t training_job
     

• Bonus features:

  • Split the terminal window: Ctrl+B then % (vertical split) or " (horizontal split) to monitor GPU usage (e.g., nvidia-smi -l) alongside training logs.
  • Switch between windows: Ctrl+B then N (next window) or P (previous window).

Pros: Stable, flexible, tons of features for managing multiple tasks in one session.
Cons: Requires learning a few shortcuts, but the most useful ones are easy to memorize.

Extra Tips for Reliability #

• Always log output: Even with screen/tmux, redirecting output to a file is a safety net. If the session crashes, you’ll still have the full training log.
• Monitor GPU usage: Use nvidia-smi or gpustat (install with pip install gpustat) to check if your GPU is actually being utilized while training.
• For critical jobs: If you want the training to restart even if the server reboots, set up a systemd service. This is more advanced, but involves creating a .service file in /etc/systemd/system/ to manage the training process as a system daemon.

I personally use tmux for almost all my long training jobs—being able to split the screen to watch GPU stats and training logs at the same time saves me a lot of hassle. Pick the method that fits your workflow best!

内容的提问来源于stack exchange，提问作者user11856735