GitLab备份未自动删除排查求助（AWS EC2 Omnibus部署）

Hey there, let’s work through why your GitLab old backups aren’t getting cleaned up automatically. I’ve troubleshooted this exact scenario with Omnibus deployments on AWS EC2 before, so here’s a step-by-step breakdown to get to the root of it:

Start by double-checking the settings in your gitlab.rb file—this is where the backup cleanup rules are defined:

• Look for the gitlab_rails['backup_keep_time'] parameter. This sets how long backups are retained (in seconds). For example, 7 days equals 604800. Make sure this value is set correctly for your needs.
• Critical: After modifying gitlab.rb, you must run sudo gitlab-ctl reconfigure for changes to take effect. It’s easy to forget this step!
• Also, confirm you’re not using object storage (like S3) for backups. If you are, local cleanup logic won’t apply, and you’ll need to check your bucket’s lifecycle rules instead. Since your disk is filling up, though, this is likely a local backup issue.

Omnibus GitLab stores all logs in /var/log/gitlab/—focus on these key logs for backup issues:

• gitlab-rails/backup.log: This is the primary log for backup operations. It’ll show exactly how GitLab evaluates old backups, why it’s choosing not to delete them, or if there are permission errors. Run sudo tail -f /var/log/gitlab/gitlab-rails/backup.log while executing your backup command to see real-time details.
• gitlab-rails/production.log: If you’re using GitLab’s scheduled backup jobs (via Sidekiq), this log may contain traces of the backup/cleanup task execution.
• sidekiq/current: For scheduled backups, search this log for BackupWorker to confirm the cleanup task is actually being triggered.

GitLab stores local backups in /var/opt/gitlab/backups/ by default, with filenames like 1690000000_2023_07_22_15.10.1_gitlab_backup.tar (timestamp + version + backup suffix).

• Run ls -la /var/opt/gitlab/backups/ to check:
  • File timestamps: Are any backups clearly older than your backup_keep_time setting? If not, the "0 removed" message is normal.
  • Permissions: Ensure all backup files and the backups/ directory are owned by the git user/group (git:git). GitLab runs as this user, so missing permissions will prevent it from deleting files.
• If you’ve manually renamed any backup files (e.g., changed the suffix or removed the timestamp), GitLab’s cleanup script won’t recognize them as valid backups—these won’t be deleted automatically.

Run the cleanup command alone with trace output to see exactly what’s happening:

sudo gitlab-rake gitlab:backup:clean --trace

The --trace flag will print every step of the cleanup logic, including how it calculates which backups are expired, and any errors encountered (like permission issues or unrecognized files).

If you’re relying on GitLab’s built-in scheduled backups:

• Confirm the scheduled task is using the same configuration as your manual runs. Running sudo gitlab-ctl reconfigure ensures the scheduled job uses the latest gitlab.rb settings.
• Check that the Sidekiq service is running with sudo gitlab-ctl status sidekiq—if it’s down, scheduled backups (and cleanup) won’t execute.

内容的提问来源于stack exchange，提问作者user_01_02