GitLab-CE打开CI/CD设置页面时出现500错误求助

GitLab-CE 环境 #

Ruby:         ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
GitLab:       16.1.2 (e60fc11f2d3) FOSS
GitLab Shell: 14.23.0
PostgreSQL:   13.11

GitLab-Runner 版本 #

Version:      16.1.0
Git revision: b72e108d
Git branch:   16-1-stable
GO version:   go1.19.9
Built:        2023-06-21T21:52:30+0000
OS/Arch:      linux/amd64

打开管理员设置或项目设置中的CI/CD板块时触发500内部错误，错误截图如下：

访问项目http://Server_IP:Server_PORT/android/sade/-/settings/ci_cd时，production.log输出如下：

{"method": "GET","path": "/android/sade/-/settings/ci_cd","format": "html","controller": "Projects::Settings::CiCdController","action": "show","status": 500,"time": "2023-07-09T15:10:22.007Z","params": [{"key": "namespace_id","value": "android"},{"key": "project_id","value": "sade"}],"correlation_id": "01H4XK8ANDM5TGZ7YPWCW4XXQ4","meta.caller_id": "Projects::Settings::CiCdController#show","meta.remote_ip": "192.168.10.118","meta.feature_category": "continuous_integration","meta.user": "root","meta.user_id": 1,"meta.project": "android/sade","meta.root_namespace": "android","meta.client_id": "user/1","remote_ip": "192.168.10.118","user_id": 1,"username": "root","ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.1 Safari/605.1.15","queue_duration_s": 0.005301,"response_bytes": 0,"request_urgency": "low","target_duration_s": 5,"gitaly_calls": 1,"gitaly_duration_s": 0.005268,"redis_calls": 22,"redis_allowed_cross_slot_calls": 1,"redis_duration_s": 0.00463,"redis_read_bytes": 3412,"redis_write_bytes": 2285,"redis_feature_flag_calls": 14,"redis_feature_flag_duration_s": 0.002336,"redis_feature_flag_read_bytes": 2839,"redis_feature_flag_write_bytes": 877,"redis_repository_cache_calls": 3,"redis_repository_cache_duration_s": 0.000753,"redis_repository_cache_read_bytes": 330,"redis_repository_cache_write_bytes": 170,"redis_sessions_calls": 3,"redis_sessions_allowed_cross_slot_calls": 1,"redis_sessions_duration_s": 0.001099,"redis_sessions_read_bytes": 243,"redis_sessions_write_bytes": 1081,"redis_shared_state_calls": 2,"redis_shared_state_duration_s": 0.000442,"redis_shared_state_write_bytes": 157,"db_count": 31,"db_write_count": 0,"db_cached_count": 2,"db_replica_count": 0,"db_primary_count": 31,"db_main_count": 26,"db_ci_count": 5,"db_main_replica_count": 0,"db_ci_replica_count": 0,"db_replica_cached_count": 0,"db_primary_cached_count": 2,"db_main_cached_count": 1,"db_ci_cached_count": 1,"db_main_replica_cached_count": 0,"db_ci_replica_cached_count": 0,"db_replica_wal_count": 0,"db_primary_wal_count": 0,"db_main_wal_count": 0,"db_ci_wal_count": 0,"db_main_replica_wal_count": 0,"db_ci_replica_wal_count": 0,"db_replica_wal_cached_count": 0,"db_primary_wal_cached_count": 0,"db_main_wal_cached_count": 0,"db_ci_wal_cached_count": 0,"db_main_replica_wal_cached_count": 0,"db_ci_replica_wal_cached_count": 0,"db_replica_duration_s": 0.0,"db_primary_duration_s": 0.018,"db_main_duration_s": 0.017,"db_ci_duration_s": 0.001,"db_main_replica_duration_s": 0.0,"db_ci_replica_duration_s": 0.0,"cpu_s": 0.116406,"mem_objects": 70437,"mem_bytes": 4655296,"mem_mallocs": 16126,"mem_total_bytes": 7472776,"pid": 4440,"worker_id": "puma_0","rate_limiting_gates": [],"exception.class": "ActionView::Template::Error","exception.message": "","exception.backtrace": ["lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'","app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'","app/models/concerns/token_authenticatable_strategies/base.rb:50:in `ensure_token!'","app/models/concerns/token_authenticatable.rb:54:in `block in add_authentication_token_field'","app/models/project.rb:2197:in `runners_token'","app/views/projects/runners/_project_runners.html.haml:11","app/views/projects/runners/_settings.html.haml:5","app/views/projects/settings/ci_cd/show.html.haml:49","app/controllers/application_controller.rb:144:in `render'","app/controllers/projects/settings/ci_cd_controller.rb:38:in `show'","app/controllers/application_controller.rb:500:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:491:in `set_session_storage'","lib/gitlab/i18n.rb:107:in `with_locale'","lib/gitlab/i18n.rb:113:in `with_user_locale'","app/controllers/application_controller.rb:482:in `set_locale'","app/controllers/application_controller.rb:475:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"exception.cause_class": "OpenSSL::Cipher::CipherError","db_duration_s": 0.01585,"view_duration_s": 0.0,"duration_s": 0.11644}

从日志回溯链可以明确，错误核心是OpenSSL::Cipher::CipherError，发生在尝试解密项目的runners_token时。这通常意味着：

• 数据库中存储的runners_token加密值损坏
• GitLab的加密密钥（gitlab-secrets.json）发生变更，导致无法解密旧的加密token
• 数据库迁移或升级过程中出现异常，导致token数据异常

方案1：重置单个项目的runners_token #

通过GitLab Rails控制台重置目标项目的runners_token：

1. 进入GitLab Rails控制台：

sudo gitlab-rails console

2. 查找目标项目：

project = Project.find_by_full_path('android/sade')

3. 重置runners_token：

project.reset_runners_token!
project.save!

4. 退出控制台：

exit

完成后刷新CI/CD设置页面，验证是否恢复正常。

方案2：批量重置所有项目的runners_token（如果多个项目出现此问题） #

如果多个项目都触发该错误，可批量重置：

sudo gitlab-rails console
Project.find_each do |p|
  p.reset_runners_token!
  p.save!
end
exit

方案3：检查并恢复加密密钥 #

如果是gitlab-secrets.json丢失或变更导致：

1. 查找是否有gitlab-secrets.json的备份（通常GitLab升级或安装时会自动备份）
2. 将备份的gitlab-secrets.json替换当前文件（默认路径：/etc/gitlab/gitlab-secrets.json）
3. 重启GitLab服务：

sudo gitlab-ctl restart

内容的提问来源于stack exchange，提问作者Hajitsu