Heroku休眠重启Rails应用Sidekiq无法连接RedisCloud故障咨询
问题描述
有一个在Heroku上休眠近一年的Rails应用,重启时发现原RedisCloud数据库已被平台回收。新创建RedisCloud数据库后,对应的REDISCLOUD_URL格式为redis://default:(key)@redis-(path),但部署后Sidekiq无法连接该Redis实例。与应用绑定的Redislabs.com后台提示「This subscription does not contain any databases」。
待确认的疑问点:
- 是否需要对RedisCloud数据库做额外的启动/初始化操作?
- URL中的
default:前缀是否会触发ACL授权缺失问题? - 故障是否由使用的v4版本redis gem导致?
环境信息
Gemfile片段
ruby '2.6.5' gem 'rails', '~> 6.0.2', '>= 6.0.2.1' gem 'redis', '~> 4.0' gem 'sidekiq' gem 'sidekiq-failures', '~> 1.0'
应用休眠前运行完全正常,未做版本变更,仍使用heroku-18栈
报错日志
Heroku生产环境Sidekiq报错
2021-12-03T11:25:25.593016+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:230:in `process' 2021-12-03T11:25:25.593016+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:125:in `call' 2021-12-03T11:25:25.593016+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:286:in `block in info' 2021-12-03T11:25:25.593017+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize' 2021-12-03T11:25:25.593017+00:00 app[worker.1]: /app/vendor/ruby-2.6.5/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' 2021-12-03T11:25:25.593017+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize' 2021-12-03T11:25:25.593018+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:285:in `info' 2021-12-03T11:25:25.593018+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/lib/sidekiq.rb:118:in `block in redis_info' 2021-12-03T11:25:25.593018+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/lib/sidekiq.rb:97:in `block in redis' 2021-12-03T11:25:25.593019+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with' 2021-12-03T11:25:25.593019+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt' 2021-12-03T11:25:25.593019+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with' 2021-12-03T11:25:25.593019+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt' 2021-12-03T11:25:25.593020+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with' 2021-12-03T11:25:25.593020+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/lib/sidekiq.rb:94:in `redis' 2021-12-03T11:25:25.593020+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/lib/sidekiq.rb:112:in `redis_info' 2021-12-03T11:25:25.593020+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/lib/sidekiq/cli.rb:61:in `run' 2021-12-03T11:25:25.593020+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.0.6/bin/sidekiq:12:in `<top (required)>' 2021-12-03T11:25:25.593021+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `load' 2021-12-03T11:25:25.593021+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `<top (required)>' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `load' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `kernel_load' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:28:in `run' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli.rb:465:in `exec' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' 2021-12-03T11:25:25.593022+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli.rb:27:in `dispatch' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/cli.rb:18:in `start' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/exe/bundle:30:in `block in <top (required)>' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' 2021-12-03T11:25:25.593023+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bundler-2.0.2/exe/bundle:22:in `<top (required)>' 2021-12-03T11:25:25.593024+00:00 app[worker.1]: /app/bin/bundle:113:in `load' 2021-12-03T11:25:25.593024+00:00 app[worker.1]: /app/bin/bundle:113:in `<main>' 2021-12-03T11:25:25.594276+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:362:in `rescue in establish_connection': Error connecting to Redis on redis-11993.c1.eu-west-1-3.ec2.cloud.redislabs.com:11993 (SocketError) (Redis::CannotConnectError)
本地开发环境启动报错
rails s Started GET "/" for 127.0.0.1 at 2021-12-04 16:53:56 +0100 PG::ConnectionBad (could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"? ):
相关配置
redis相关依赖版本
# Gemfile gem 'redis', '~> 4.0' # Gemfile.lock redis (4.1.3)
config/cable.yml
development: adapter: redis url: <%= ENV.fetch("REDISCLOUD_URL") { "redis://localhost:6379/1" } %> channel_prefix: _wadzatsong_development test: adapter: test production: adapter: redis url: <%= ENV.fetch("REDISCLOUD_URL") { "redis://localhost:6379/1" } %> channel_prefix: _wadzatsong_production
config/initializers/redis.rb
$redis = Redis.new if Rails.env=="production" url = ENV["REDISCLOUD_URL"] elsif Rails.env=="development" url = "redis://localhost:6379/1" end if url Sidekiq.configure_server do |config| config.redis = { url: url } end Sidekiq.configure_client do |config| config.redis = { url: url } end $redis = Redis.new(:url => url) end
故障排查与解决
核心报错为Redis::CannotConnectError结合SocketError,再叠加Redislabs后台提示订阅下无数据库,问题按优先级排序解决即可:
- 先处理Redislabs订阅与数据库绑定问题
确认新创建的RedisCloud数据库是否关联到了Heroku应用对应的订阅下。后台提示无数据库说明创建的实例没有和当前绑定的订阅关联,导致Heroku侧读取到的订阅下无有效实例,自然无法连接。不需要额外初始化操作,完成订阅与实例的绑定即可。 default:前缀和redis gem v4版本不存在兼容性问题
redis 4.1.3版本完全支持带用户名的Redis URL格式,default是Redis 6+默认的ACL用户名,格式正确不会触发授权问题,无需修改URL格式。- 修正初始化配置逻辑
当前redis.rb初始化代码存在逻辑问题:第一行的$redis = Redis.new会在URL读取完成前先初始化一个无配置的Redis实例,可能读取到错误的默认配置,建议删除该行,只保留后面根据环境变量初始化的逻辑即可。 - 本地PG报错为独立问题
本地环境的PostgreSQL服务未启动,启动本地PG服务即可解决,和生产环境Redis故障无关。
内容的提问来源于stack exchange,提问作者pierrecode
相关产品推荐
相关产品推荐

