如何为EC2上的Rails应用搭建专用Sidekiq服务器？

Hey there! Let's break down exactly how to turn one of your EC2 instances into a dedicated Sidekiq server—since you already have a working Rails setup with Nginx/Capistrano and Sidekiq/Redis, this just needs a few targeted config tweaks.

First, you need to tell Capistrano which server handles web traffic and which is dedicated to Sidekiq. Update your Capistrano deployment config (usually config/deploy/production.rb or config/deploy.rb) to split the roles:

# Before (both servers handle web + Sidekiq)
# server 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com', user: 'deploy', roles: %w[web app db sidekiq]
# server 'ec2-yy-yy-yy-yy.compute-1.amazonaws.com', user: 'deploy', roles: %w[web app db sidekiq]

# After splitting roles
# Instance 1: Web server (handles Rails requests + Nginx)
server 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com', user: 'deploy', roles: %w[web app db]
# Instance 2: Dedicated Sidekiq server
server 'ec2-yy-yy-yy-yy.compute-1.amazonaws.com', user: 'deploy', roles: %w[sidekiq]

Next, ensure your Capistrano Sidekiq integration (like the capistrano-sidekiq gem) only runs Sidekiq tasks on the dedicated server. Add this to config/deploy.rb:

set :sidekiq_roles, :sidekiq

Since this instance won't handle web traffic anymore, stop and disable Nginx and your Rails web server (Puma, Unicorn, etc.) to free up resources:

# Stop running services
sudo systemctl stop nginx
sudo systemctl stop puma # Replace with your web server (e.g., unicorn)

# Disable auto-start on boot
sudo systemctl disable nginx
sudo systemctl disable puma

To prevent Capistrano from trying to restart web processes on the Sidekiq server during deployments, add a conditional skip to config/deploy.rb:

before 'deploy:restart', :skip_web_restart_on_sidekiq do
  on roles(:sidekiq) do
    execute :sudo, :systemctl, :stop, 'puma' # Match your web server service name
  end
end

You already set max_pool_size: 50 for database connections, but since this server only runs Sidekiq, you can likely increase the concurrency limit (take advantage of that 50GB memory!).

Update your config/sidekiq.yml to adjust concurrency (start with a conservative value like 75, then test to avoid OOM errors):

:concurrency: 75 # Adjust based on your task memory usage (each worker uses ~100-200MB)
:max_pool_size: 50 # Keep or adjust to match your database's connection limit
:queues:
  - default
  - [critical, 2] # Keep your existing queue priorities if you have them

If you use systemd to manage Sidekiq, double-check your service file (e.g., /etc/systemd/system/sidekiq.service) to ensure it uses the updated config.

Make sure your dedicated Sidekiq server can connect to Redis, and lock down Redis for security:

1. On your Redis server (either instance), edit /etc/redis/redis.conf:

# Bind to all IPs (or specifically the Sidekiq server's IP for tighter security)
bind 0.0.0.0
# Set a strong password to prevent unauthorized access
requirepass "your-secure-redis-password"

2. Restart Redis to apply changes:

sudo systemctl restart redis-server

3. Update your Rails app's Sidekiq initializer (config/initializers/sidekiq.rb) to use the authenticated Redis connection:

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://:your-secure-redis-password@your-redis-server-ip:6379/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://:your-secure-redis-password@your-redis-server-ip:6379/0' }
end

Run your Capistrano deployment to apply all changes:

cap production deploy

Then verify everything works:

• On the dedicated Sidekiq server, check if Sidekiq is running:

  sudo systemctl status sidekiq
  # Or use ps to list processes
  ps aux | grep sidekiq
  

• On your web server, confirm Nginx and Rails are handling requests normally.
• Submit a test background job and check if it's processed by the dedicated Sidekiq server (use the Sidekiq web UI or logs at log/sidekiq.log).

Since this is a dedicated Sidekiq server, keep an eye on resource usage to avoid overloading:

• Use htop to real-time monitor CPU and memory.
• Enable the Sidekiq web UI (restrict access to your web server or add authentication) to track queue lengths and worker status.
• For long-term monitoring, set up tools like Prometheus + Grafana to track Sidekiq metrics and server health.

内容的提问来源于stack exchange，提问作者user2256233