Tornado+MySQL+Redis架构WebSocket服务性能瓶颈排查与优化

First off, let’s break down the common culprits in your setup (Tornado + tormysql + tornadis on AWS m5.large) that could be causing latency to jump once you exceed 25 connections, along with actionable fixes.

Potential Code Issues to Check #

1. Blocking I/O Operations Stalling the Event Loop #

Tornado’s power comes from its single-threaded asynchronous event loop—if any operation blocks this loop, all pending connections will queue up, leading to massive latency. Here’s what to look for:

• Are you using synchronous DB/Redis calls? For example, if you’re calling tormysql’s synchronous query methods (not using await or yield) or tornadis’s call() (sync) instead of call_async() (async), you’re blocking the loop.
• Did you accidentally use run_sync() in your handler? This method will block the event loop until the sync task completes, which is deadly for concurrent connections.

2. Under-Sized Connection Pools #

If your MySQL/Redis connection pools are too small, requests will spend most of their time waiting for a free connection instead of processing data:

• tormysql Connection Pool: By default, tormysql’s pool might have a low max_connections (e.g., 10). When 25+ WebSocket connections all try to hit the DB at once, most will queue for a connection.
• tornadis Connection Pool: If you’re creating a new Redis client per request instead of reusing a pool, the overhead of establishing new connections will add up quickly.

3. Blocking Logic in WebSocket Handlers #

If your on_message method does heavy computation, parsing, or any other synchronous work that takes more than a few milliseconds, it’ll block the event loop. Even small delays multiply when you have dozens of connections.

Fixes to Reduce Latency #

1. Enforce Fully Asynchronous I/O #

Double-check all your DB and Redis interactions are non-blocking:

• For tormysql: Use the async connection pool with await (Python 3.5+) or yield (older versions) for all queries:

  async def fetch_data(self, query):
      async with self.db_pool.Connection() as conn:
          async with conn.cursor() as cursor:
              await cursor.execute(query)
              return await cursor.fetchall()
  

• For tornadis: Use Client with call_async() and ensure you’re awaiting the result:

  async def get_redis_value(self, key):
      result = await self.redis_client.call_async("GET", key)
      return result
  

2. Tune Connection Pool Sizes #

Adjust your pool sizes to match your concurrency needs:

• MySQL: For your m5.large instance (2vCPU, 8GB RAM), set max_connections to 30-50. Each MySQL connection uses ~5-10MB RAM, so 50 connections would use ~500MB—well within your 8GB limit. Update your tormysql pool config:

  db_pool = tormysql.ConnectionPool(
      max_connections=30,
      host="your-db-host",
      user="your-user",
      password="your-pass",
      database="your-db"
  )
  

• Redis: Tornadis connections are lightweight—set max_connections to 100+ to avoid connection contention:

  from tornadis import Client, ConnectionPool
  redis_pool = ConnectionPool(max_connections=100, host="your-redis-host")
  redis_client = Client(connection_pool=redis_pool)
  

3. Offload Blocking Work to Thread Pools #

If you have unavoidable synchronous work (e.g., heavy computation), use Tornado’s thread pool to run it without blocking the event loop:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

async def on_message(self, message):
    # Offload heavy work to the thread pool
    result = await self.io_loop.run_in_executor(executor, self.process_message, message)
    await self.write_message(result)

def process_message(self, message):
    # Synchronous heavy logic here
    return processed_data

4. Optimize Tornado’s Event Loop #

• Switch to uvloop: For Python 3, uvloop is a faster drop-in replacement for the default asyncio event loop. Install it with pip install uvloop, then add this at the start of your app:

  import uvloop
  import tornado.platform.asyncio
  
  tornado.platform.asyncio.AsyncIOMainLoop().install()
  uvloop.install()
  

• Use Multi-Process Mode: Tornado is single-threaded by default—leverage your m5.large’s 2vCPUs by running multiple worker processes. Add this to your app startup:

  import tornado.process
  
  if __name__ == "__main__":
      tornado.process.fork_processes(2)  # Match number of vCPUs
      app.listen(8888)
      tornado.ioloop.IOLoop.current().start()
  

5. Database and Cache Optimization #

• Check MySQL Slow Queries: Enable slow query logs to find and optimize slow SQL (e.g., add missing indexes, rewrite inefficient queries). Even a query that takes 100ms will cause chaos at 25+ concurrent requests.
• Cache Aggressively: Use Redis to cache frequent DB queries so you don’t hit the database for every request. For example, cache user profiles or frequent lookup data with appropriate TTLs.

6. Deployment Tweaks #

• Add Nginx as a Reverse Proxy: Nginx can handle WebSocket load balancing, connection pooling, and SSL termination, freeing up Tornado to focus on application logic. Configure Nginx to forward WebSocket traffic to your Tornado workers.

内容的提问来源于stack exchange，提问作者aja