如何调试async aiohttp运行缓慢问题？Ubuntu物理机与VM性能差异排查

Hey there, that's a really weird discrepancy between your physical Ubuntu machine and the VM—let's break down what could be causing this slowdown with your asyncio/aiohttp script, and how to debug it step by step.

First, Let's Rule Out Common Asyncio/Linux Gotchas #

1. Check the Event Loop Implementation #

Python 3.7 on Linux defaults to the SelectorEventLoop, which is functional but not the fastest option. Your VM might be using a different setup, or maybe you have an optimized loop installed there without realizing it.

Add this line right after getting the event loop to see what you're working with:

loop = asyncio.get_event_loop()
print(f"Current event loop: {type(loop).__name__}")

If it's just SelectorEventLoop, try installing uvloop (a faster, drop-in replacement for asyncio's default loop) and see if that fixes the slowdown:

pip install uvloop

Then add this at the top of your script:

import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

This often gives a huge performance boost on Linux systems.

2. Fix the SSL Context Handling #

Your code explicitly passes ssl=ssl.SSLContext() to each session.get() call. This might be forcing a new SSL handshake setup for every request, which is slow—especially if your physical machine has older SSL libraries.

Try removing that parameter entirely and let aiohttp handle SSL automatically:

# Before
async with session.get(url, params={'id': app_id}, ssl=ssl.SSLContext()) as response:

# After
async with session.get(url, params={'id': app_id}) as response:

Also, check the OpenSSL version on both your physical machine and VM:

# Run this in both environments
python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
openssl version

If your physical machine has an older OpenSSL version, upgrading it might help with SSL handshake speed.

3. Check DNS Resolution #

Asyncio in Python 3.7 uses synchronous DNS resolution by default, which can bottleneck parallel requests if DNS is slow on your physical machine.

Test DNS speed on both systems:

time nslookup play.google.com

If the physical machine takes significantly longer, try enabling DNS caching in aiohttp's TCPConnector:

async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(use_dns_cache=True)) as session:

This will cache DNS results so you don't resolve the same URL multiple times.

4. File Descriptor Limits #

Linux has a default limit on open file descriptors, and asyncio needs one per active connection. If your physical machine has a low limit, it could be throttling your parallel requests.

Check the current limit:

ulimit -n

If it's 1024 or lower, try temporarily raising it and re-running your script:

ulimit -n 2048
python3 your_script.py

If this fixes the slowdown, you can make the change permanent by editing /etc/security/limits.conf.

Decoding Your Performance Profile #

Since you have a performance profile but don't know how to read it, let's focus on the key metrics you need to look for:

If you used cProfile, run this to get a sorted view of the slowest functions:

import pstats
stats = pstats.Stats('your_profile_file.stats')
# Sort by cumulative time (time spent in the function + all its subfunctions)
stats.sort_stats('cumulative').print_stats(20)

Look for these red flags:

• High cumtime for socket.getaddrinfo: This means DNS resolution is the bottleneck.
• High cumtime for SSL-related functions (like ssl.wrap_socket): SSL handshakes are slow—go back to the SSL context fix above.
• High cumtime for asyncio.sleep or asyncio.wait_for: Your event loop is being blocked somewhere (maybe a synchronous function you didn't notice).
• If fetch has a huge cumtime but most of it is spent in response.text(): That could be network latency, but you said you ruled out network issues—so maybe it's a system-level network bottleneck on the physical machine.

Quick Code Tweaks to Test #

Here's a modified version of your script with the suggested fixes applied, to make testing easier:

import aiohttp
import asyncio
import ssl
import time
from bs4 import BeautifulSoup

# Add uvloop if installed
try:
    import uvloop
    asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
except ImportError:
    print("uvloop not installed, using default event loop")

async def get_app_updated_date(html_content):
    soup = BeautifulSoup(html_content, 'lxml')
    section_titles_divs = [x for x in soup.select('div.hAyfc div.BgcNfc')]
    title_normalization = {
        'Updated': 'updated',
    }
    data = {
        'updated': None,
    }
    for title_div in section_titles_divs:
        section_title = title_div.string
        if section_title in title_normalization:
            title_key = title_normalization[section_title]
            value_div = title_div.next_sibling.select_one('span.htlgb')
            value = value_div.text
            data[title_key] = value
    return data

async def fetch(session, url, app_id):
    print(f'Fetching information for {app_id}')
    # Removed custom SSL context
    async with session.get(url, params={'id': app_id}) as response:
        html_resp = await response.text()
        app_lastupdated_date = await get_app_updated_date(html_resp)
        return {app_id: app_lastupdated_date}

async def main():
    url = 'https://play.google.com/store/apps/details'
    app_list = ['com.google.android.youtube', 'com.whatsapp', 'com.instagram.android', 'com.google.android.apps.maps', 'com.kiloo.subwaysurf', 'com.halfbrick.fruitninjafree', 'com.adobe.reader', 'org.mozilla.firefox', 'com.zeptolab.ctr.ads', 'com.fingersoft.hillclimb']
    # Added DNS caching to connector
    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(use_dns_cache=True)) as session:
        url_requests = [fetch(session, url, app_id) for app_id in app_list]
        print(url_requests)
        results = await asyncio.gather(*url_requests)
        for r in results:
            print(r)
        print(f'Result size = {len(results)}')

if __name__ == '__main__':
    start_time = time.time()
    loop = asyncio.get_event_loop()
    print(f"Using event loop: {type(loop).__name__}")
    loop.run_until_complete(main())
    print(f'Script execution completed in: {time.time() - start_time} seconds')

Try running this version and see if the performance improves. If it does, you can narrow down which fix was the key one by removing them one by one.

Let me know what you find—happy to help dig deeper if needed!

内容的提问来源于stack exchange，提问作者chidori