使用Python抓取socket.io数据异常：无法接收BTC交易平台买卖报价

I'm trying to scrape the latest bid/ask prices from http://btc-exchange.com/, and I've identified that the price data is provided via a Socket.IO WebSocket interface at wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid=XXX. The session ID (SID) is generated by sending a request to https://pusher.mistertango.com/socket.io/?EIO=3&transport=polling&t=1517079662330-10.

Here's my current code:

import requests
from websocket import create_connection
import json

SID_url = "https://pusher.mistertango.com/socket.io/?EIO=3&transport=polling"
SID_req = requests.get(SID_url, headers={'User-Agent': 'Mozilla/5.0'}).text
SID = SID_req[SID_req.index("sid")+6:SID_req.index(",")-1]
print(SID_req)
print(SID)

ws = create_connection("wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid="+SID)
ws.send('2probe')
print(ws.recv())
ws.send('5')
print(ws.recv())
ws.send('42["subscribe",{"chan":"market-e559906eda4362f58bcaab40a4bfb5b4"}]')
while True:
    result = ws.recv()
    print(result)
    ws.close()

When I run this code, the output is:

ÿ0{"sid":"mURV8OnaNqax_AmvAAF2","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}
mURV8OnaNqax_AmvAAF2
3probe
40

I modeled the messages after what I saw in Chrome DevTools, but after receiving '40', the connection stops receiving any further data. Where am I going wrong?

Solution & Fixes #

Let's break down the issues in your code and fix them step by step:

1. Incorrect Connection Closure Timing #

The most obvious mistake is calling ws.close() immediately inside your while True loop. This closes the connection as soon as you receive the first message, preventing you from getting any subsequent price updates. Move the close call to only run when you intentionally want to terminate the connection (like on a keyboard interrupt).

2. Incomplete Socket.IO Handshake & Ping/Pong Handling #

Socket.IO v2 (which uses EIO=3) requires proper handshake and active ping/pong maintenance to keep the connection alive. The '40' response you're getting is a confirmation of the WebSocket upgrade, but you need to handle the server's periodic ping messages to avoid being disconnected.

3. Fragile SID Parsing #

Your current method of extracting the SID via string indices is fragile—if the response format changes slightly, this will break. Instead, strip the Socket.IO frame prefix and parse the response as JSON to reliably get the SID.

Fixed Code Example #

import requests
from websocket import create_connection
import json
import time

def get_sid():
    SID_url = "https://pusher.mistertango.com/socket.io/?EIO=3&transport=polling"
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(SID_url, headers=headers)
    
    # Strip Socket.IO frame prefix (the leading non-JSON characters)
    json_start = response.text.find('{')
    if json_start == -1:
        raise ValueError("Failed to parse SID response")
    
    sid_data = json.loads(response.text[json_start:])
    return sid_data['sid']

def main():
    try:
        SID = get_sid()
        print(f"Successfully retrieved SID: {SID}")
        
        ws_url = f"wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid={SID}"
        ws = create_connection(ws_url)
        print("WebSocket connection established")
        
        # Complete Socket.IO handshake
        ws.send('2probe')
        print(f"Received: {ws.recv()}")  # Should get '3probe'
        
        ws.send('5')
        print(f"Received: {ws.recv()}")  # Should get '40' (upgrade confirmation)
        
        # Subscribe to the market data channel
        subscribe_msg = '42["subscribe",{"chan":"market-e559906eda4362f58bcaab40a4bfb5b4"}]'
        ws.send(subscribe_msg)
        print("Sent subscription request")
        
        # Handle incoming messages and maintain connection
        while True:
            result = ws.recv()
            print(f"\nRaw message: {result}")
            
            # Respond to server ping (keep connection alive)
            if result == '2':
                ws.send('3')
                print("Sent pong to server")
            
            # Parse business data messages (prefix '42' indicates JSON payload)
            elif result.startswith('42'):
                payload = json.loads(result[2:])
                print(f"Parsed market data: {payload}")
                
            time.sleep(0.1)
            
    except KeyboardInterrupt:
        print("\nClosing connection...")
        ws.close()
    except Exception as e:
        print(f"Error occurred: {str(e)}")

if __name__ == "__main__":
    main()

Key Notes #

• Ping/Pong Maintenance: The server sends a ping ('2') every 25 seconds (as indicated in the SID response's pingInterval). You must respond with a pong ('3') to avoid being disconnected.
• Message Prefixes: Socket.IO uses numeric prefixes to indicate message types:
  • 2: Ping
  • 3: Pong
  • 42: JSON-formatted business data
• Request Headers: If you still encounter issues, try adding additional headers (like Origin or Cookie) copied directly from Chrome DevTools to fully mimic a browser's request behavior.

内容的提问来源于stack exchange，提问作者arjun sachdeva