如何在Python 2.7中实现httplib连接的全局持久性？

Hey there, let's tackle this Keep-Alive timeout issue with httplib in Python 2.7—totally get why those TCP handshake delays are a killer for your use case. Here are a few practical, actionable approaches you can try:

1. Send Heartbeat Requests to Keep Connections Alive #

Since your server has a super short timeout (<30s), you can proactively send lightweight requests to "refresh" the connection's timeout timer before it gets dropped. A HEAD request to the root path (or a dedicated health endpoint if available) works perfectly here—it's fast and doesn't transfer unnecessary data.

Here's a wrapped HTTPConnection class that handles this automatically:

import httplib
import time

class HeartbeatConnection(httplib.HTTPConnection):
    def __init__(self, host, port=None, timeout=30):
        super(HeartbeatConnection, self).__init__(host, port, timeout)
        self.last_req_time = time.time()
        # Set this to 2-5s less than the server's timeout to leave buffer
        self.server_timeout_threshold = 25

    def _send_heartbeat(self):
        # Send a lightweight HEAD request to keep the connection alive
        self.request("HEAD", "/")
        resp = self.getresponse()
        resp.read()  # Critical: must read the full response to free the connection
        self.last_req_time = time.time()

    def request(self, method, url, body=None, headers={}):
        # Check if we're approaching the server's timeout—send heartbeat if needed
        if time.time() - self.last_req_time > self.server_timeout_threshold:
            self._send_heartbeat()
        # Add explicit Keep-Alive header to reinforce the connection
        headers.setdefault("Connection", "Keep-Alive")
        super(HeartbeatConnection, self).request(method, url, body, headers)
        self.last_req_time = time.time()

2. Gracefully Handle Dropped Connections Without Full Reconnects #

Even with heartbeats, there's a chance the server drops the connection unexpectedly. Instead of blindly creating a new connection every time, you can check if the existing connection is still valid and only reconnect when necessary:

def safe_request(conn, method, url, body=None, headers={}):
    try:
        # First attempt to use the existing connection
        conn.request(method, url, body, headers)
        return conn.getresponse()
    except (httplib.NotConnected, httplib.ResponseNotReady):
        # Connection was dropped—reconnect and retry the request
        conn.connect()
        conn.request(method, url, body, headers)
        return conn.getresponse()

This way, you only pay the TCP handshake cost when the connection is actually dead, not on every request.

3. Build a Simple Connection Pool for Reuse #

If you're making multiple requests across different parts of your code, a connection pool can help manage active connections and automatically clean up timed-out ones:

import httplib
import time
from collections import OrderedDict

class ConnectionPool:
    def __init__(self, host, port=None, max_conns=5, server_timeout=25):
        self.host = host
        self.port = port
        self.max_conns = max_conns
        self.server_timeout = server_timeout
        # Use OrderedDict to track least-recently-used connections
        self.pool = OrderedDict()

    def get_conn(self):
        # Clean up connections that have timed out
        now = time.time()
        stale_conns = []
        for conn_id, (conn, last_used) in self.pool.items():
            if now - last_used > self.server_timeout:
                stale_conns.append(conn_id)
        
        for conn_id in stale_conns:
            del self.pool[conn_id]

        # Reuse an existing connection if available
        if self.pool:
            conn_id, (conn, _) = self.pool.popitem(last=False)
            self.pool[conn_id] = (conn, now)
            return conn
        # Create a new connection if pool is empty
        conn = httplib.HTTPConnection(self.host, self.port)
        conn.connect()
        self.pool[id(conn)] = (conn, now)
        return conn

    def release_conn(self, conn):
        # Mark connection as recently used and add back to pool
        self.pool[id(conn)] = (conn, time.time())
        # Trim pool if we exceed max connections
        while len(self.pool) > self.max_conns:
            self.pool.popitem(last=True)

Quick Notes to Keep in Mind #

• Always make sure to read the full response from the server (even for heartbeats)—if you don't, the connection will get stuck waiting for unread data.
• Some servers might block HEAD requests—if that's the case, swap it for a lightweight GET request to a minimal endpoint (like /health if your server has one).
• httplib in Python 2.7 defaults to HTTP/1.1 (which uses Keep-Alive by default), but adding the explicit Connection: Keep-Alive header can help reinforce this with finicky servers.

内容的提问来源于stack exchange，提问作者ajsp