批量POST商品至WooCommerce API并发超时及Pending状态求助

我需要将数百万条商品数据通过REST API POST至WooCommerce网站，尝试了两种方案：

方案1：单循环逐个请求 #

from woocommerce import API

wcapi = API(url=url, consumer_key=consumer_key, consumer_secret=consumer_secret,timeout=50)

for product in my_json_list:
    print(wcapi.post("products", product).json())

该方案可成功执行，但因商品数据量巨大，效率极低。

方案2：线程池并发请求 #

def post_data(product):
    return wcapi.post("products", product).json()

# protect the entry point
if __name__ == '__main__':
    # create the thread pool
    with ThreadPoolExecutor() as ex:
        # issue many asynchronous tasks systematically
        futures = [ex.submit(post_data, page) for page in data]
        # enumerate futures and report results
        for future in futures:
            print(future.result())

执行该方案时出现ReadTimeoutError（read timeout=50），且Future对象状态显示为<Future at 0x124d1f7d0 state=pending>，完整错误栈如下：

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1278, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1134, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='new.turbo-diesel.co.uk', port=443): Read timed out. (read timeout=50)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/martinhewing/Downloads/Python_Code/Woo_web_pages/create_DEP_pages.py", line 74, in <module>
    futures = [ex.submit(wcapi.post("products", page).json()) for page in data]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martinhewing/Downloads/Python_Code/Woo_web_pages/create_DEP_pages.py", line 74, in <listcomp>
    futures = [ex.submit(wcapi.post("products", page).json()) for page in data]
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/woocommerce/api.py", line 110, in post
    return self.__request("POST", endpoint, data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/woocommerce/api.py", line 92, in __request
    return request(
           ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/adapters.py", line 578, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='new.turbo-diesel.co.uk', port=443): Read timed out. (read timeout=50)

问题原因 #

1. 无限制并发过载服务器：默认ThreadPoolExecutor会创建大量线程，瞬间发起数万请求，WooCommerce服务器资源耗尽，无法及时响应，导致请求超时。
2. API实例非线程安全：多线程共享同一个wcapi实例，底层的requests会话连接池会出现竞争冲突，引发连接异常。
3. 超时时间设置不足：高负载下服务器响应延迟远超50秒，直接触发超时。

解决方法 #

1. 限制线程池大小 #

根据服务器性能设置合理的并发数（建议10-20），避免瞬间压垮服务器：

with ThreadPoolExecutor(max_workers=10) as ex:

2. 每个线程独立创建API实例 #

避免共享会话，在任务函数内部初始化API：

def post_data(product, url, consumer_key, consumer_secret):
    wcapi = API(url=url, consumer_key=consumer_key, consumer_secret=consumer_secret, timeout=120)
    try:
        return wcapi.post("products", product).json()
    except Exception as e:
        return {"error": str(e), "product_sku": product.get("sku")}

if __name__ == '__main__':
    with ThreadPoolExecutor(max_workers=10) as ex:
        futures = [ex.submit(post_data, page, url, consumer_key, consumer_secret) for page in data]

3. 添加重试机制+延长超时时间 #

针对临时超时和服务器错误添加自动重试，同时延长超时时间：

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def post_data(product, url, consumer_key, consumer_secret):
    wcapi = API(url=url, consumer_key=consumer_key, consumer_secret=consumer_secret, timeout=120)
    # 配置重试策略
    retry_strategy = Retry(
        total=3,
        backoff_factor=1,  # 重试间隔递增：1s,2s,4s
        status_forcelist=[429, 500, 502, 503, 504]
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    wcapi.session.mount("https://", adapter)
    wcapi.session.mount("http://", adapter)
    
    try:
        return wcapi.post("products", product).json()
    except Exception as e:
        return {"error": str(e), "product_sku": product.get("sku")}

4. 使用WooCommerce批量API（最优解） #

WooCommerce支持批量创建商品，一次请求可提交多个商品，直接减少90%以上的请求量：

# 按50个商品一组拆分数据（可根据服务器限制调整）
batch_size = 50
batches = [data[i:i+batch_size] for i in range(0, len(data), batch_size)]

wcapi = API(url=url, consumer_key=consumer_key, consumer_secret=consumer_secret, timeout=120)

for batch in batches:
    payload = {"create": batch}
    response = wcapi.post("products/batch", payload).json()
    print(response)

5. 优化Future结果处理 #

用as_completed异步获取结果，同时记录失败的商品以便后续重试：

from concurrent.futures import as_completed

if __name__ == '__main__':
    failed_items = []
    with ThreadPoolExecutor(max_workers=10) as ex:
        # 绑定商品与future对象，方便追踪失败项
        futures = {ex.submit(post_data, page, url, consumer_key, consumer_secret): page for page in data}
        for future in as_completed(futures):
            product = futures[future]
            try:
                result = future.result()
                print(result)
            except Exception as e:
                print(f"商品{product.get('sku')}提交失败: {e}")
                failed_items.append(product)
    
    # 重试失败商品
    if failed_items:
        print(f"重试{len(failed_items)}个失败商品...")
        with ThreadPoolExecutor(max_workers=5) as ex:
            for future in as_completed([ex.submit(post_data, item, url, consumer_key, consumer_secret) for item in failed_items]):
                print(future.result())

内容的提问来源于stack exchange，提问作者Martin H