如何使用Python通过Websocket抓取Keepa平台的品牌自动补全建议？

我尝试从 https://keepa.com/#!finder 抓取品牌建议内容：例如在「brand」输入框中键入「ni」时，会出现所有以「ni」开头的品牌的建议列表，如「nike」。
经确认该功能由websocket会话实现：按下按键时会向会话发送gzip压缩的JSON字符串，收到的响应中的高亮项为gzip压缩字符串。
但使用Python复现该逻辑时失败，所用代码如下：

from websocket import create_connection
import zlib
import json

websocket_resource_url = 'wss://dyn.keepa.com/apps/cloud/?app=keepaWebsite&version=1.6'

ws = create_connection(websocket_resource_url)
msg = {"path":"pro/autocomplete","content":"nike","domainId":1,"maxResults":25,"type":"brand","version":3,"id":111111,"user":"user"}

ws.send(zlib.compress((json.dumps(msg)+'\n').encode('utf-8')))

print('Result: {}'.format(zlib.decompress(ws.recv())))

运行后返回{"status":108, "n":73}，但期望的输出类似如下：

{"startTimeStamp":1637576348774,"id":7596,"timeStamp":1637576348774,"status":200,"version":3,"lastUpdate":0,"suggestions":[{"value":"nike","count":1100713},{"value":"nike golf","count":7497},{"value":"nikeelando","count":358},{"value":"nikea","count":195},{"value":"nike sb","count":87}]}

问题原因和修复方案 #

• 状态码108代表请求参数校验失败，你当前的zlib压缩使用了默认参数，Keepa的接口要求使用不带gzip头的原始deflate压缩，而Python默认zlib.compress生成的是带zlib头的压缩数据，和接口要求不匹配。
• 缺少连接建立后的握手步骤，Keepa的websocket连接建立后，需要先收到服务端的初始化响应后再发送业务请求，不能直接建立连接就发数据。
• 请求参数里的id字段不能是固定值，需要和后续响应的id对应，user字段也需要替换为实际的访客唯一标识，不能填固定的user字符串。

修复后的示例代码：

from websocket import create_connection
import zlib
import json

websocket_resource_url = 'wss://dyn.keepa.com/apps/cloud/?app=keepaWebsite&version=1.6'
# 初始化不带zlib头的压缩/解压缩器
compressor = zlib.compressobj(wbits=-15)
decompressor = zlib.decompressobj(wbits=-15)

ws = create_connection(websocket_resource_url)
# 先接收服务端握手消息
handshake = decompressor.decompress(ws.recv())
print("握手响应：", handshake)

msg = {"path":"pro/autocomplete","content":"nike","domainId":1,"maxResults":25,"type":"brand","version":3,"id":1,"user":""}
compressed_msg = compressor.compress((json.dumps(msg)+'\n').encode('utf-8')) + compressor.flush()
ws.send(compressed_msg)

resp = decompressor.decompress(ws.recv())
print('Result: {}'.format(resp))
ws.close()

内容的提问来源于stack exchange，提问作者sevatster