基于libtorrent的DHT爬虫性能低下问题及优化咨询

问题背景 #

我正在用libtorrent开发DHT爬虫，目标是发现种子文件及其元数据，但爬虫运行速度极慢，处理的种子数量极少，大量种子无响应或元数据获取失败。调整过会话设置，性能还是不达标。

简化代码 #

import time
import libtorrent as lt

MAX_TORRENTS = 200  # Maximum number of simultaneous torrents

session = lt.session()

dht_bootstrap_nodes = [
    "router.utorrent.com",
    "router.bittorrent.com:6881",
    "dht.transmissionbt.com:6881",
    "dht.libtorrent.org:25401",
    "router.bitcoment.com:6881",
    "router.bt.ouinet.work:6881",
    "dht.aelitis.com:6881",
]

# Configure session settings
settings = {
    "active_downloads": MAX_TORRENTS,
    "active_seeds": MAX_TORRENTS,
    "active_limit": MAX_TORRENTS * 2,
    "enable_dht": True,
    "dht_bootstrap_nodes": ",".join(dht_bootstrap_nodes),
    "alert_mask": lt.alert.category_t.dht_notification
    | lt.alert.category_t.status_notification,
}
session.apply_settings(settings)

current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
print(f"[{current_time}] Start...")
try:
    while True:
        alerts = session.pop_alerts()
        for alert in alerts:
            current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
            if isinstance(alert, lt.dht_get_peers_alert):
                print(f"[{current_time}] Found info hash: {alert.info_hash}")
            elif isinstance(alert, lt.metadata_received_alert):
                print(
                    f"[{current_time}] Metadata received for info hash: {alert.info_hash}"
                )
        time.sleep(0.01)
except KeyboardInterrupt:
    print("Stopping the DHT crawler...")

current_time = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
print(f"[{current_time}] Stopping the DHT crawler...")

运行结果 #

[2025-03-07 08:24:21] Start...
[2025-03-07 08:24:22] Found info hash: 707de37217f78b300dea728cc84981d4df74bd79
[2025-03-07 08:24:23] Found info hash: 7168eb3468a2858449e148920820efebebaac0d3
[2025-03-07 08:24:40] Found info hash: 72b00a1d82a2432eece385c160ae0e22195bb5f9
[2025-03-07 08:24:48] Found info hash: 7365e551ff7733ac8c9196f2f63e66981fccdb14
[2025-03-07 08:24:50] Found info hash: 7365ebbaa583e61fa7a921411dca9c39e85b09ac
[2025-03-07 08:25:33] Found info hash: 7365e87e242ecf777b963852958292e95b52f1a5
[2025-03-07 08:25:45] Found info hash: 7365e5d78f817202b86b27395bd59f00340c1835
[2025-03-07 08:26:07] Found info hash: 7365e5d78f817202b86b27395bd59f00340c1835
[2025-03-07 08:26:44] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:26:45] Found info hash: 7365e514f5b40caff0e842c92b80c62004ccb156
[2025-03-07 08:26:56] Found info hash: a440faf6639806df76e15607cd71be421a6b2728
[2025-03-07 08:27:04] Found info hash: a440f18f096e3e215f4e8eacb2e9699972fa8368
[2025-03-07 08:27:28] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:27:32] Found info hash: a440f6f0fcaa3a62ffa50cc008db025e5dddb70b
[2025-03-07 08:27:45] Found info hash: 7365e4866e555f0e991cd4136df7d7d2381d696d
[2025-03-07 08:27:59] Found info hash: 7365e13c29484a0304ee35a844835fe1b81bca2f
[2025-03-07 08:28:00] Found info hash: 7365e4866e555f0e991cd4136df7d7d2381d696d
[2025-03-07 08:28:22] Found info hash: a440f5cb77ea154ebf7a5ec972061e05d3bb7e94
[2025-03-07 08:28:34] Found info hash: 0fe6bf6df3b4f0e9c214e5a69bd93562e5606351
[2025-03-07 08:28:34] Found info hash: 0fe6ce9f178bc8f0eb501a2342b670e4fd7be0b1
[2025-03-07 08:28:34] Found info hash: 0fe6e4ef11d16a71dc12d27d508e3ce8272d8a25
[2025-03-07 08:28:34] Found info hash: 0fe6f97701a9906dcdb58070f7e24b5f353c0066
[2025-03-07 08:28:34] Found info hash: 0fe6fccc925f09926d391695b40eeeebb0ee4c91
[2025-03-07 08:28:54] Found info hash: 7365e514f5b40caff0e842c92b80c62004ccb156
[2025-03-07 08:29:21] Found info hash: a440fecb25c736e2f6625ac238c9291ffeaa926c
^CStopping the DHT crawler...
[2025-03-07 08:29:26] Stopping the DHT crawler...

全告警输出（修改后） #

修改循环打印所有告警后，发现大量非DHT告警，DHT相关告警极少：

[2025-03-07 08:32:14] Start...
[2025-03-07 08:32:14] Alert: Local Service Discovery error: Invalid argument
[2025-03-07 08:32:14] Alert type: <class 'libtorrent.lsd_error_alert'>
[2025-03-07 08:32:14] Alert: Local Service Discovery error: Invalid argument
[2025-03-07 08:32:14] Alert type: <class 'libtorrent.lsd_error_alert'>
[2025-03-07 08:32:15] Alert: external IP received: 78.26.199.15
[2025-03-07 08:32:15] Alert type: <class 'libtorrent.external_ip_alert'>
[2025-03-07 08:32:16] Alert: incoming dht get_peers: 7365e90ffc5205a0d7a781e8e27b10f28f9fc97b
[2025-03-07 08:32:16] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:21] Alert: incoming dht get_peers: a440f4c7ff178202be2e1a1f2020ce45e25ae38f
[2025-03-07 08:32:21] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:22] Alert: incoming dht get_peers: a440f49544d1e8ef1112ad8c2ef73aa627f7d6db
[2025-03-07 08:32:22] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:30] Alert: DHT bootstrap complete
[2025-03-07 08:32:30] Alert type: <class 'libtorrent.alert'>
[2025-03-07 08:32:39] Alert: incoming dht get_peers: a440f39d3604b0510c3024c2dfdee946ada60e78
[2025-03-07 08:32:39] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:39] Alert: incoming dht get_peers: a440fb5561ecada7f18dd4de87fe7644202a245e
[2025-03-07 08:32:39] Alert type: <class 'libtorrent.dht_get_peers_alert'>
[2025-03-07 08:32:48] Alert: incoming dht get_peers: a440f3d386310aad792ce271646aa0d169e63541
[2025-03-07 08:32:48] Alert type: <class 'libtorrent.dht_get_peers_alert'>
^CStopping the DHT crawler...
[2025-03-07 08:33:00] Stopping the DHT crawler...

已尝试的操作 #

• 调整会话设置
• 添加多个DHT引导节点提升连通性
• 缩短主循环休眠时间以更快处理告警

咨询问题 #

1. 为何爬虫速度如此缓慢，且大量种子无响应？
2. 是否有特定的会话设置或配置可调整以提升性能？
3. 使用libtorrent优化DHT爬虫有哪些最佳实践？

内容的提问来源于stack exchange，提问作者RomaP