使用Algolia DocSearch Scraper爬取数据时遇Unreachable hosts错误

我使用以下命令尝试爬取数据：

docker run -it --env-file=./config/development/dev.env -e "CONFIG=$(cat ./config/config.json | jq -r tostring)" algolia/docsearch-scraper

尽管已确认API-key和App-ID准确无误，但仍遇到「Unreachable hosts」错误。

错误信息 #

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/src/index.py", line 119, in <module>
    run_config(environ['CONFIG'])
  File "/root/src/index.py", line 45, in run_config
    config.query_rules
  File "/root/src/algolia_helper.py", line 21, in __init__
    self.index_name_tmp
  File "/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.6/site-packages/algoliasearch/search_client.py", line 127, in copy_rules
    return self.copy_index(src_index_name, dst_index_name, request_options)
  File "/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.6/site-packages/algoliasearch/search_client.py", line 94, in copy_index
    request_options,
  File "/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.6/site-packages/algoliasearch/http/transporter.py", line 35, in write
    return self.request(verb, hosts, path, data, request_options, timeout)
  File "/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.6/site-packages/algoliasearch/http/transporter.py", line 72, in request
    return self.retry(hosts, request, relative_url)
  File "/root/.local/share/virtualenvs/root-BuDEOXnJ/lib/python3.6/site-packages/algoliasearch/http/transporter.py", line 94, in retry
    raise AlgoliaUnreachableHostException("Unreachable hosts")
algoliasearch.exceptions.AlgoliaUnreachableHostException: Unreachable hosts

我的config.json配置 #

{
    "index_name": "dev_RESORTIFI_HELP",
    "start_urls": [
      "https://help.resortifi.com/"
    ],
    "sitemap_urls": [
      "https://help.resortifi.com/sitemap.xml"
    ],
    "sitemap_alternate_links": true,
    "stop_urls": [
      "/tests"
    ],
    "selectors": {
      "lvl0": {
        "selector": "(//ul[contains(@class,'menu__list')]//a[contains(@class, 'menu__link menu__link--sublist menu__link--active')]/text() | //nav[contains(@class, 'navbar')]//a[contains(@class, 'navbar__link--active')]/text())[last()]",
        "type": "xpath",
        "global": true,
        "default_value": "Documentation"
      },
      "lvl1": "header h1",
      "lvl2": "article h2",
      "lvl3": "article h3",
      "lvl4": "article h4",
      "lvl5": "article h5, article td:first-child",
      "lvl6": "article h6",
      "text": "article p, article li, article td:last-child"
    },
    "strip_chars": " .,;:#",
    "custom_settings": {
      "separatorsToIndex": "_",
      "attributesForFaceting": [
        "language",
        "version",
        "type",
        "docusaurus_tag"
      ],
      "attributesToRetrieve": [
        "hierarchy",
        "content",
        "anchor",
        "url",
        "url_without_anchor",
        "type"
      ]
    },
    "conversation_id": [
      "833762294"
    ],
    "nb_hits": 46250
  }

排查与解决方法 #

• 调整Docker网络模式
  容器默认网络可能存在访问限制，尝试使用宿主机网络运行命令：

  docker run -it --network host --env-file=./config/development/dev.env -e "CONFIG=$(cat ./config/config.json | jq -r tostring)" algolia/docsearch-scraper
  

• 验证容器内网络连通性
  先进入容器测试网络：

  docker run -it --entrypoint /bin/bash --env-file=./config/development/dev.env algolia/docsearch-scraper
  

  执行ping -c 3 algolia.net或curl https://<你的App-ID>-dsn.algolia.net（替换为实际App-ID对应的DSN），确认能否连通Algolia服务器。

• 清理无效配置字段
  conversation_id和nb_hits不属于DocSearch官方支持的配置项，可能引发逻辑异常，移除这两个字段后重新尝试。

• 确认环境变量注入正确性
  检查dev.env文件，确保APPLICATION_ID和API_KEY变量无多余空格、引号，值完全匹配Algolia后台的配置。

内容的提问来源于stack exchange，提问作者Mubina