Erlang并行HTTP请求网络问题：地理信息服务器数据同步异常

Hey Chris, let's break down how to fix that network hiccup you're facing when parallelizing downloads after extracting ISO country codes from countryInfo.txt. Erlang's concurrency is powerful, but unmanaged parallelism plus public service constraints can easily cause these kinds of issues. Here's what you can do:

1. Control Parallelism to Respect GeoNames' Rate Limits #

Public services like GeoNames almost always enforce rate limits to prevent abuse. Spawning too many concurrent download processes will get you throttled or blocked, which looks like a network error.

Fix: Use a Process Pool #

Instead of spawning an unbounded number of processes, use a process pool to cap concurrency. You can use the popular poolboy library, or roll a simple one yourself:

% Simple fixed-size worker pool
start_pool(Size) ->
    Pids = [spawn(fun() -> worker_loop() end) || _ <- lists:seq(1, Size)],
    register(download_pool, spawn(fun() -> pool_loop(Pids) end)).

pool_loop(Workers) ->
    receive
        {download, Url, From} ->
            [Worker | Rest] = Workers,
            Worker ! {task, Url, From},
            pool_loop(Rest ++ [Worker]);
        _ ->
            pool_loop(Workers)
    end.

worker_loop() ->
    receive
        {task, Url, From} ->
            Result = download_url(Url),
            From ! {download_result, Url, Result},
            worker_loop()
    end.

Start the pool with a reasonable size (e.g., 5-10 workers) based on GeoNames' implied limits—start small and adjust if you get 429 Too Many Requests errors.

2. Add Timeouts, Retries, and Exponential Backoff #

Network flakiness or temporary service unavailability is common. Without retry logic, single failures will break your sync.

Fix: Implement Retry with Backoff #

Wrap your download function with a retry mechanism that uses exponential backoff to avoid overwhelming the server:

download_with_retry(Url, MaxRetries) ->
    download_with_retry(Url, MaxRetries, 1).

download_with_retry(_Url, 0, _Delay) ->
    {error, max_retries_exhausted};
download_with_retry(Url, RetriesLeft, Delay) ->
    case httpc:request(get, {Url, []}, [{timeout, 15000}, {connect_timeout, 5000}], []) of
        {ok, _Result} ->
            {ok, _Result};
        {error, Reason} ->
            error_logger:info_msg("Download failed for ~p: ~p, retrying in ~p ms~n", [Url, Reason, Delay * 1000]),
            timer:sleep(Delay * 1000),
            download_with_retry(Url, RetriesLeft - 1, Delay * 2)
    end.

This gives the service time to recover between attempts and reduces the chance of hitting rate limits.

3. Validate ISO Codes and URLs Before Downloading #

Sometimes what looks like a network error is actually a bad request caused by malformed ISO codes from countryInfo.txt.

Fix: Sanitize and Validate Extracted Data #

After parsing countryInfo.txt, validate each ISO code and construct the URL properly. Add a check to ensure the resource exists before downloading:

validate_and_download(ISOCode) ->
    Url = lists:flatten(io_lib:format("https://download.geonames.org/export/dump/~s.zip", [ISOCode])),
    % Send a HEAD request to check if the resource exists
    case httpc:request(head, {Url, []}, [{timeout, 5000}], []) of
        {ok, {_StatusLine, _Headers, _}} ->
            download_with_retry(Url, 3);
        {error, Reason} ->
            error_logger:warning_msg("Invalid URL for ISO ~p: ~p~n", [ISOCode, Reason]),
            {error, invalid_url}
    end.

This filters out bad URLs early, so you don't waste resources on failed requests.

4. Optimize Erlang HTTP Client Configuration #

The default httpc settings might not be ideal for high parallelism. Tweak the connection pool and keepalive settings:

% Initialize httpc with optimized settings
init_httpc() ->
    inets:start(),
    httpc:set_options([
        {max_sessions, 20}, % Increase concurrent sessions
        {keepalive, true},  % Reuse TCP connections
        {pipeline, true}    % Enable HTTP pipelining if supported
    ]).

This reduces the overhead of establishing new TCP connections for each request.

5. Add Detailed Logging and Monitoring #

To debug exactly what's going wrong, add fine-grained logging to track each download's status. Use Erlang's built-in logger module or a library like lager:

log_download_result(Url, Result) ->
    case Result of
        {ok, {{_HttpVersion, 200, _}, _, _}} ->
            logger:info("Successfully downloaded ~s", [Url]);
        {ok, {{_HttpVersion, StatusCode, _}, _, _}} ->
            logger:warning("Download failed for ~s with status code ~p", [Url, StatusCode]);
        {error, Reason} ->
            logger:error("Download error for ~s: ~p", [Url, Reason])
    end.

Logs will help you distinguish between actual network issues, rate limiting, and invalid requests.

内容的提问来源于stack exchange，提问作者Chris W