为何使用自定义SC模块的Elixir脚本未完成任务就退出？

我在Elixir脚本wget.exs中实现了SC模块，调用Erlang的OS模块执行命令并打印输出，用于批量下载软件包。脚本逻辑是先下载包含包列表的文本文件，再并行下载所有包。但使用SC模块时，脚本启动下载后提前退出并抛出超时错误；换成Elixir原生的System模块则能正常完成所有下载并打印提示信息。

脚本代码 #

#!/usr/bin/env elixir

defmodule SC do
  def run(cmmnd) do
    output = :os.cmd(String.to_charlist(cmmnd))    
    IO.puts output
  end
end

defmodule Downloader do
  def download_packages do
    lst_f_pckgs = "wget-list-sysv"

    #System.cmd("wget", ["https://www.linuxfromscratch.org/lfs/view/stable/#{lst_f_pckgs}"])
    SC.run("wget -c https://www.linuxfromscratch.org/lfs/view/stable/#{lst_f_pckgs}")
    
    {:ok, file_content} = File.read(lst_f_pckgs)
    arr_f_pckgs = String.split(file_content, ~r/\s+/)

    tasks =
      arr_f_pckgs
      |> Enum.map(fn l ->
        #Task.async(fn -> System.cmd("wget", [l]) end)
        Task.async(fn -> SC.run("wget -c #{l}") end)
      end)

    tasks
    |> Enum.each(&Task.await/1)

    IO.puts "All of them are downloaded."
  end
end

Downloader.download_packages()

错误输出 #

5550K .......... .......... .......... .......... .......... 98% 11.6M 0s
  5600K .......... .......... .......... .......... .......... 99%  457K 0s
  5650K .......... .......... ..                              100%  145M=8.1s

2024-01-15 13:38:05 (702 KB/s) - ‘coreutils-9.3.tar.xz’ saved [5808696/5808696]


--2024-01-15 13:37:56--  https://github.com/systemd/systemd/archive/v254/systemd-254.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/systemd/systemd/tar.gz/refs/tags/v254 [following]
--2024-01-15 13:37:56--  https://codeload.github.com/systemd/systemd/tar.gz/refs/tags/v254
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘systemd-254.tar.gz’

     0K .......... .......... .......... .......... ..........  267K
    50K .......... .......... .......... .......... .......... 27.2M
   100K .......... .......... .......... .......... ..........  442K
   150K .......... .......... .......... .......... ..........  202K
[many similar lines]
8.10M 0s
  6200K .......... .......... .......... .......... .......... 97% 31.7M 0s
  6250K .......... .......... .......... .......... .......... 98% 1.48M 0s
  6300K .......... .......... .......... .......... .......... 98%  700K 0s
  6350K .......... .......... .......... .......... .......... 99% 5.73M 0s
  6400K .......... .......... .......                         100%  460M=10s

2024-01-15 13:38:07 (639 KB/s) - ‘grub-2.06.tar.xz’ saved [6581924/6581924]


** (exit) exited in: Task.await(%Task{mfa: {:erlang, :apply, 2}, owner: #PID<0.96.0>, pid: #PID<0.110.0>, ref: #Reference<0.1214714622.3043557380.75212>}, 5000)
    ** (EXIT) time out
    (elixir 1.14.0) lib/task.ex:830: Task.await/2
    (elixir 1.14.0) lib/enum.ex:975: Enum."-each/2-lists^foreach/1-0-"/2
    wget.exs:28: Downloader.download_packages/0

问题原因 #

1. :os.cmd/1的输出收集特性：:os.cmd/1会同步阻塞直到命令执行完毕，并且会收集命令的所有输出内容，直到命令结束才返回结果。当下载大文件时，命令执行时间超过Task.await/1默认的5秒超时限制，就会触发超时错误。
2. System.cmd/2的差异：System.cmd/2默认支持实时处理输出（可通过into参数控制），不会等待所有输出收集完成再返回，同时参数传递方式更安全（避免命令注入风险），因此不会触发Task超时。

解决方法 #

方法1：延长Task等待时间

修改Task.await/1调用，显式设置无限等待时间：

tasks
|> Enum.each(&Task.await(&1, :infinity))

这种方式能快速解决超时问题，但:os.cmd/1收集全部输出再返回的特性依然存在，大文件下载时会占用更多内存。

方法2：替换为System.cmd/2

直接使用Elixir原生的System.cmd/2替代自定义SC模块，既安全又能实时输出下载进度：

defmodule Downloader do
  def download_packages do
    lst_f_pckgs = "wget-list-sysv"

    System.cmd("wget", ["-c", "https://www.linuxfromscratch.org/lfs/view/stable/#{lst_f_pckgs}"], into: IO.stream(:stdio, :line))
    
    {:ok, file_content} = File.read(lst_f_pckgs)
    arr_f_pckgs = String.split(file_content, ~r/\s+/)

    tasks =
      arr_f_pckgs
      |> Enum.map(fn l ->
        Task.async(fn ->
          System.cmd("wget", ["-c", l], into: IO.stream(:stdio, :line))
        end)
      end)

    tasks
    |> Enum.each(&Task.await(&1, :infinity))

    IO.puts "All of them are downloaded."
  end
end

通过into: IO.stream(:stdio, :line)让wget的输出实时打印到终端，同时避免输出堆积导致的超时。

方法3：改进SC模块为非阻塞模式

如果需要保留自定义模块，可以使用Erlang的Port实现非阻塞命令执行，实时处理输出：

defmodule SC do
  def run(cmmnd) do
    port = Port.open({:spawn, cmmnd}, [:binary, :exit_status, :use_stdio, :stderr_to_stdout])
    handle_port_output(port)
  end

  defp handle_port_output(port) do
    receive do
      {^port, {:data, data}} ->
        IO.write(data)
        handle_port_output(port)
      {^port, {:exit_status, 0}} ->
        :ok
      {^port, {:exit_status, status}} ->
        {:error, status}
    end
  end
end

这种方式会实时输出命令结果，不会因为输出收集导致Task超时，同时保持自定义模块的灵活性。

内容的提问来源于stack exchange，提问作者John Smith