Ruby中OptionParser处理含特殊字符文件名时触发UTF-8编码错误

问题概述 #

在Linux环境下使用Ruby 2.7.1p83编写程序，借助optparse组件解析命令行参数时，若参数包含带“特殊”字符的文件名，parse!方法会抛出以下错误：

invalid byte sequence in UTF-8

出错代码片段：

parser = OptionParser.new {
  |opts|
  ... etc. ...
}
parser.parse! # error occurs here

已知Ruby的scrub等编码处理方法，但错误发生在OptionParser#parse!库函数内部，无法直接修改库的字符串处理逻辑；若预处理命令行参数将特殊字符替换为UTF-8可接受的形式，后续会因文件名被修改而无法打开原文件；虽可通过建立编码参数与原参数的哈希映射、修改ARGV解析后再替换的复杂方案解决，但希望找到更简便的Ruby原生解决方案。

补充测试信息 #

编写最小测试程序rtest.rb：

#!/usr/bin/env ruby                                                                                                                               
# -*- ruby -*-                                                                                                                                         

require 'optparse'

parser = OptionParser.new {
}
parser.parse!

Process.exit(0)

测试环境：当前目录仅包含rtest.rb和名为Äfoo的文件，执行以下命令：

export LC_TYPE='en_us.UTF-8'
export LC_COLLATE='en_us.UTF-8'
./rtest.rb *

触发错误及栈跟踪：

Traceback (most recent call last):
    7: from /home/hippo/bin/rtest.rb:8:in `<main>'
    6: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1691:in `parse!'
    5: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1666:in `permute!'
    4: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1569:in `order!'
    3: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1575:in `parse_in_order'
    2: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1575:in `catch'
    1: from /opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1579:in `block in parse_in_order'
/opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb:1579:in `===': invalid byte sequence in UTF-8 (ArgumentError)

查看/opt/rubies/ruby-2.7.1/lib/ruby/2.7.0/optparse.rb第1579行相关代码：

1572   def parse_in_order(argv = default_argv, setter = nil, &nonopt)  # :nodoc:                                                                      
 1573     opt, arg, val, rest = nil
 1574     nonopt ||= proc {|a| throw :terminate, a}
 1575     argv.unshift(arg) if arg = catch(:terminate) {
 1576       while arg = argv.shift
 1577         case arg
 1578           # long option                                                                                                                           
 1579           when /\A--([^=]*)(?:=(.*))?/m
 1580             opt, rest = $1, $2

可见错误源于参数正则匹配时的编码问题。

进一步测试结果 #

在Ubuntu 20.0.4（Ruby 2.7.0）和Debian 8（Ruby 2.7.1）环境均触发该错误，尝试升级至Ruby 3.3.0仍报错：

% /opt/local/rubies/ruby-3.3.0/bin/ruby ./rtest.rb *
/opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1640:in `===': invalid byte sequence in UTF-8 (ArgumentError)
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1640:in `block in parse_in_order'
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1636:in `catch'
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1636:in `parse_in_order'
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1630:in `order!'
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1739:in `permute!'
    from /opt/local/rubies/ruby-3.3.0/lib/ruby/3.3.0+0/optparse.rb:1764:in `parse!'
    from ./rtest.rb:8:in `<main>'

后续定位问题根源为文件名编码异常：执行echo *显示Äfoo rtest.rb，但/bin/ls *显示''$'\304''foo' rtest.rb；用'Äfoo'无法访问文件，使用编码后的名称则可正常访问，推测Ruby无法处理该特殊编码形式。

内容的提问来源于stack exchange，提问作者HippoMan