如何以惯用方式移除DataFrame中缺失值占比过高的列？

问题背景 #

需要移除DataFrame中缺失值占比超过指定阈值的列，已有一种繁琐的实现方式，尝试Julia惯用写法时出现报错，实际数据集为330列、350000行，各列缺失值占比0-100%不等。

原有繁琐实现 #

using DataFrames
df = DataFrame(
    "A" => [1, 7, 2],
    "B" => [3, missing, -3],
    "C" => [3, 0, 6],
    "D" => [missing, 4, 2],
    "E" => [4, 3, -4])

nmis = describe(df, :nmissing)
filter!(row -> (row.nmissing / nrow(df) ) > 0.2, nmis)
for v in eachrow(nmis)
    var=v.variable
    select!(df, Not("$var"))
end
println(describe(df))

错误的惯用写法及报错 #

尝试的代码：

for col in eachcol(df)
    println(count(ismissing, col))
end
println(describe(df[!, [any(x -> count(ismissing, x) < 1, col) for col in eachcol(df)]]))

报错信息：

0
1
0
1
0
ERROR: LoadError: MethodError: no method matching iterate(::Missing)

Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen})
   @ Base range.jl:880
  iterate(::Union{LinRange, StepRangeLen}, ::Integer)
   @ Base range.jl:880
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}}
   @ Base dict.jl:698
  ...

Stacktrace:
  [1] iterate(::Base.Generator{Missing, typeof(ismissing)})
    @ Base .\generator.jl:44
  [2] _simple_count_helper(g::Base.Generator{Missing, typeof(ismissing)}, init::Int64)
    @ Base .\reduce.jl:1356
  [3] _simple_count(pred::Function, itr::Missing, init::Int64)
    @ Base .\reduce.jl:1352
  [4] count(f::Function, itr::Missing; init::Int64)
    @ Base .\reduce.jl:1350
  [5] count(f::Function, itr::Missing)
    @ Base .\reduce.jl:1350
  [6] (::var"#6#8")(x::Missing)
    @ Main c:\Users\TGebbels\OneDrive - The National Lottery Community Fund\Documents\360Giving\ShrinkCSV.jl:26
  [7] _any
    @ .\reduce.jl:1215 [inlined]
  [8] #any#829
    @ .\reducedim.jl:1004 [inlined]
  [9] any
    @ .\reducedim.jl:1004 [inlined]
 [10] (::var"#5#7")(col::Vector{Union{Missing, Int64}})
    @ Main .\none:0
 [11] iterate
    @ .\generator.jl:47 [inlined]
 [12] collect_to!(dest::Vector{Bool}, itr::Base.Generator{DataFrames.DataFrameColumns{DataFrame}, var"#5#7"}, offs::Int64, st::Int64)
    @ Base .\array.jl:840
 [13] collect_to_with_first!(dest::Vector{Bool}, v1::Bool, itr::Base.Generator{DataFrames.DataFrameColumns{DataFrame}, var"#5#7"}, st::Int64)
    @ Base .\array.jl:818
 [14] collect(itr::Base.Generator{DataFrames.DataFrameColumns{DataFrame}, var"#5#7"})
    @ Base .\array.jl:792
 [15] top-level scope
    @ c:\Users\TGebbels\OneDrive ...\Documents\360Giving\ShrinkCSV.jl:26
in expression starting at c:\Users\TGebbels\OneDrive ...\Documents\360Giving\ShrinkCSV.jl:26

错误原因 #

错误代码中any(x -> count(ismissing, x) < 1, col)逻辑错误：col是列向量，any会遍历列中的每个元素x，而count(ismissing, x)试图对单个值（如missing或整数）进行计数，但单个值不是可迭代对象，因此抛出MethodError。正确做法是直接对整个列统计缺失值数量，无需使用any。

符合Julia惯用风格的实现方法 #

以下几种方法简洁高效，适合大数据集：

方法1：筛选列名后更新DataFrame #

using DataFrames

df = DataFrame(
    "A" => [1, 7, 2],
    "B" => [3, missing, -3],
    "C" => [3, 0, 6],
    "D" => [missing, 4, 2],
    "E" => [4, 3, -4])

threshold = 0.2  # 指定缺失值占比阈值
total_rows = nrow(df)

# 筛选出缺失值占比≤阈值的列名
keep_cols = [col_name for col_name in names(df) if count(ismissing, df[!, col_name]) / total_rows ≤ threshold]

# 更新原DataFrame
select!(df, keep_cols)

println(describe(df))

方法2：结合eachcol与列名枚举 #

using DataFrames

df = DataFrame(
    "A" => [1, 7, 2],
    "B" => [3, missing, -3],
    "C" => [3, 0, 6],
    "D" => [missing, 4, 2],
    "E" => [4, 3, -4])

threshold = 0.2
total_rows = nrow(df)

# enumerate(eachcol(df, true)) 会返回(列名, 列数据)的元组
keep_cols = [col_name for (col_name, col_data) in enumerate(eachcol(df, true)) 
             if count(ismissing, col_data) / total_rows ≤ threshold]

select!(df, keep_cols)

println(describe(df))

方法3：使用Cols函数的简洁写法 #

using DataFrames

df = DataFrame(
    "A" => [1, 7, 2],
    "B" => [3, missing, -3],
    "C" => [3, 0, 6],
    "D" => [missing, 4, 2],
    "E" => [4, 3, -4])

threshold = 0.2
total_rows = nrow(df)

# 直接在select!中用Cols筛选符合条件的列
select!(df, Cols(col -> count(ismissing, col) / total_rows ≤ threshold))

println(describe(df))

以上三种方法均为Julia惯用的函数式风格写法，代码简洁且效率较高，适合处理大规模数据集。

内容的提问来源于stack exchange，提问作者Tim Gebbels