超大规模文本文件行过滤求助：基于包含/排除列表的精准属性匹配方案（替代失效的grep/awk命令）

Let's tackle this large-scale filtering problem properly—since you've got millions of lines to process and hit memory limits with previous attempts, prioritizing the include.txt approach is the right call. Here are two robust solutions tailored to different system memory constraints:

Solution 1: Optimized awk (Fastest if You Have Enough RAM) #

awk is ideal for this task, but we need to handle the 15 million include entries efficiently. This script loads the include list into a hash set first, then checks each line of original.txt against it:

BEGIN {
    # Split lines by double quotes to easily grab the name attribute value
    FS = "\""
}
# Load all include names into a hash map during the first pass
NR == FNR {
    include[$1] = 1
    next
}
# For each line in original.txt, check if the block-list:name value is in our include set
{
    # For your consistent line format, the block-list:name value is the 4th field when splitting by quotes
    if ($4 in include) {
        print $0
    }
}

Run it with:

awk -f filter.awk include.txt original.txt > final.txt

Flexible Alternative (If Line Format Varies) #

If your lines don't follow the exact structure every time, use a regex to safely extract the block-list:name value instead:

BEGIN {
    # Load include list into memory explicitly
    while ((getline < "include.txt") > 0) {
        include[$0] = 1
    }
}
# Use regex to isolate the block-list:name value
match($0, /block-list:name="([^"]+)"/, match_arr) {
    if (match_arr[1] in include) {
        print $0
    }
}

Memory Note: Storing 15 million strings will take roughly 1-2GB of RAM (depending on average string length). If your system can't spare that, move to the next solution.

Solution 2: Sort + Join (Low-Memory, Highly Scalable) #

This approach uses standard Unix tools that stream data to disk instead of loading everything into memory—perfect for systems with limited RAM:

1. First, sort the include file (one-time setup cost):

   sort include.txt > include_sorted.txt
   

2. Extract the block-list:name value from each line of original.txt, prepend it to the line (with a tab separator), then sort the result:

   sed -E 's/.*block-list:name="([^"]+)".*/\1\t&/' original.txt | sort > original_sorted.txt
   

3. Join the sorted files to keep only lines where the name is in the include list, then strip off the prepended name to restore the original line format:

   join include_sorted.txt original_sorted.txt | cut -f2- > final.txt
   

How This Works #

• We tag each line of original.txt with its target name, sort both tagged lines and the include list, then use join to match only lines with names in the include set.
• sort will use temporary disk storage if needed, so it works seamlessly even on systems with minimal RAM.

Why Your Previous Attempts Failed #

Commands like grep -f or naive awk scripts that load huge lists inefficiently either eat up RAM or take forever because they check every line against every entry in the exclude/include list. The solutions above are either linear (awk) or O(n log n) (sort+join), which are vastly more efficient for million-scale datasets.

内容的提问来源于stack exchange，提问作者shantanuo