如何使用grep检查匹配模式的行是否存在并写入另一文件
解决日志文件按模式筛选并去重写入目标文件的问题
问题场景
现有源日志文件内容如下:
The color is Orange then can used to binding. The color is Black then can used to binding. The animal Lion need to carefully. The color is Black then can used to binding. The animal Zebra need to carefully. The animal Tiger need to carefully. The animal Bee need to skip. The color is White then can used to binding. The color is Yellow then can used to binding. The animal Ant need to skip. The animal Tiger need to carefully. The color is Red then can used to filled. The color is Green then can used to filled.
需求为:筛选源文件中匹配.*then can used to binding或.*need to carefully的行,仅将目标文件中不存在的行写入目标文件,最终目标文件需保留去重后的符合条件行,期望输出:
The color is Orange then can used to binding. The color is Black then can used to binding. The animal Lion need to carefully. The animal Zebra need to carefully. The animal Tiger need to carefully. The color is White then can used to binding. The color is Yellow then can used to binding.
原脚本直接使用grep >>会重复写入已有行,无法实现去重需求,同时还存在dest_file变量未闭合双引号的语法错误。
修改后的脚本
#!/bin/bash source_file="/home/user1/source/source_sample.log" dest_file="/home/user1/dest/dest_sample.log" # 定义需要匹配的模式 pattern1=".*then can used to binding" pattern2=".*need to carefully" # 筛选符合模式的行 → 去重 → 过滤目标文件已存在的行 → 追加到目标文件 grep -e "$pattern1" -e "$pattern2" "$source_file" | sort -u | grep -vxFf "$dest_file" >> "$dest_file"
核心逻辑说明
- 筛选符合模式的行:
grep -e "$pattern1" -e "$pattern2" "$source_file"从源文件中提取所有匹配指定模式的行 - 去重处理:
sort -u对筛选后的行排序并去重,确保同一内容的行仅保留一份 - 过滤已有行:
grep -vxFf "$dest_file"以目标文件的内容为匹配模板,反向筛选出目标文件中不存在的行(-x匹配整行、-F按固定字符串匹配、-f读取目标文件作为匹配规则、-v输出不匹配的行) - 追加写入:
>> "$dest_file"将筛选出的新行追加到目标文件末尾
内容的提问来源于stack exchange,提问作者Brian Crist
相关产品推荐
相关产品推荐

