Python对比两个Markdown文件：相似度计算与缺失内容排查问题

我有两个Markdown文件old_file.md和new_file.md，需要获取new_file相对old_file的相似度百分比，以及找出创建new_file时遗漏的内容。我用difflib库写了对比代码，但得到的相似度结果明显不符合实际，想问问当前方法是不是有问题，或者有没有其他Python库/方案能实现需求。

相关文件内容 #

old_file.md：

10

For some cases a higher input temperature can be allowed when requested and checked by the manufacturer.

The outlet water temperature rise is from 5–10 K.

The minimum pressure and amount of cooling water for the basic construction of a water-cooled motor is shown in the following table. Please check the requirements for pressure and the amount of cooling water in the case of special constructions.

If the amount of water varies, its temperature rise will be inversely proportional to the flow rate.

### 9.2. Filling or draining cooling water

When filling, open the air vent plug on top of the motor (see figure 2). Let the cooling water flow into the motor until it comes out of the air gap. Close the air gap with a plug and seal the joint with sealing tape or strip. Filling must be done carefully so that no air is left in the motor’s cooling channels. Check for possible leaks after the piping and joints have been connected.

Emptying can be done with pressurized air. After emptying, the plugs must be re-fitted, and the seals of the joints must be checked.

## 10. Consendation drain holes

It is of special importance with water cooled motors that the condensation drain holes are located in the correct position (fig. 1). Check that the condensation drain holes face downwards, especially when the mounting arrangement differs from standard.

## 11. Water leakage detector

This motor is equipped with float type leakage detector in non-drive end (see figure 2 and 3). The leakage detector has a magnetic float switch. The magnetic float switch is positioned on a non-magnetic guide tube. When a specified water level is reached, the magnetic field produced by the magnet in the float actuates a reed switch (sealed contact) inside the guide tube.

This closes the electric circuit that transmits the alarm signal to the control board.

3GZF500725-144 EN 03-2023 | ABB IEC LV MOTORS

new_file.md：

<p>10</p><p>3GZF500725-144 EN 03-2023 | ABB IEC LV MOTORS</p><p>For some cases a higher input temperature can be allowed when requested and checked by
the manufacturer.</p><p>The outlet water temperature rise is from 5–10 K.</p><p>The minimum pressure and amount of cooling water for the basic construction of a water-
cooled motor is shown in the following table. Please check the requirements for pressure and
the amount of cooling water in the case of special constructions.</p><p>If the amount of water varies, its temperature rise will be inversely proportional to the flow
rate.</p><p>9.2.
Filling or draining cooling water</p><p>When filling, open the air vent plug on top of the motor (see figure 2). Let the cooling water
flow into the motor until it comes out of the air gap. Close the air gap with a plug and seal
the joint with sealing tape or strip. Filling must be done carefully so that no air is left in the
motor’s cooling channels. Check for possible leaks after the piping and joints have been con-
nected.</p><p>Emptying can be done with pressurized air. After emptying, the plugs must be re-fitted, and
the seals of the joints must be checked.</p><p>10.
Consendation drain holes</p><p>It is of special importance with water cooled motors that the condensation drain holes are lo-
cated in the correct position (fig. 1). Check that the condensation drain holes face down-
wards, especially when the mounting arrangement differs from standard.</p><p>11.
Water leakage detector</p><p>This motor is equipped with float type leakage detector in non-drive end (see figure 2 and 3).
The leakage detector has a magnetic float switch. The magnetic float switch is positioned on
a non-magnetic guide tube. When a specified water level is reached, the magnetic field pro-
duced by the magnet in the float actuates a reed switch (sealed contact) inside the guide
tube.</p><p>This closes the electric circuit that transmits the alarm signal to the control board.</p>

使用的代码 #

from difflib import ndiff

def compare_files_take_5(gold_file_path, predicted_file_path, threshold=0.8):
  try:
    with open(gold_file_path, 'r') as gold_file, open(predicted_file_path, 'r') as predicted_file:
      gold_content = gold_file.read()
      predicted_content = predicted_file.read()

    differences = list(ndiff(gold_content, predicted_content))
    
    added_text = ''.join([diff[2:] for diff in differences if diff.startswith('+')])
    deleted_text = ''.join([diff[2:] for diff in differences if diff.startswith('-')])

    added_length = len(added_text)
    deleted_length = len(deleted_text)

    total_length = max(len(gold_content), len(predicted_content))

    similarity_ratio = 1 - (added_length + deleted_length) / total_length

    is_similar = similarity_ratio >= threshold

    return is_similar, similarity_ratio, added_length, deleted_length, added_text, deleted_text

  except Exception as e:
    print(f"Error: {e}")
    return False, 0, 0, 0, '', ''

# Example usage
gold_standard_file = "/Old_file.md"
predicted_file = "/New_file.md"

is_similar, similarity_ratio, added_length, deleted_length, added_text, deleted_text = compare_files_take_5(gold_standard_file, predicted_file)

print(f"Similarity Ratio: {similarity_ratio:.2%}")
print(f"Is Similar: {is_similar}")
print(f"Added Length: {added_length} characters")
print(f"Deleted Length: {deleted_length} characters")

print("\nAdded Text:")
print(added_text)

print("\nDeleted Text:")
print(deleted_text)

异常输出 #

Similarity Ratio: -91.44%
Is Similar: False
Added Length: 1925 characters
Deleted Length: 1854 characters

Added Text:
<p></></><></p><p>For some cases a higher input temperature can be allowed when requested and checked by
the manufacturer.</p><p>The outlet water temperature rise is from 5–10 K.</p><p>The minimum pressure and amount of cooling water for the basic construction of a water-
cooled motor is shown in the following table. Please check the requirements for pressure and
the amount of cooling water in the case of special constructions.</p><p>If the amount of water varies, its temperature rise will be inversely proportional to the flow
rate.</p><p>9.2.
Filling or draining cooling water</p><p>When filling, open the air vent plug on top of the motor (see figure 2). Let the cooling water
flow into the motor until it comes out of the air gap. Close the air gap with a plug and seal
the joint with sealing tape or strip. Filling must be done carefully so that no air is left in the
motor’s cooling channels. Check for possible leaks after the piping and joints have been con-
nected.</p><p>Emptying can be done with pressurized air. After emptying, the plugs must be re-fitted, and
the seals of the joints must be checked.</p><p>10.
Consendation drain holes</p><p>It is of special importance with water cooled motors that the condensation drain holes are lo-
cated in the correct position (fig. 1). Check that the condensation drain holes face down-
wards, especially when the mounting arrangement differs from standard.</p><p>11.
Water leakage detector</p><p>This motor is equipped with float type leakage detector in non-drive end (see figure 2 and 3).
The leakage detector has a magnetic float switch. The magnetic float switch is positioned on
a non-magnetic guide tube. When a specified water level is reached, the magnetic field pro-
duced by the magnet in the float actuates a reed switch (sealed contact) inside the guide
tube.</p><p>This closes the electric circuit that transmits the alarm signal to the control board.</p>

Deleted Text:


For some cases a higher inut temerature can be allowed when requested and checked by the manufacturer.

The outlet water temperature rise is from 5–10 K.

The minimum pressure and amount of cooling water for the basic construction of a water-cooled motor is shown in the following table. Please check the requirements for pressure and the amount of cooling water in the case of special constructions.

If the amount of water varies, its temperature rise will be inversely proportional to the flow rate.

### 9.2. Filling or draining cooling water

When filling, open the air vent plug on top of the motor (see figure 2). Let the cooling water flow into the motor until it comes out of the air gap. Close the air gap with a plug and seal the joint with sealing tape or strip. Filling must be done carefully so that no air is left in the motor’s cooling channels. Check for possible leaks after the piping and joints have been connected.

Emptying can be done with pressurized air. After emptying, the plugs must be re-fitted, and the seals of the joints must be checked.

## 10. Consendation drain holes

It is of special importance with water cooled motors that the condensation drain holes are located in the correct position (fig. 1). Check that the condensation drain holes face downwards, especially when the mounting arrangement differs from standard.

## 11. Water leakage detector

This motor is equipped with float type leakage detector in non-drive end (see figure 2 and 3). The leakage detector has a magnetic float switch. The magnetic float switch is positioned on a non-magnetic guide tube. When a specified water level is reached, the magnetic field produced by the magnet in the float actuates a reed switch (sealed contact) inside the guide tube.

This closes the electric circuit that transmits the alarm signal to the control board.

问题分析 #

当前方法的核心问题在于直接对比原始文本内容，没有处理两个文件的格式差异：

1. 格式干扰：new_file.md包含大量HTML标签（<p>），old_file.md是纯Markdown格式，这些格式标签被当作新增内容，严重干扰相似度计算。
2. 排版差异：两个文件的换行、断字（比如con-nected vs connected）被视为字符差异，进一步放大计算误差。
3. 相似度公式不合理：1 - (added_length + deleted_length)/total_length的计算方式，当新增和删除的字符总和超过文件长度时，会出现负的相似度，不符合逻辑。

解决方案 #

步骤1：预处理文本，统一格式 #

先去除无关的格式标签，统一换行和排版，只保留核心文本内容：

• 去除new_file.md中的HTML标签
• 去除old_file.md中的Markdown标题符号（#、##等）
• 修复断字、统一换行符，合并多余空行

步骤2：使用更合理的相似度计算方法 #

推荐用difflib.SequenceMatcher，它能更准确地计算文本相似度，避免字符级差异的过度放大；或者使用基于词频的相似度算法。

改进后的代码 #

from difflib import SequenceMatcher
import re

def clean_text(text):
    # 去除HTML标签
    text = re.sub(r'<[^>]+>', '', text)
    # 去除Markdown标题符号
    text = re.sub(r'^#{1,3}\s', '', text, flags=re.MULTILINE)
    # 修复断字（比如con-nected转为connected）
    text = re.sub(r'(\w+)-\n(\w+)', r'\1\2', text)
    # 统一换行，合并多余空行
    lines = [line.strip() for line in text.split('\n') if line.strip()]
    return ' '.join(lines)

def compare_files(gold_file_path, predicted_file_path):
    try:
        with open(gold_file_path, 'r') as f:
            gold_content = clean_text(f.read())
        with open(predicted_file_path, 'r') as f:
            predicted_content = clean_text(f.read())
        
        # 计算相似度
        matcher = SequenceMatcher(None, gold_content, predicted_content)
        similarity_ratio = matcher.ratio()
        
        # 找出old_file中存在但new_file中缺失的内容
        missing_parts = []
        for tag, i1, i2, j1, j2 in matcher.get_opcodes():
            if tag == 'delete':
                missing_parts.append(gold_content[i1:i2])
        
        return similarity_ratio, '\n'.join(missing_parts)
    
    except Exception as e:
        print(f"Error: {e}")
        return 0, ''

# 使用示例
similarity, missing_content = compare_files("/Old_file.md", "/New_file.md")
print(f"相似度: {similarity:.2%}")
print("\n遗漏内容:")
print(missing_content if missing_content else "无遗漏内容")

其他可选方案 #

• fuzzywuzzy库：基于Levenshtein距离计算文本相似度，使用简单，适合快速实现：

  from fuzzywuzzy import fuzz
  # 预处理文本后，直接计算
  similarity = fuzz.ratio(gold_content, predicted_content)
  

• nltk库：如果需要基于语义的相似度，可以用词嵌入或TF-IDF进行对比，适合更复杂的文本场景。

内容的提问来源于stack exchange，提问作者Harshal Naik