如何在字符串数组中查找并替换重复的连续词组？

我有如下字符串数组：

let prompts = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else"
];

需要编写一个函数实现以下功能：

• 查找数组中的重复项（例如This is some more word），将后续重复项替换为"REPEATED"，保留第一个实例
• 支持设置最小词数阈值，避免误匹配

调用函数后的期望输出：

[
  "This is some word",
  "This is some more word",
  "REPEATED",
  "REPEATED",
  "REPEATED",
  "This is something else"
];

这个需求是为了清理OpenAI提示词中的重复内容，已经困扰我很久了，提前感谢解答！

我目前写出的最接近需求的代码： #

function detectRepetitions(arr, minWords) {
  const uniqueWords = new Set();
  const repeatedWords = new Set();

  // 遍历数组，找出重复的词组
  for (let i = 0; i < arr.length; i++) {
    const words = arr[i].split(" ");

    for (let j = 0; j <= words.length - minWords; j++) {
      const subArray = words.slice(j, j + minWords);
      const subString = subArray.join(" ");

      if (uniqueWords.has(subString)) {
        repeatedWords.add(subString);
      } else {
        uniqueWords.add(subString);
      }
    }
  }

  // 将重复词组替换为'REPEATED'
  const result = arr.map((sentence) => {
    let words = sentence.split(" ");
    let repeatedFound = false;

    for (let i = 0; i <= words.length - minWords; i++) {
      const subArray = words.slice(i, i + minWords);
      const subString = subArray.join(" ");

      if (repeatedWords.has(subString)) {
        if (repeatedFound) {
          words.splice(i, minWords, "REPEATED");
        } else {
          repeatedFound = true;
        }
      }
    }

    return words.join(" ");
  });

  return result;
}

let prompt = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else",
];

const minWords = 4;
const result = detectRepetitions(prompt, minWords);
console.log(result);

符合需求的修正方案 #

当前代码逻辑是检测词组片段重复，但需求是检测整个字符串重复并替换后续实例，同时支持按最小词数过滤。以下是精准匹配需求的实现：

function replaceDuplicatePrompts(arr, minWordCount) {
  // 记录已出现的、符合词数要求的完整字符串
  const seen = new Set();
  return arr.map(str => {
    const wordCount = str.split(" ").length;
    // 词数不足阈值的字符串直接保留，不参与重复检测
    if (wordCount < minWordCount) {
      return str;
    }
    // 已出现过则替换为REPEATED，否则记录并保留原字符串
    if (seen.has(str)) {
      return "REPEATED";
    } else {
      seen.add(str);
      return str;
    }
  });
}

// 测试示例
const prompts = [
  "This is some word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is some more word",
  "This is something else"
];

// 设置最小词数为4
const processedPrompts = replaceDuplicatePrompts(prompts, 4);
console.log(processedPrompts);

代码说明 #

• minWordCount参数控制参与重复检测的字符串最小词数，避免短字符串误判为重复
• 使用Set存储已出现的符合要求的完整字符串，遍历数组时直接判断匹配，逻辑简洁高效
• 完全符合需求中的输出结果，精准处理整个字符串的重复替换

内容的提问来源于stack exchange，提问作者Fariz Luqman