词干提取及标点移除后如何按换行符拆分正则匹配结果列表
这个问题和正则表达式的or运算符没有关系,是循环逻辑和临时变量作用域的错误导致的,核心问题有两个:
- 你将
result.append(words)、unstemmed_list.append(words2)写在了遍历result2的循环外部,整个遍历完成后只会执行1次追加操作,自然最终得到的是只有1个元素的超长列表,这个元素就是所有匹配内容拼接后的结果。 - 用来临时存储单条文本分词结果的
tokenized_txt、auxlist、tokenized_txt_latin都是全局定义的列表,处理每一条新的匹配内容时都没有清空,之前处理过的文本内容会一直累积在列表里,所以你把追加操作移到循环内后,每次新增的内容都是之前所有文本加当前文本的总和,才会出现重复累加的情况。
修正后的代码如下:
import csv import re import string import nltk from nltk.tokenize import punkt, word_tokenize from nltk.corpus import stopwords from nltk.stem import SnowballStemmer import langid h=SnowballStemmer("hungarian") # 匈牙利语词干提取器 stop_words=set(stopwords.words("hungarian")) i=0.0 j=0.0 latin_counter=0.0 result=[] result2=[] unstemmed_list=[] stop_words_latin={'ab', 'ac', 'ad', 'adhic', 'aliqui', 'aliquis', 'an', 'ante', 'apud', 'at', 'atque', 'aut', 'autem', 'cum', 'cur', 'de', 'deinde', 'dum', 'ego', 'enim', 'ergo', 'es', 'est', 'et', 'etiam', 'etsi', 'ex', 'fio', 'haud', 'hic', 'iam', 'idem', 'igitur', 'ille', 'in', 'infra', 'inter', 'interim', 'ipse', 'is', 'ita', 'magis', 'modo', 'mox', 'nam', 'ne', 'nec', 'necque', 'neque', 'nisi', 'non', 'nos', 'o', 'ob', 'per', 'possum', 'post', 'pro', 'quae', 'quam', 'quare', 'qui', 'quia', 'quicumque', 'quidem', 'quilibet', 'quis', 'quisnam', 'quisquam', 'quisque', 'quisquis', 'quo', 'quoniam', 'sed', 'si', 'sic', 'sive', 'sub', 'sui', 'sum', 'super', 'suus', 'tam', 'tamen', 'trans', 'tu', 'tum', 'ubi', 'uel', 'uero'} with open('data/onkology.csv', 'r') as csv_file: csv_reader= csv.reader(csv_file ,delimiter=';') exp = r'[lL]u.r[oi]n\b|\w*peptyl\b|\w*lutamid\b' for line in csv_reader: i += 1 for lineElem in line: if re.search(exp, lineElem) and len(lineElem) > 80: result2.append(lineElem) j += 1 for listElem in result2: k,_ = langid.classify(listElem) # 每次处理新条目时初始化临时列表,避免内容累加 tokenized_txt = [] auxlist = [] tokenized_txt_latin = [] if k == 'la': latin_counter += 1 words = word_tokenize(listElem) # 移除停用词 for w in words: if w not in stop_words_latin: tokenized_txt_latin.append(w) # 移除标点 tokenized_txt_latin = [word for word in tokenized_txt_latin if word.isalpha()] processed_words = ' '.join(tokenized_txt_latin) # 拉丁语内容追加到结果 result.append(processed_words) unstemmed_list.append(processed_words) else: words = word_tokenize(listElem) # 移除停用词 for w in words: if w not in stop_words: auxlist.append(w) tokenized_txt.append(h.stem(w)) # 移除标点 auxlist = [word for word in auxlist if word.isalpha()] tokenized_txt = [word for word in tokenized_txt if word.isalpha()] words2 = ' '.join(auxlist) processed_stem_words = ' '.join(tokenized_txt) # 匈牙利语内容追加到结果 result.append(processed_stem_words) unstemmed_list.append(words2) print("匹配率为:", (j/i)) print(unstemmed_list ,"\n") print(result,"\n") # 结果写入文件 with open('listfile.txt', 'w') as filehandle: for listitem in result: filehandle.write('%s\n' % listitem) with open('listfile_unstemmed.txt', 'w') as filehandle: for listitem in unstemmed_list: filehandle.write('%s\n' % listitem)
内容的提问来源于stack exchange,提问作者Kafka
相关产品推荐
相关产品推荐

