You need to enable JavaScript to run this app. Scala Spark基于SparkConf实现RDD[String]推文停用词移除的技术问题
优惠活动
大模型
产品
解决方案
定价
更多

Scala Spark基于SparkConf实现RDD[String]推文停用词移除的技术问题

原有代码的核心问题
  • 过滤逻辑完全偏离需求:原filter的判断条件是「行以两个城市开头 OR 整行字符串不属于停用词」,导致所有非城市开头的普通推文只要不是单个停用词都会被保留,完全没实现“只保留两个城市开头推文”的筛选目标
  • 停用词处理维度错误:原逻辑把整行推文当成单个元素去判断是否匹配停用词,从来没对推文内部的单个词汇做拆分过滤,根本达不到移除行内停用词的效果
  • 误用算子导致结构破坏:之前尝试用flatMap处理停用词时,会把单条推文拆分出的多个词汇直接作为独立RDD元素输出,彻底丢失原推文的完整行结构
适配要求的解决路径(完全保留SparkConf+SparkContext初始化结构,不引入SparkSession)
  • 优化停用词查找效率:把原有的停用词数组转为HashSet存储,避免数组逐元素查找的性能损耗
  • 分两步处理RDD:
    • 第一步用filter算子严格筛选:仅保留以San Francisco、Chicago开头的推文行,丢弃其余所有不符合筛选条件的内容
    • 第二步用map算子逐行处理:对筛选后的每一条推文,先按分隔规则拆分为独立词汇,清理词汇首尾附带的标点符号,过滤掉匹配停用词的元素,再把剩余词汇拼接为完整的文本行输出,全程不改变RDD的行结构
  • 关于StopWordsRemover的适配说明:该组件原生基于DataFrame/Dataset API设计,依赖SparkSession上下文,强行在纯RDD代码中适配会需要额外新增SparkSession初始化逻辑,违背原有代码结构保留的要求,基于内存停用词集合的行内处理方案零额外依赖、完全适配现有代码
修正后的可运行代码
val sc = new SparkContext(conf)

val tweetsPath = args(0)
val outputDataset = args(1)

val tweetsRaw: RDD[String] = sc.textFile(tweetsPath)

val stopWords = Array("a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your","ain't","aren't","can't","could've","couldn't","didn't","doesn't","don't","hasn't","he'd","he'll","he's","how'd","how'll","how's","i'd","i'll","i'm","i've","isn't","it's","might've","mightn't","must've","mustn't","shan't","she'd","she'll","she's","should've","shouldn't","that'll","that's","there's","they'd","they'll","they're","they've","wasn't","we'd","we'll","we're","weren't","what'd","what's","when'd","when'll","when's","where'd","where'll","where's","who'd","who'll","who's","why'd","why'll","why's","won't","would've","wouldn't","you'd","you'll","you're","you've")
// 停用词转哈希集合并统一小写,优化匹配效率
val stopWordSet = stopWords.map(_.toLowerCase).toSet

// 第一步:严格筛选两个城市开头的推文
val cityOriginTweets = tweetsRaw.filter( tweetLine =>
  tweetLine.startsWith("San Francisco") || tweetLine.startsWith("Chicago")
)

// 第二步:逐行移除停用词,保留完整推文结构
val cleanTxt = cityOriginTweets.map( tweetLine => {
  // 按空白符拆分词汇,避免拆碎带撇的缩写类停用词
  val rawWords = tweetLine.split("\\s+")
  // 清理每个词首尾的标点,保留词内部的撇号兼容缩写类停用词
  val cleanedWords = rawWords.map(_.replaceAll("^[^a-zA-Z']+|[^a-zA-Z']+$", ""))
  // 过滤空值与停用词后,拼接回完整文本行
  cleanedWords.filter( word => 
    word.nonEmpty && !stopWordSet.contains(word.toLowerCase)
  ).mkString(" ")
})

cleanTxt.saveAsTextFile(outputDataset)

内容的提问来源于stack exchange,提问作者Jonathan Quendoz

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.15 21:54:08