Sklearn训练与验证特征维度不匹配的ValueError问题求助

在学校项目中使用Sklearn构建文本分析自定义Pipeline，已为自定义Transformer添加日志，但遇到报错：
ValueError: X has 7930 features, but SelectKBest is expecting 25050 features as input.

我的处理流程：

• 预处理前提取特征
• 执行预处理
• 预处理后提取特征

训练集特征维度为(334,25050)，验证集仅为(72,7930)。已在训练阶段拟合Pipeline，调用.predict处理验证集时，不解为何Sklearn未自动为缺失特征补0，恳请帮忙排查原因。

自定义Transformer代码 #

class FeatureExtractorBeforePreprocessing(BaseEstimator, TransformerMixin):
    """
    Extracts all features combined before preprocessing.
    """
    def __init__(self, stopWords=False, errorDetector=False, punctuationFrequency=False, sentenceLength=False):
        """
        Initialize the FeatureExtractorBeforePreprocessing.
        
        Args:
            stopWords (bool): Whether to include stop words as a feature.
            errorDetector (bool): Whether to include error detection as a feature.
            punctuationFrequency (bool): Whether to include punctuation frequency as a feature.
            sentenceLength (bool): Whether to include sentence length as a feature.
        """
        self.stopWords = stopWords
        self.errorDetector = errorDetector
        self.punctuationFrequency = punctuationFrequency
        self.sentenceLength = sentenceLength
        self.feature_union = []
        self.combined_transformers = None

        # Add transformers based on selected options
        if self.stopWords:
            self.feature_union.append(("stopWords", StopWords()))
        if self.errorDetector:
            self.feature_union.append(("errorDetector", ErrorDetector()))
        if self.punctuationFrequency:
            self.feature_union.append(("punctuationFrequency", PunctuationFrequency()))
        if self.sentenceLength:
            self.feature_union.append(("sentenceLength", SentenceLength()))
        if self.feature_union:
            self.combined_transformers = FeatureUnion(self.feature_union)
        
    def fit(self, X, y=None):
        # Fit the combined transformers if they exist
        if self.combined_transformers:
            self.combined_transformers.fit(X)
        return self

    def transform(self, X):
        if self.feature_union:
            logger.info("Extracting features before preprocessing..")
            combined_features = self.combined_transformers.transform(X)
            logger.info(f"Shape of combined features before preprocessing: {combined_features.shape}")
            return combined_features
        else:
            X_array = np.empty((len(X), 0))
            logger.info("Nothing to extract before preprocessing..")
            logger.info(f"Shape of combined features before preprocessing: {X_array.shape}")
            return X_array

class FeatureExtractorAfterPreprocessing(BaseEstimator, TransformerMixin):
   
    def __init__(self, config=config, textWordCounter=False, wordLength=False, vocabularySize=False):
        self.config = config
        self.textWordCounter = textWordCounter
        self.wordLength = wordLength
        self.vocabularySize = vocabularySize
        self.feature_union = []
        
        if self.textWordCounter:
            self.feature_union.append(("textWordCounter", TextWordCounter(self.config.getboolean("TextWordCounter","freqDist"), self.config.getboolean("TextWordCounter", "bigrams"))))
        
        if self.vocabularySize:
            self.feature_union.append(("vocabularySize", VocabularySize()))
        
        self.combined_transformers = FeatureUnion(self.feature_union)
        
    def fit(self, X, y=None):
        if self.combined_transformers:
            self.combined_transformers.fit(X)
        return self

    def transform(self, X):
        if self.feature_union:
            logger.info("Extracting features after preprocessing..")
            combined_features = self.combined_transformers.transform(X)
            logger.info(f"Shape of combined features after preprocessing: {combined_features.shape}")
            return combined_features
        else:
            X_array = np.empty((len(X), 0))
            logger.info("Nothing to extract before preprocessing..")
            logger.info(f"Shape of combined features after preprocessing: {X_array.shape}")
            return X_array

日志信息 #

2024-03-21 20:31:55,868 [INFO] Creating custom pipeline...
2024-03-21 20:31:55,868 [INFO] pipeline config: {'stopWords': False, 'errorDetector': False, 'punctuationFrequency': False, 'sentenceLength': False, 'textWordCounter': True, 'wordLength': False, 'vocabularySize': False, 'featureSelector': SelectKBest(k=10000, score_func=<function chi2 at 0x000001ACCFDC4040>)}
2024-03-21 20:31:55,868 [INFO] Creating pipeline...
2024-03-21 20:31:55,870 [INFO] Pipeline config: [('featureExtractionUnion', FeatureUnion(transformer_list=[('featureExtractionBeforePreprocessing',
                                FeatureExtractorBeforePreprocessing()),
                               ('afterPreprocessingPipeline',
                                Pipeline(steps=[('preprocessing',
                                                 Preprocessing()),
                                                ('featureExtractionAfterPreprocessing',
                                                 FeatureExtractorAfterPreprocessing(textWordCounter=True))]))])), ('featureSelection', SelectKBest(k=10000, score_func=<function chi2 at 0x000001ACCFDC4040>))]
2024-03-21 20:31:55,870 [INFO] Pipeline created.
2024-03-21 20:31:55,870 [INFO] Selected classifier: svm
2024-03-21 20:31:55,870 [INFO] Custom pipeline created.
2024-03-21 20:31:55,870 [INFO] UserConfigPipeline initialized.
2024-03-21 20:31:55,873 [INFO] Training custom model...
2024-03-21 20:31:55,873 [INFO] Nothing to extract before preprocessing..
2024-03-21 20:31:55,873 [INFO] Shape of combined features before preprocessing: (334, 0)
2024-03-21 20:31:55,873 [INFO] Preprocessing..
2024-03-21 20:31:55,874 [INFO] No preprocessing applied..
2024-03-21 20:31:55,874 [INFO] Returned list X of 334 texts.
2024-03-21 20:31:55,874 [INFO] Extracting features after preprocessing..
2024-03-21 20:31:55,960 [INFO] Extracting freqDict features..
2024-03-21 20:31:55,990 [INFO] Shape of freqDict features: (334, 25050)
2024-03-21 20:31:55,991 [INFO] Shape of combined features after preprocessing: (334, 25050)
2024-03-21 20:31:56,166 [INFO] Successfully trained custom model...
2024-03-21 20:31:56,166 [INFO] Validating model...
2024-03-21 20:31:56,166 [INFO] Predicting evaluation set...
2024-03-21 20:31:56,166 [INFO] Nothing to extract before preprocessing..
2024-03-21 20:31:56,166 [INFO] Shape of combined features before preprocessing: (72, 0)
2024-03-21 20:31:56,166 [INFO] Preprocessing..
2024-03-21 20:31:56,166 [INFO] No preprocessing applied..
2024-03-21 20:31:56,166 [INFO] Returned list X of 72 texts.
2024-03-21 20:31:56,166 [INFO] Extracting features after preprocessing..
2024-03-21 20:31:56,185 [INFO] Extracting freqDict features..
2024-03-21 20:31:56,195 [INFO] Shape of freqDict features: (72, 7930)
2024-03-21 20:31:56,196 [INFO] Shape of combined features after preprocessing: (72, 7930)

内容的提问来源于stack exchange，提问作者Hahanaki