使用keras-text的train_val_split/update_test_indices时遇ValueError求助

Let's dive right into your issue—first off, the root cause here is mismatched sample counts between your input features X (shape (21643,1)) and labels y (shape (21634,)). You've got 9 more data points in X than there are corresponding labels in y, which is why methods like update_test_indices() are throwing a ValueError—the library expects every input sample to have a matching label.

Here's how to resolve this step by step:

• First, verify the mismatch explicitly
  Run these quick checks to confirm the discrepancy:

  print(f"Number of samples in X: {X.shape[0]}")
  print(f"Number of labels in y: {y.shape[0]}")
  

  This will confirm the 9-sample gap we suspect.

• Align your features and labels
  You have two main options here:

  1. Remove unlabeled samples from X: If you don't have access to the missing 9 labels, filter X to only keep indices that have corresponding entries in y. For example:

     # Assuming X is a numpy array, trim it to match y's length
     X_aligned = X[:y.shape[0]]
     # If the mismatch is scattered, you'll need to track original indices to identify unlabeled samples
     

  2. Add missing labels to y: If you can recover the 9 missing labels (e.g., from your original dataset source), append them to y so its length matches X. Double-check that the order of labels corresponds correctly to the input samples.

• Reinitialize the dataset with aligned data
  Once X_aligned and y have the same number of samples (both should be 21634 or 21643, depending on your choice), re-run the dataset initialization:

  from keras_text.data import dataset
  
  # Use aligned X and y here
  data = dataset(X_aligned, y, tokenizer=WordTokenizer())
  

  Now when you call methods like update_test_indices(), the library won't hit a sample count mismatch error.

• Prevent this in future workflows
  A good practice is to validate sample counts immediately after loading data:

  assert X.shape[0] == y.shape[0], f"Sample count mismatch: X has {X.shape[0]}, y has {y.shape[0]}"
  

  This assertion will catch mismatches early, before you spend time on tokenization and other preprocessing steps.

内容的提问来源于stack exchange，提问作者Daan Wiltenburg