It sounds like you’re hitting a wall with text-only approaches because overlapping content between classes is masking meaningful patterns. Let’s break down alternative strategies that combine structural, non-text, and even raw binary features to get better results:

First: Don’t Write Off NLP Entirely (Fix Your Pipeline) #

Your initial NLP attempts might have failed because you weren’t leveraging the XML structure or modern text representations:

• Stop ignoring XML structure: Converting XML to docx throws away critical metadata like tag names, nesting depth, and attribute presence. For example, Class A might consistently use <confidential> tags while Class B doesn’t. Use xml.etree.ElementTree (Python) to parse and extract structural metrics (count of specific tags, tree depth, number of child nodes) alongside text.
• Replace Naive Bayes with contextual embeddings: Bag-of-words or ordinal encoding can’t capture nuanced context. Try fine-tuning a small pre-trained model like distilbert-base-uncased on your text data—even with overlapping content, it will pick up subtle contextual differences (e.g., how words are used in different XML sections).
• Focus on relevant text sections: Instead of using all text, extract only specific parts of the XML (like <description> or <content> tags) that are more likely to differ between classes.

Approach 2: Add Non-Text Features #

Text isn’t the only signal in your files. Combine these with text embeddings for a richer feature set:

• File size: Yes, this is a quick win! Normalize it (e.g., log-transform) since raw sizes can vary drastically, then add it as a numerical feature. Plot size distributions for each class first—if there’s a clear gap, this alone could help.
• XML structural features: Extract metrics like:
  • Number of unique tag types per file
  • Maximum nesting depth of the XML tree
  • Presence/absence of critical attributes (e.g., status="active")
  • Ratio of text content to total file size
• Text metadata: Word count, average word length, frequency of special characters, or uppercase/lowercase ratios—small differences here can add up.

Approach 3: Feed Raw Binary Data to a Neural Network #

Absolutely, you can use the raw binary content of your XML files as input to a neural network. This captures both text and structural patterns in one go:

• Preprocess binary data: Load each file as a sequence of bytes (0-255 values). Pad/truncate all files to a fixed length (e.g., 10KB if most files fit) or use dynamic padding with models that support variable lengths.
• Choose the right model:
  • CNNs: Conv1D layers excel at capturing local patterns in byte sequences (like XML tag patterns or header differences). Pair with global average pooling to handle variable lengths.
  • Byte-level transformers: Models like ByteNet or even a simple LSTM can learn long-range patterns in the binary data, though they’re more computationally heavy.
• Example workflow (Python/PyTorch):

  import torch
  from torch.utils.data import Dataset
  
  class BinaryFileDataset(Dataset):
      def __init__(self, file_paths, labels, max_len=10240):
          self.file_paths = file_paths
          self.labels = labels
          self.max_len = max_len
  
      def __getitem__(self, idx):
          with open(self.file_paths[idx], 'rb') as f:
              bytes_data = f.read(self.max_len)
          # Pad with zeros if shorter than max_len
          padded = bytes_data.ljust(self.max_len, b'\x00')
          return torch.tensor(list(padded), dtype=torch.float32), torch.tensor(self.labels[idx])
  

Hybrid Approach: Combine All Signals #

The best results often come from merging multiple feature types:

1. Extract text embeddings using a pre-trained transformer.
2. Compute structural/non-text features and normalize them.
3. Concatenate the embeddings with the numerical features, then feed into a small fully connected classifier.
   Or, use a multi-modal model that processes binary data and text separately, then fuses their outputs.

Practical Tips to Get Started #

• Analyze your data first: Spend time exploring class differences—plot size distributions, count tag frequencies, and run keyword frequency analysis. This will tell you which features are worth prioritizing.
• Start small: Test simple models (like a CNN on binary data + logistic regression) before moving to complex transformers. This helps you iterate quickly and identify what works.
• Use cross-validation: Split your 10k samples into train/validation/test sets (70/15/15) to avoid overfitting and get reliable performance metrics.

内容的提问来源于stack exchange，提问作者hassansid