如何在预测阶段复用Keras one_hot函数并保证编码结果一致？

Ah, I've hit this exact problem before! The trouble with keras.preprocessing.text.one_hot is that it relies on hash-based encoding that's not stable across script runs by default—let me break down why this happens and how to fix it properly:

Why Your Encodings Keep Changing #

The root cause boils down to two key points:

• Python’s built-in hash() function (which one_hot uses under the hood via hashing_trick) uses a random seed that resets every time you start a new Python process. That means the same string will get a different hash value on different runs, leading to inconsistent one-hot outputs.
• Unlike a proper vocabulary-based encoder, one_hot doesn’t store a persistent mapping of words to integers. It computes the hash on the fly every time you call it, so there’s no way to "reuse" the same encoding logic unless you force stability.

Solution 1: Use Tokenizer (Highly Recommended) #

The Tokenizer class is designed explicitly for this scenario—it builds a fixed vocabulary from your text data, which you can save and reuse later for perfectly consistent encodings. Here’s how to implement it step by step:

Step 1: Train and Save the Tokenizer #

When you first process your training data, initialize and fit the tokenizer, then save it for future use:

from keras.preprocessing.text import Tokenizer
import pickle

# Initialize tokenizer with your desired vocabulary size (match your original 100)
tokenizer = Tokenizer(num_words=100)

# Fit it on all unique words you expect to encounter (use your training text here)
training_texts = ['hello', 'world', 'hi', 'nice', 'to', 'meet', 'you']
tokenizer.fit_on_texts(training_texts)

# Save the tokenizer to a file so you can load it during prediction
with open('my_tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

Step 2: Load and Reuse the Tokenizer for Prediction #

When you need to encode text during prediction, just load the saved tokenizer and use it:

import pickle
from keras.preprocessing.text import Tokenizer

# Load the saved tokenizer
with open('my_tokenizer.pickle', 'rb') as handle:
    tokenizer = pickle.load(handle)

# Encode your text consistently every time
test_text = ['hello world']
encoded_sequences = tokenizer.texts_to_sequences(test_text)
print(encoded_sequences)  # Same output every single run

# If you need one-hot style binary vectors instead of integer sequences:
one_hot_vectors = tokenizer.texts_to_matrix(test_text, mode='binary')

This approach avoids hash collisions entirely and gives you full control over your vocabulary—definitely the way to go for production or any scenario where consistency matters.

Solution 2: Force Stable Hashing (Workaround, Not Ideal) #

If you absolutely have to stick with one_hot for some reason, you can override the hash function to use a stable, non-random method. The hashing_trick function (which one_hot wraps) lets you pass a custom hash function. Here’s an example using md5 for consistent results:

from keras.preprocessing.text import hashing_trick
import hashlib

def stable_hash(word):
    # Use MD5 to generate a fixed hash value for each word
    hash_digest = hashlib.md5(word.encode('utf-8')).hexdigest()
    return int(hash_digest, 16)

# Create a stable version of one_hot
def stable_one_hot(texts, vocab_size):
    return [hashing_trick(text, vocab_size, hash_function=stable_hash) for text in texts]

# Test it—this will produce the same encoding every run
print(stable_one_hot(['hello', 'world', 'hi'], 100))

⚠️ Heads up: Hash-based encoding still carries the risk of hash collisions (different words getting the same integer encoding), which Tokenizer avoids entirely by using an explicit vocabulary. This is a quick workaround, not a robust long-term solution.

内容的提问来源于stack exchange，提问作者Hyeon Gyu Jang