如何将基于IMDB评论训练的机器学习模型存储到Redis？

Great question! Storing your pre-trained IMDB sentiment model in Redis is a perfect way to slash inference latency by eliminating repeated disk I/O delays. Let’s walk through the exact implementation steps, tailored to common ML frameworks like scikit-learn and Keras:

Why Redis? #

Redis is an in-memory data store, so once your model is loaded into Redis, retrieving it is nearly instant. This eliminates the slow disk reads that are likely causing your 3-minute per-task delay (especially if you’re reloading the model from disk for each classification run).

Step 1: Install Dependencies #

First, make sure you have the required libraries installed:

pip install redis joblib scikit-learn  # For scikit-learn models
# Or for Keras/TensorFlow:
pip install redis tensorflow

Step 2: Serialize and Store Your Model #

You’ll need to convert your model into a byte string to store it in Redis. Choose a serialization method that works for your framework:

Example 1: Scikit-learn Model (e.g., SVM, Logistic Regression) #

Most scikit-learn models work best with joblib (more efficient than pickle for large objects):

import redis
import joblib
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import SVC

# Assume you already have your trained model and vectorizer
# model = SVC()  # Trained on IMDB data
# vectorizer = TfidfVectorizer()  # Fitted on training text

# Connect to Redis (default localhost:6379)
redis_conn = redis.Redis(host="localhost", port=6379, db=0)

# Serialize model and vectorizer to bytes
model_bytes = joblib.dumps(model)
vectorizer_bytes = joblib.dumps(vectorizer)

# Store in Redis with descriptive keys
redis_conn.set("imdb_sentiment_model", model_bytes)
redis_conn.set("imdb_tfidf_vectorizer", vectorizer_bytes)

Example 2: Keras/TensorFlow Model #

For deep learning models, use an in-memory buffer to serialize the model:

import redis
import tensorflow as tf
from io import BytesIO

# Assume you have a trained Keras model
# model = tf.keras.models.load_model("trained_imdb_model.h5")

# Connect to Redis
redis_conn = redis.Redis(host="localhost", port=6379, db=0)

# Serialize model to bytes
buffer = BytesIO()
model.save(buffer)
buffer.seek(0)
model_bytes = buffer.read()

# Store in Redis
redis_conn.set("imdb_keras_sentiment_model", model_bytes)

Step 3: Retrieve and Use the Model for Inference #

Now, load the model once from Redis (not every time you run a task) and reuse it for all 3 classification runs:

Scikit-learn Inference #

import redis
import joblib

# Reuse the same Redis connection (don't reconnect every time!)
redis_conn = redis.Redis(host="localhost", port=6379, db=0)

# Retrieve and deserialize model components
model_bytes = redis_conn.get("imdb_sentiment_model")
vectorizer_bytes = redis_conn.get("imdb_tfidf_vectorizer")

model = joblib.loads(model_bytes)
vectorizer = joblib.loads(vectorizer_bytes)

# Batch classify comments (optimized for speed)
def classify_batch(comments):
    transformed_text = vectorizer.transform(comments)
    return model.predict(transformed_text)

# Run your 3 classification tasks
for _ in range(3):
    your_10_comments = ["This movie blew me away!", "Total garbage...", ...]  # Your 10 comments
    predictions = classify_batch(your_10_comments)
    # Process predictions here

Keras Inference #

import redis
import tensorflow as tf
from io import BytesIO

redis_conn = redis.Redis(host="localhost", port=6379, db=0)

# Retrieve and load model
model_bytes = redis_conn.get("imdb_keras_sentiment_model")
buffer = BytesIO(model_bytes)
buffer.seek(0)
model = tf.keras.models.load_model(buffer)

# Preprocess and classify (adjust preprocessing to match your model)
def preprocess_and_classify(comments):
    # Add your tokenization/padding logic here
    processed_text = your_preprocessing_function(comments)
    return model.predict(processed_text)

# Run your 3 tasks
for _ in range(3):
    predictions = preprocess_and_classify(your_10_comments)

Pro Tips for Optimization #

• Persistent Connections: Keep your Redis connection open across inference runs instead of creating a new one each time.
• Redis Hashes: For better organization, store model components (model, vectorizer, tokenizer) in a single Redis hash:

  redis_conn.hset("imdb_sentiment_system", mapping={
      "model": model_bytes,
      "vectorizer": vectorizer_bytes
  })
  # Retrieve with: model_bytes = redis_conn.hget("imdb_sentiment_system", "model")
  

• Check Memory: Ensure your Redis instance has enough RAM to hold your model. Run redis-cli info memory to verify.

This approach should cut your total runtime from ~10 minutes to just seconds (minus the actual inference time for 30 comments), keeping users from dropping off while waiting.

内容的提问来源于stack exchange，提问作者Ahmad Raza