如何在A/B测试后衡量推荐引擎模型的性能？

Great question—this is something I’ve walked through with multiple teams when adding recommendation engines to well-established systems. Since your system’s already stable, the goal is to pick a model that moves the needle on business goals without breaking existing trust or performance. Let’s break down the key metrics and how to use them:

These are the metrics that directly tie to your system’s purpose—they should be your top priority because they reflect whether the recommendation engine is actually adding value:

• Click-Through Rate (CTR) / Conversion Rate: The most immediate signal—what percentage of users click on or convert from recommended content? Compare this to your control group (existing system) and use statistical tests like chi-squared test or t-test to confirm the difference isn’t due to random chance. For example, if you’re running an e-commerce system, track the conversion rate of recommended products vs. non-recommended ones.
• User Retention & Engagement: Does the model keep users coming back? Look at 7-day/30-day retention rates, weekly active users, or average session duration. A model that boosts short-term CTR but tanks long-term retention is a bad bet for a stable system.
• Revenue Contribution: If your system has direct revenue ties (e.g., e-commerce GMV, ad eCPM), calculate the revenue generated from recommended content, plus per-user revenue changes. For example, does the new model increase average order value from recommended items?

These measure how well the model understands user preferences—they’re the backbone of why recommendations work:

• Precision: The percentage of recommended items that a user actually interacts with (clicks, buys, etc.). If you recommend 10 items and 3 get clicks, precision is 30%. This tells you if your recommendations are hitting the mark.
• Recall: The percentage of items a user was interested in that your model actually recommended. For example, if a user bought 5 items in a category, and 2 of them were in your recommendations, recall is 40%. Balance this with precision—high recall but low precision means users see too many irrelevant items.
• MAP (Mean Average Precision): This accounts for ranking quality—relevant items at the top of the list are more valuable than those at the bottom. It averages precision scores across all users, weighting higher positions more heavily.
• NDCG (Normalized Discounted Cumulative Gain): Similar to MAP, but lets you assign different weights to user actions (e.g., a purchase is worth more than a click). This is great for aligning model performance with your business’s action priorities.

These prevent your model from optimizing for short-term gains at the cost of long-term user trust:

• Diversity: Avoid "filter bubbles" by measuring how varied your recommendations are. For example, track the percentage of unique categories in each user’s recommendation list. A model that only shows one type of content will bore users over time.
• Novelty: Are you introducing users to new content they haven’t seen before? Calculate the percentage of recommended items that a user has never interacted with. This helps keep the experience fresh and drives discovery.
• Fairness: Ensure your model doesn’t favor a small subset of content (e.g., only popular products) or ignore niche options. Compare the exposure rate of different content categories/sellers to their overall availability in your system.
• Direct User Feedback: Don’t rely solely on behavioral data—add a "Not Interested" button or short surveys to capture subjective feedback. High rates of "Not Interested" clicks are a red flag, even if CTR looks good.

Since your system is already stable, you can’t let the new recommendation engine degrade performance:

• Latency: The time it takes for the model to return recommendations. If your existing system responds in 100ms, the new model should stay within a similar threshold (e.g., <200ms) to avoid frustrating users.
• Throughput: Can the model handle peak traffic? Test its ability to process requests at your system’s maximum concurrent load—aim for 99.9% success rate under stress.
• Resource Usage: Track CPU, memory, and storage consumption of the recommendation service. It shouldn’t hog resources that your existing system depends on.

A Quick Practical Note #

No single metric tells the full story. For example, a model with high CTR but low diversity might look great at first, but it’ll hurt retention over time. Also, make sure your A/B test runs for at least one full user behavior cycle (e.g., a week for e-commerce) and has a large enough sample size to avoid skewed results.

内容的提问来源于stack exchange，提问作者Gregorius Edwadr