Elasticsearch高频更新产品数据建模方案咨询

Core Problem Recap #

Your current single-index approach suffers from:

• Severe base data redundancy (duplicated in every daily KPI document) leading to bloated index size.
• Updating the product-family field requires modifying millions of historical KPI documents, causing system lag and query timeouts.

You’re evaluating two solutions: Parent-Child Relationships and Nested Fields. Below is a detailed comparison tailored to your requirements:

Option 1: Parent-Child Relationship (Single Index) #

Structure #

• Parent Documents: Store static base data (name, category, URL, product attributes, product-family). One parent per product.
• Child Documents: Store daily KPI data (date, daily sales, daily price, daily views). One child per product per day.
• Use Elasticsearch’s join field type to define the parent-child relationship within a single index.

Pros #

1. Eliminates Base Data Redundancy: Base data is stored once per product, drastically reducing index size.
2. Efficient product-family Updates: Only the parent document needs to be updated when product-family changes—no need to touch thousands of historical KPI entries. This perfectly aligns with your daily update requirement for thousands of products.
3. High Indexing Rate for Daily KPI: Adding new daily KPI data only requires indexing new child documents, no updates to existing parent/child docs. This minimizes indexing overhead and maintains efficiency even for 1M products.
4. No Cross-Index Join Overhead: Unlike splitting into separate indices, all data resides in one index, avoiding the backend load of cross-index associations.

Cons & Mitigations #

• Increased Query Latency: Parent-child joins are slower than nested queries. Mitigate this with:
  • Routing: Use the product ID as the routing key for both parent and child documents to ensure they’re stored on the same shard, reducing join overhead.
  • Optimize Queries: Use has_child/has_parent with score_mode: none if scoring isn’t required, and apply filters early to reduce the number of documents involved in joins.
  • Cluster Resources: Allocate sufficient CPU and memory to handle join operations—modern Elasticsearch versions have optimized parent-child performance significantly.

Option 2: Nested Fields (Single Index) #

Structure #

• Each document represents a product, with static base data at the top level and a nested array containing all daily KPI entries.

Pros #

1. No Base Data Redundancy: Same as parent-child—base data is stored once per product.
2. Fast Queries & Aggregations: No joins required, so queries combining base data and KPI are faster than parent-child.
3. Efficient product-family Updates: Use partial updates (_update API with doc parameter) to modify only the product-family field without reindexing the entire document.

Cons #

• Daily KPI Indexing Overhead: Adding a new daily KPI entry requires updating the entire product document to append to the nested array. For 1M products, this means 1M full document updates daily—each modifying a document with 730+ nested entries (for 2 years of data). This will strain cluster resources (disk I/O, CPU, segment merging) and degrade indexing performance over time.

Final Recommendation #

Choose Parent-Child Relationship for your use case. Here’s why:

• It directly solves your two core pain points (redundancy and product-family update inefficiency) without introducing the daily reindexing load of nested fields.
• The query latency trade-off is manageable with proper routing and query optimization, and avoids the cross-index overhead of splitting indices.
• It aligns perfectly with your requirement to maintain high indexing rates for daily KPI data.

If query latency is absolutely critical and your cluster can’t handle parent-child joins, nested fields could be an alternative—but only if you can accept the significant daily indexing overhead and have sufficient cluster resources to support it.

内容的提问来源于stack exchange，提问作者Tomer.Ov