Spring Boot应用Elasticsearch内存节点客户端实现合理性及最佳实践咨询

Hey there! Let's dive into your current Elasticsearch implementation and address your questions clearly.

i) Is your current Elasticsearch setup ideal? Recommendations for improvement #

Your current approach using in-memory Elasticsearch nodes in each Spring Boot instance, synced via JMS topics, is not ideal for a production environment—here's why:

• In-memory node limitations: In-memory Elasticsearch is built for local testing, not production. It doesn’t persist data (so all search data is lost if an instance restarts), has strict memory constraints (scaling with growing data becomes impossible), and lacks the high availability features of a dedicated cluster.
• JMS sync risks: While JMS can propagate DB2 changes, maintaining consistency across 4 instances is fragile. You might face issues like message loss, out-of-order message consumption, or delayed syncs—all of which lead to inconsistent search results. Coordinating 4 separate in-memory stores also adds unnecessary complexity to your architecture.

Recommended alternative approach: #

1. Deploy a dedicated Elasticsearch cluster
   • Set up a multi-node production cluster (minimum 3 data nodes for high availability) with persistent storage. This eliminates the need to sync in-memory instances and provides built-in replication, failover, and scalability.
2. Use a reliable data sync mechanism
   • Change Data Capture (CDC): Tools like Debezium can directly listen to DB2's transaction logs to capture real-time changes, then push them to Elasticsearch. This is more reliable than JMS because it’s based on database-level logs (no risk of missing changes) and ensures ordered processing.
   • If you stick with JMS: Ensure messages are persistent, implement retry logic with exponential backoff, and make all Elasticsearch update operations idempotent (e.g., use consistent document IDs tied to DB2 primary keys to avoid duplicate or conflicting updates).
3. Connect your Spring Boot apps to the cluster
   • Use Spring Data Elasticsearch's RestClient or the official RestHighLevelClient to connect your apps to the dedicated ES cluster, instead of embedding in-memory nodes.

ii) Elasticsearch Best Practices for Production #

Here are key best practices to follow:

• Cluster & Node Configuration
  • Always use dedicated data nodes (avoid running ES on the same server as your app or DB). For high availability, use at least 3 data nodes with 1 replica per primary shard.
  • Set appropriate heap size: Allocate 50% of the node's RAM to ES heap (capped at 32GB to avoid JVM garbage collection issues), leave the other 50% for the OS and filesystem cache.
• Index Design
  • Define explicit mappings instead of relying on dynamic mapping. This prevents unexpected field types (e.g., strings being mapped as both text and keyword unnecessarily) and improves query performance.
  • Keep primary shard size between 20-50GB. Too many small shards add cluster overhead; too few large shards make recovery slow.
  • Use index aliases to manage your indices. This lets you switch between old/new indices (e.g., after reindexing) without changing application code.
• Data Sync & Ingestion
  • Make all write operations idempotent. Use unique, stable document IDs so retries don't create duplicate or corrupted documents.
  • Use the Bulk API for batch updates instead of single-document requests. This reduces network overhead and improves ingestion speed.
• Query Optimization
  • Avoid leading wildcards (e.g., *query) in full-text searches—they force a full index scan and kill performance. Use prefix queries only if absolutely necessary, and consider edge n-grams for autocomplete instead.
  • Use filter clauses for non-scoring conditions (e.g., status checks). Filter results are cached, which speeds up repeated queries.
  • For deep pagination, use search_after instead of from/size. The latter becomes slow as you go to higher page numbers because ES has to fetch and discard all previous results.
• Monitoring & Maintenance
  • Enable Elasticsearch monitoring (via built-in features or tools like Metricbeat) to track cluster health, node CPU/memory usage, query latency, and ingestion rates.
  • Implement Index Lifecycle Management (ILM) to automate tasks like deleting old data, archiving indices, or rolling over to new indices when they reach a size/age threshold.
  • Never use in-memory nodes in production—reserve them for local development and testing only.

内容的提问来源于stack exchange，提问作者mysap