Awesome question! Let's walk through the best distributed storage options to replace your local HashMap in this Storm-based vehicle data merging scenario.

Option 1: Trident (Storm's Built-in State Management) #

Trident is Storm's high-level API that comes with native distributed state management—this makes it a seamless fit for your use case.

• Why it works for you:

  • No extra third-party services needed: It’s tightly integrated with Storm, so you avoid the overhead of managing another system.
  • Partitioned state support: You can partition your data by VehicleRegistrationNo, meaning each vehicle’s location history is managed by a specific Trident node. This keeps read/write latency low since you’re not bouncing data across unnecessary nodes.
  • Easy to maintain "last 3 location entries": Use Trident’s mapState with a custom state updater to maintain a fixed-size queue for each vehicle. Every time a new location comes in from S1, you add it to the queue and automatically drop the oldest entry once you hit 3 records.
  • Built-in fault tolerance: Trident handles state snapshots and failure recovery out of the box, so you don’t have to worry about losing location data if a node goes down.

• Quick implementation tips:

  • Refactor your topology to use TridentTopology instead of a standard Storm topology.
  • Partition the S1 stream by VehicleRegistrationNo, then use persistentAggregate or mapState to track each vehicle’s recent locations.
  • Partition the S2 stream the same way, so you can directly query the corresponding state partition for the latest location data to merge with speed.

Option 2: Redis (High-Performance Distributed Key-Value Store) #

If you need maximum speed and scalability, Redis is a tried-and-true choice for real-time stream processing scenarios.

• Why it works for you:

  • Blazing fast read/write speeds: Redis can handle hundreds of thousands of operations per second, which is more than enough for high-volume vehicle data.
  • Perfect data structure for your use case: Redis Lists let you easily store and trim the last 3 location entries. When you get a new location from S1, use LPUSH to add it to the list, then LTRIM to keep only the most recent 3 entries—all in one atomic operation.
  • Scalable: Redis Cluster lets you scale horizontally as your vehicle count grows, so you won’t hit bottlenecks.
  • Easy Storm integration: There are mature Storm-Redis components (like the official storm-redis library) that let you add Redis read/write logic to your existing Bolts with minimal code changes.

• Quick implementation tips:

  • Use VehicleRegistrationNo as your Redis key, and store location data (with timestamp) as a JSON-serialized value in a List.
  • In your S1 processing logic: Run LPUSH {vehicleId} "{locationData}" followed by LTRIM {vehicleId} 0 2 to cap the list at 3 entries.
  • In your S2 processing logic: Use LRANGE {vehicleId} 0 0 to grab the latest location entry (assuming S1 messages arrive in timestamp order), then merge it with the speed data before sending to the next Bolt.

Which Should You Choose? #

Let’s narrow it down based on your priorities:

• Go with Trident if: You want to keep your stack simple, avoid extra infrastructure, and leverage Storm’s native fault tolerance. It’s ideal if you’re willing to refactor your topology to use Trident’s high-level API.
• Go with Redis if: You need maximum performance, plan to scale to a huge number of vehicles, or want a shared data store that other systems can also access. It’s also a great pick if you don’t want to rewrite your existing Storm topology—you can just swap out the HashMap for Redis calls.

Bonus: Minimal Change Approach #

If you’re not ready to refactor your entire topology to Trident, replacing the local HashMap with Redis is the fastest way to fix the multi-node inefficiency. Just update your MergeS1andS2 Bolt to use Redis for storing/retrieving location data instead of the in-memory map—this requires minimal code changes and gives you immediate distributed state support.

内容的提问来源于stack exchange，提问作者SheCodes