Splunk技术问询：查找特定字段变更事务与乱序事件

Alright, let's tackle your two Splunk use cases step by step, using your provided sample logs as a reference. I'll share practical search queries and break down how they work so you can adapt them to your system.

1. Find Transactions Where a Specific Field (Block) Changed #

Your goal here is to identify scenarios where the Block value in a BlockChange event doesn't match the value in its corresponding BlockChangeConfirmed event (for the same ScenarioId). Here's how to do it:

Splunk Search Query #

index=<your-index-name> sourcetype=<your-sourcetype> (Event=BlockChange OR Event=BlockChangeConfirmed)
| stats values(Event) as event_types, values(Block) as block_values, values(Time) as timestamps by ScenarioId
| where mvcount(event_types)=2 AND block_values[1] != block_values[2]
| eval BlockChange_Block=block_values[mvindex(event_types, "BlockChange")], 
       BlockChangeConfirmed_Block=block_values[mvindex(event_types, "BlockChangeConfirmed")],
       BlockChange_Time=timestamps[mvindex(event_types, "BlockChange")],
       BlockChangeConfirmed_Time=timestamps[mvindex(event_types, "BlockChangeConfirmed")]
| table ScenarioId BlockChange_Time BlockChange_Block BlockChangeConfirmed_Time BlockChangeConfirmed_Block

How This Works #

• First, we filter events to only include BlockChange and BlockChangeConfirmed entries—no need to process irrelevant events.
• We group events by ScenarioId using stats, collecting all event types, block values, and timestamps for each scenario.
• The where clause ensures we only keep scenarios that have both event types, and where the two block values don't match.
• We use mvindex to map each event type to its corresponding block value and timestamp, making the results easy to read.
• Finally, we format everything into a clean table for quick analysis.

Expected Results from Your Sample Logs #

This query will return two entries:

• ScenarioId=7: BlockChange had Block=A-A, while BlockChangeConfirmed had Block=1-7
• ScenarioId=2: BlockChange had Block=E-6, while BlockChangeConfirmed had Block=B-2

2. Identify Out-of-Order Events in Splunk #

Out-of-order events happen when an event's timestamp doesn't align with its logical sequence (like Id increasing but timestamp decreasing, or events in the same ScenarioId appearing out of time order). Here are two reliable ways to detect this:

Method 1: Detect Time Backsliding per ScenarioId #

Use this to find events in the same scenario that come earlier in time than the previous event:

index=<your-index-name> sourcetype=<your-sourcetype>
| sort 0 ScenarioId Time
| streamstats current=f last(Time) as previous_time by ScenarioId
| where strptime(Time, "%H:%M.%S") < strptime(previous_time, "%H:%M.%S")
| eval time_difference_seconds=strptime(Time, "%H:%M.%S") - strptime(previous_time, "%H:%M.%S")
| table ScenarioId Id previous_time Time time_difference_seconds Event

How This Works #

• We sort events first by ScenarioId, then by Time to get a baseline sequence.
• streamstats grabs the timestamp of the previous event in the same scenario.
• We filter for events where the current timestamp is earlier than the previous one (converting time strings to epoch for comparison).
• We calculate the time difference to show how far out of order the event is.

Method 2: Detect Mismatch Between Id and Timestamp #

If your Id field is strictly incremented (so higher Id should mean later timestamp), use this query to spot discrepancies:

index=<your-index-name> sourcetype=<your-sourcetype>
| sort 0 Id
| streamstats current=f last(Time) as previous_time by ScenarioId
| where strptime(Time, "%H:%M.%S") < strptime(previous_time, "%H:%M.%S")
| table ScenarioId Id previous_time Time Event

Note on Your Sample Logs #

Your provided logs don't have out-of-order events, but if something like Id=15 had a timestamp earlier than Id=14, this query would flag it immediately.

内容的提问来源于stack exchange，提问作者ilitirit