Handling Out-of-Order Events in Distributed Systems
Published on 11/3/2025
Handling Events Coming in an Unknown Order
TLDR: When building event-driven systems, you often face events arriving out of sequence. This article explores practical strategies including using summary events to reduce coupling, leveraging logical clocks with revision numbers, and implementing pending command queues to handle incomplete information gracefully.
Summary:
The challenge of out-of-order event delivery is fundamental to distributed systems. Unlike a single-threaded application where operations execute sequentially, distributed systems resemble a department store with multiple cash registers—each queue processes in order, but there's no guarantee across queues. The author frames this brilliantly: you know fresh fruit arrives Monday morning and dairy at noon, but delays mean dairy might arrive first. Is that a problem? Only if you're making strawberry milkshakes and need both ingredients.
The article tackles a concrete scenario: handling shopping cart events where items get added, removed, and eventually confirmed. When these events arrive scrambled—say you receive ItemRemovedFromCart before ItemAddedToCart—how do you know if data is missing or just delayed? The naive approach of using timestamps fails because wall-clock time doesn't tell you about gaps in your knowledge. Clock drift between nodes makes physical timestamps unreliable for ordering.
The first and most powerful strategy is separating internal and external events. Your shopping cart module tracks every granular change internally (every add, every remove), but external systems like the kitchen only need a summary: "Cart confirmed with these items." This summary event pattern acts as an anti-corruption layer, letting you change internal workflows without breaking other teams. You can even use different infrastructure—maybe AWS SQS internally and Kinesis for cross-module communication—with different scaling, security, and retention policies.
When you can't control the event producer, the article proposes using logical clocks—monotonic, gapless revision numbers tied to optimistic concurrency. Each state change increments a version, and that version travels with the event. Now when you receive events 2, 1, 5, 6, you know you're missing 3 and 4. The implementation maintains a list of pending commands in your data model, essentially treating it like a local git repository. Commands accumulate until you can apply a consecutive sequence, then you flush them and update the last processed revision.
For architects and teams, this highlights a critical tradeoff between granular event streams (high fidelity, complex ordering) and summary events (simpler consumption, some detail loss). The pending command pattern adds implementation complexity but preserves eventual consistency without blocking. However, the author correctly notes that proper domain modeling—defining essential events with complete information—prevents most ordering headaches. If you're correlating multiple records, you're often fighting poor boundaries.
Key takeaways:
- Physical timestamps don't solve ordering problems because they indicate when operations happen, not what's missing
- Summary events (external API) versus granular events (internal) dramatically reduce ordering complexity for downstream consumers
- Logical clocks with revision numbers from optimistic concurrency provide gapless, monotonic ordering per aggregate
- Pending command queues allow you to accumulate out-of-order events and apply them once consecutive sequences arrive
- Proper bounded context modeling is the first line of defense—if ordering is constantly painful, your boundaries are probably wrong
Tradeoffs:
- Summary events reduce ordering complexity for consumers but sacrifice granular change history that some analytics use cases need
- Pending command pattern ensures eventual consistency but increases storage overhead and code complexity compared to immediate processing
- Using revision-based logical clocks requires optimistic concurrency control at the producer, which adds retry logic and conflicts under high contention
- Separating internal and external event topics improves isolation but requires maintaining an enrichment/translation layer and doubles infrastructure