Home / APIs & Automation / Event Streaming
StreamingAdvanced

Event Streaming

Event streaming is the continuous consumption of a sequence of events — ticks, fills, signals — where the consumer must handle ordering, at-least-once delivery with deduplication, and backpressure to process the stream correctly and safely.

Quick answer: Event streaming is the continuous consumption of a sequence of events — ticks, fills, signals — where the consumer must handle ordering, at-least-once delivery with deduplication, and backpressure to process the stream correctly and safely.

In simple words

Event streaming means your system reacts to a never-ending flow of small messages — a price ticked, an order filled, a signal fired — rather than asking for data in batches. The challenge is that this flow can arrive out of order, can deliver the same event twice, and can come faster than you handle it. Doing streaming right means designing for all three.

Purpose

Event streaming exists because markets are event-driven by nature; modelling the system as a consumer of events keeps live and backtest code aligned and lets components react the instant something happens, but only if delivery guarantees are respected.

Professional explanation

The event-driven model

An event-streaming architecture treats every meaningful occurrence as an immutable event on a stream: a market tick, an order acknowledgement, a fill, a risk-limit breach, a generated signal. Components subscribe and react, rather than polling. This mirrors how markets actually behave and, importantly, lets the same event-handling code drive both a live feed and a replayed historical feed during backtesting — a major reason event-driven design is favoured for trading systems. But adopting the model means inheriting the hard problems of distributed messaging: what order events arrive in, how many times, and what happens when you cannot keep up.

Delivery guarantees: at-least-once is the norm

Messaging systems offer one of three guarantees. At-most-once may drop events (unacceptable for a fill you must know about). Exactly-once is ideal but expensive and often only approximated. In practice most real streams — including broker feeds after a reconnect — give at-least-once: every event is delivered, but some may be delivered more than once, for example when an acknowledgement is lost and the producer re-sends. The consequence is unavoidable: a robust consumer must assume duplicates will happen and be built to tolerate them, because you cannot get at-least-once safety and also assume each event is unique.

Deduplication and idempotent consumption

The answer to at-least-once duplicates is idempotent consumption: processing the same event twice must have the same effect as processing it once. Two techniques dominate. First, deduplicate by a unique event id — keep a record of seen ids (within a bounded window) and ignore repeats. Second, make the handler naturally idempotent — for instance, setting a position to an absolute value from a snapshot rather than incrementing it, so a replayed event cannot double-count. For a fill event that increments position, dedup by fill id is essential; treating each delivered fill as new would silently inflate your recorded position and desynchronise you from the broker.

Ordering and sequence numbers

Events can arrive out of order — across a reconnect, across sharded connections, or through the messaging layer — and out-of-order processing corrupts state. If a cancel is processed before the placement it cancels, or an older tick overwrites a newer one, the system's view goes wrong. Streams therefore usually carry a monotonic sequence number or timestamp per partition, and the consumer uses it to detect gaps (a missing sequence means a dropped event, prompting a reconciliation) and to reject stale updates (ignore an event older than the last applied). Ordering guarantees typically hold only within a partition or a single connection, so any design that shards a symbol's events across connections must preserve per-symbol order deliberately.

Backpressure: when you cannot keep up

During volatile periods events can arrive faster than the consumer processes them. Without a strategy the buffer grows unboundedly, latency climbs, memory exhausts, and the producer may disconnect you — all at the worst possible moment. Backpressure is the set of techniques for handling this mismatch: bound the queue and decide what to do when it fills. Options include slowing the producer where the protocol allows, dropping or conflating stale data (for pure price ticks, only the latest matters, so older queued ticks can be collapsed), or shedding load by priority. What you must never do is silently let latency grow until the system acts on minutes-old prices. Separating a light network-read thread from the processing worker is the standard structural defence.

Consumer offsets, replay and recovery

Durable event systems (as opposed to ephemeral broker sockets) let a consumer track an offset — its position in the stream — and resume from there after a restart, replaying events it had not yet processed. This gives strong recovery but reintroduces duplicates (events after the last committed offset may be reprocessed), which is again why idempotent consumption matters. When you commit the offset relative to when you process the event determines your guarantee: commit before processing risks losing an event (at-most-once); commit after processing risks reprocessing (at-least-once). Trading systems almost always choose the latter and lean on deduplication, because losing a fill is worse than handling one twice.

Delivery guarantees

GuaranteeBehaviourTrade-off
At-most-onceMay drop eventsSimple but can miss a fill — unsafe
At-least-onceNever drops, may duplicateNeeds dedup; the practical default
Exactly-onceNo loss, no duplicateCostly, often only approximated

Practical example

Illustrative example (Indian market)

Suppose your system consumes an order-event stream and, after a reconnect, the broker re-delivers the last few events to guarantee none were missed. One fill for 75 units of Nifty arrives twice, carrying the same fill id. Because the consumer deduplicates by fill id, the second copy is ignored and the recorded position stays at one lot rather than jumping to two. Meanwhile a burst of ticks arrives faster than the strategy computes; the tick handler conflates them, keeping only the latest price per instrument so the strategy never acts on a stale queued tick. The numbers are illustrative of the mechanics, not a trade.

A broker WebSocket that re-sends recent order updates after a dropped connection is effectively an at-least-once stream, so an Indian intraday system must deduplicate fills by order/fill id and reconcile against the REST order book. Treating each re-delivered fill as new would silently double a Bank Nifty position in the system's books while the broker shows only one — a dangerous desync.

Advantages

  • Reacts to market and order events the instant they occur, matching how markets work
  • The same event-handling code can drive live trading and historical backtest replay
  • Idempotent, deduplicated consumption tolerates the duplicates at-least-once delivery brings
  • Backpressure handling keeps the system responsive and current under heavy load

Limitations

  • At-least-once delivery means duplicates are guaranteed to occur and must be handled
  • Out-of-order events can corrupt state unless sequence numbers are checked
  • Backpressure under bursts can exhaust memory or cause disconnection if unbounded
  • Exactly-once is expensive and usually only approximated, so you cannot rely on uniqueness
  • Correct offset/commit semantics are subtle and easy to get wrong, risking loss or reprocessing

Common mistakes

  • Assuming each event is unique and skipping deduplication under at-least-once delivery
  • Incrementing position on every delivered fill instead of deduping by fill id
  • Ignoring sequence numbers, letting out-of-order or stale events overwrite newer state
  • Letting an unbounded queue grow under backpressure until latency or memory blows up
  • Committing the offset before processing, risking silently lost events
  • Sharding one symbol's events across connections without preserving per-symbol order

Professional usage

Professional systems design consumers to be idempotent from the start — deduplicating by event id and preferring absolute-state updates over increments — and treat at-least-once as the baseline guarantee. They enforce per-partition ordering via sequence numbers, detect gaps to trigger reconciliation, and bound queues with an explicit backpressure policy (conflate stale ticks, shed by priority). Offsets are committed after successful processing so no event is lost, and stream health — lag, duplicate rate, gap count — is monitored as a core operational signal.

Key takeaways

  • Event streaming consumes a continuous flow of events, matching the event-driven nature of markets
  • Assume at-least-once delivery: duplicates will happen, so consume idempotently and dedup by event id
  • Check sequence numbers to preserve ordering and detect gaps rather than acting on stale events
  • Bound queues and handle backpressure — conflate or shed under load, never let latency grow silently

Frequently asked questions

What is event streaming?
It is the continuous consumption of a sequence of events — ticks, fills, signals — where components react as events arrive rather than polling in batches. It matches how markets behave, but requires handling ordering, duplicate delivery and backpressure to be safe.
What is at-least-once delivery?
It is a guarantee that every event is delivered but some may be delivered more than once, typically when an acknowledgement is lost and the producer re-sends. It is the practical default for most streams, so consumers must tolerate duplicates rather than assume uniqueness.
How do I handle duplicate events?
Consume idempotently: deduplicate by a unique event id kept within a bounded window, or make handlers naturally idempotent by setting absolute state from a snapshot rather than incrementing. For fills, dedup by fill id so a replayed fill does not inflate your position.
Why does event ordering matter?
Because out-of-order processing corrupts state — a cancel applied before its placement, or an old tick overwriting a newer one, gives a wrong view. Sequence numbers let the consumer keep order, detect gaps and reject stale updates.
What is backpressure?
It is the condition where events arrive faster than you can process them. Handled well, you bound the queue and conflate or shed load; handled badly, buffers grow, latency climbs and the system acts on stale prices or gets disconnected.
What is exactly-once delivery?
It is the ideal where each event is processed once with no loss and no duplication. It is expensive and usually only approximated in practice, so trading systems generally rely on at-least-once delivery plus deduplication instead.
How is event streaming different from a WebSocket?
A WebSocket is a transport that can carry a stream; event streaming is the broader model and discipline of consuming events reliably. A broker WebSocket after reconnect behaves like an at-least-once stream, so the same ordering, dedup and backpressure concerns apply.
What is idempotent consumption?
It means processing the same event twice has the same effect as processing it once. Achieved via deduplication by id or by using absolute-state operations, it is the essential defence against the duplicates that at-least-once delivery produces.
How do sequence numbers help?
A monotonic sequence number per partition lets the consumer detect a missing event (a gap prompting reconciliation) and ignore a stale one (older than the last applied). They are how ordering and completeness are enforced on a stream.
What should I do when the queue fills under load?
Apply a backpressure policy: conflate stale data (for pure ticks, keep only the latest), shed lower-priority load, or slow the producer if the protocol allows. Never let an unbounded queue grow until the system acts on minutes-old data.
What is a consumer offset?
In durable event systems it is the consumer's tracked position in the stream, letting it resume after a restart. Committing the offset after processing gives at-least-once (safe against loss, needs dedup); committing before risks losing events.
Why not just use exactly-once and forget deduplication?
Because true exactly-once is costly and often unavailable — broker feeds after a reconnect give at-least-once. Relying on an assumption of uniqueness you do not actually have leads to double-counted fills, so deduplication is the pragmatic safeguard.

Voice search & related questions

Natural-language questions people ask about Event Streaming.

What is event streaming in trading?
It is your system reacting to a constant flow of little messages — a price ticked, an order filled — instead of asking for data in batches. It is how markets naturally work.
What does at-least-once delivery mean?
It means no event is ever lost, but some may arrive twice. So your code has to expect duplicates and handle them, rather than trusting every message is unique.
How do I stop counting a fill twice?
Deduplicate by the fill's unique id — remember which ids you have seen and ignore repeats. Or set your position from a snapshot instead of adding on each fill.
What is backpressure?
It is when events come in faster than you can handle them. If you ignore it, your queue and lag balloon until you are trading on old prices. So you cap the queue and drop stale ticks.
Why does the order of events matter?
Because if a cancel is handled before the order it cancels, or an old price overwrites a new one, your view goes wrong. Sequence numbers keep events in the right order.
Is exactly-once delivery realistic?
Usually not — it is costly and often only approximated. Most real streams give at-least-once, so the safe approach is to expect duplicates and deduplicate them.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Retry Logic
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.