Home / System Architecture / The Data Layer
ComponentIntermediate

The Data Layer

The data layer is the component that ingests, normalises, stores and serves market and reference data so every other part of the system reads one consistent, corrected view of the world.

Quick answer: The data layer is the component that ingests, normalises, stores and serves market and reference data so every other part of the system reads one consistent, corrected view of the world.

In simple words

The data layer is the plumbing that gets prices into your system and keeps them clean. It pulls quotes and trades from a feed, fixes them up into a standard shape, saves the history, and hands the latest values to the strategy. If this layer is wrong, everything above it is wrong too.

Purpose

Every signal, size, risk check and order derives from data. Centralising ingest, cleaning and storage means the whole system shares one authoritative view instead of each module fetching and interpreting raw feeds differently.

Visual explanation

The Data Layer

Raw feed to normaliser to store, with a live cache serving the strategy and a backfill path for history.

Data PipelineSource / FeedIngestNormaliseCorp-actionAdjustValidateStoreServe

Professional explanation

Single responsibility

The data layer owns one thing: producing a correct, timestamped, normalised stream of market and reference data. It should not generate signals, size positions or place orders. Keeping it isolated means you can swap a data vendor, replay history, or run a backtest through the same interface without touching strategy code. The contract it exposes to the rest of the system is a small set of read operations: give me the latest bar for this symbol, give me history between two timestamps, subscribe me to live ticks.

Inputs and outputs

Inputs are heterogeneous: a REST endpoint for historical bars, a WebSocket for live ticks and order-book updates, and reference feeds for instrument masters, lot sizes, expiry dates and corporate actions. Outputs are homogeneous by design: one canonical bar or tick object with a fixed schema (symbol, exchange, timestamp in a single timezone, OHLCV, and a source tag). The value of the layer is precisely this narrowing from many messy inputs to one predictable output that downstream code can rely on.

Normalisation is the hard part

Vendors disagree on everything: symbol naming (NIFTY vs NIFTY 50 vs the exchange token), timestamp timezone and precision, whether a bar is stamped at its open or its close, tick size, and how splits or bonuses are handled. Normalisation resolves these into house conventions: a canonical symbol map, all timestamps in IST stored as UTC epoch, bars labelled by their close time, and adjusted-vs-raw price series kept separately. Corporate actions must be applied consistently, or an indicator will jump on an ex-date for no real reason.

Storage design

Two access patterns dominate and they want different stores. Backtests need long contiguous history read sequentially, which suits columnar files (Parquet) or a time-series database partitioned by symbol and date. Live trading needs the last value and a short rolling window with microsecond reads, which suits an in-memory ring buffer or an in-process cache. A common pattern is a durable append-only historical store plus a hot in-memory cache the feed handler keeps current; the strategy reads the cache, and the same records are periodically flushed to durable storage for tomorrow's backtest.

The live feed handler and concurrency

The live path is concurrent by nature: a socket thread receives messages while the strategy thread reads the latest state. The safe pattern is a producer-consumer queue with the socket as producer and a single writer updating the shared cache, so readers never see a half-written bar. Bound the queue and decide a policy for overflow: for market data you usually drop the oldest and log a gap rather than let the queue grow unbounded and blow memory. Sequence numbers or exchange timestamps let you detect gaps, out-of-order messages and duplicates.

How it fails

Feeds disconnect silently, send stale ticks that never update, deliver a spurious zero or a fat-finger print, skip a sequence number, or double-send a message on reconnect. A frozen socket that is technically open is the most dangerous case because nothing errors. Defences: a heartbeat or expected-tick-rate watchdog that flags staleness, a reconnect-and-backfill routine that fills the gap from REST after a drop, sanity filters (reject prices outside a band of the last value, reject zero volume where impossible), and deduplication keyed on sequence or exchange timestamp.

Observability

Instrument the layer so you can see its health without guessing. Emit metrics for messages per second per symbol, time since last tick, queue depth, dropped-message count, reconnect count, and the age of the newest bar. Log every gap detected and every value rejected by a sanity filter, with the raw payload. A dashboard that shows the last-tick age per subscribed symbol turns an invisible frozen feed into an obvious red tile, and a data-staleness alert should feed the same monitoring and kill-switch path the rest of the system uses.

Historical store vs live cache

AspectHistorical storeLive cache
Primary useBacktesting, researchLive signal generation
Access patternLong sequential readsLatest value + short window
DurabilityPersistent (disk/DB)In-memory, rebuilt on start
Latency needSeconds is fineMicroseconds to milliseconds
Typical techParquet / time-series DBRing buffer / in-process cache

Practical example

Illustrative example (Indian market)

A Nifty options bot subscribes to a broker WebSocket for the Nifty spot and the near-expiry option chain. The feed handler receives ticks on a socket thread and pushes them to a bounded queue; a single writer updates an in-memory dict keyed by instrument token, stamping each with the exchange timestamp converted to a UTC epoch. A watchdog checks that the Nifty spot has updated within the last two seconds during market hours; if not, it flags a stale feed. On a mid-session disconnect the handler reconnects, then backfills the missing one-minute bars via the REST history endpoint before resuming, so the strategy never sees a hole. Every raw tick is also appended to a Parquet file partitioned by date, giving tomorrow a clean historical set to backtest on.

NSE distributes lot sizes, expiry calendars and instrument tokens in a daily instruments dump; the data layer must reload this each morning because tokens and the tradable option strikes change with expiry, and a stale instrument master will subscribe you to the wrong or an expired contract.

Advantages

  • One authoritative, corrected view of data shared by every module
  • Vendor and backtest-vs-live swappable behind one interface
  • Corporate actions and timezone handling done once, correctly
  • Failures (gaps, staleness, bad prints) caught centrally, not per-strategy

Limitations

  • Normalisation is vendor-specific and brittle; a feed format change can break ingest silently
  • An in-memory cache is lost on restart and must be rebuilt or backfilled before trading
  • Historical and live data can diverge (vendor revisions, adjusted vs raw), producing backtest-vs-live mismatches
  • Tick storage grows fast; a full Nifty option chain generates large volumes needing retention policy and compaction

Why it matters in practice

  • Data quality caps the achievable quality of every downstream decision
  • A silent stale feed can make the system trade on a price that no longer exists

Common mistakes

  • Treating an open socket as proof of a live feed, so a frozen connection goes undetected until orders fire on stale prices
  • Mixing timezones (IST vs UTC, exchange vs local) so bars misalign and indicators compute on the wrong window
  • Storing only adjusted prices, then being unable to reconcile fills and P&L against the raw traded price
  • Letting the ingest queue grow unbounded under load until the process runs out of memory instead of dropping and logging
  • Backtesting on a different, cleaner dataset than the live feed, hiding the noise and gaps the strategy will actually face
  • Not reloading the daily instrument master, so the bot subscribes to an expired or wrong token after expiry

Professional usage

Professional desks treat the market-data platform as a first-class system with its own team, SLAs and monitoring, not a helper script. They separate a golden historical store (survivorship-free, corporate-action-adjusted, versioned) from a low-latency live bus, capture raw feeds verbatim so they can re-normalise if a bug is found, and reconcile vendor feeds against each other to catch bad prints. The discipline is that data is the product the rest of the system consumes, so it gets the same rigour as execution.

Key takeaways

  • The data layer narrows many messy feeds into one canonical, timestamped, corrected stream
  • Separate a durable historical store from a hot live cache; they have opposite access patterns
  • The dangerous failure is a silently stale feed, so watchdog staleness explicitly
  • Normalise timezones, symbols and corporate actions once, at the boundary, not per strategy

Frequently asked questions

What is the data layer in a trading system?
It is the component responsible for ingesting market and reference data, normalising it into a single consistent schema, storing history, and serving the latest values to the rest of the system. It is the sole source of truth for prices, so every signal and risk check reads from it.
What is the difference between market data and reference data?
Market data is the fast-changing stream of prices, trades and quotes. Reference data is the slower-changing context: instrument masters, symbol mappings, lot sizes, tick sizes, expiry dates and corporate actions. Both are needed to interpret a tick correctly.
Why does market data need normalisation?
Vendors differ in symbol naming, timezones, timestamp conventions, tick sizes and corporate-action handling. Normalisation converts all of these into one house standard so downstream code never has to special-case a source.
How should I store historical tick data?
Use a columnar or time-series format partitioned by symbol and date, such as Parquet or a purpose-built time-series database. These support the long sequential reads a backtest needs and compress well, which matters because tick data volumes are large.
How do I detect a stale or frozen feed?
Track the time since the last update per symbol and compare it to an expected tick rate during market hours. If no tick arrives within the expected window, flag staleness and trigger a reconnect or a halt; an open socket alone does not prove the feed is live.
What happens to my data cache when the system restarts?
An in-memory cache is lost and must be rebuilt. The usual approach is to backfill recent history from a REST endpoint or the durable store on startup, then resume live subscription, so the strategy has its rolling window before it trades.
How do I handle a mid-session feed disconnect?
Reconnect automatically, detect the gap using sequence numbers or timestamps, backfill the missing bars from a REST history endpoint, and only then resume live processing. Log the gap so you know the period was reconstructed rather than streamed.
Should backtests and live trading use the same data layer?
Ideally yes, behind one interface, so the strategy code is identical in both modes. A replay source feeds historical bars through the same path a live feed would, which reduces the risk that live behaviour differs from the backtest for data-handling reasons.
What are corporate actions and why do they matter to the data layer?
Corporate actions are splits, bonuses, dividends and similar events that change a stock's raw price for mechanical reasons. The data layer must adjust historical prices consistently, or an indicator will register a false jump on the ex-date.
How much history should the live path keep in memory?
Only what the strategy needs: usually a rolling window sized to the longest lookback of any indicator, plus a small margin. Keeping the hot cache small keeps reads fast; everything else lives in durable storage.
Why keep both raw and adjusted price series?
Adjusted prices give continuous returns for signals and indicators, but fills, margins and P&L happen at the raw traded price. Keeping both lets you compute signals correctly and still reconcile against what the broker actually charged.
What is a producer-consumer queue in a feed handler?
It is a bounded buffer where the socket thread (producer) drops incoming messages and a single writer thread (consumer) applies them to shared state. It decouples receiving from processing and prevents readers from seeing a half-updated record.
How do I catch bad prints in the feed?
Apply sanity filters at ingest: reject prices outside a plausible band around the last value, reject zero or negative prices where impossible, and reject volumes that violate known limits. Log every rejection with the raw payload so you can review whether the filter is too tight.
Is the data layer part of an event-driven architecture?
Yes. In an event-driven system the data layer emits market-data events (new tick, new bar) that the strategy and other components react to, which keeps the same code path usable for both backtesting and live trading.

Voice search & related questions

Natural-language questions people ask about The Data Layer.

What does the data layer do in an algo trading system?
It takes messy price feeds, cleans and standardises them, stores the history, and hands the latest prices to your strategy so everything reads the same numbers.
Why is clean data so important for a trading bot?
Because every signal, size and order is calculated from the data. If a price is wrong or stale, the bot can trade on something that is not real.
How do I know if my price feed has frozen?
Watch the time since the last update. If no new tick has arrived in the last couple of seconds during market hours, treat the feed as stale even if the connection still looks open.
Where should I keep my historical price data?
In a columnar file format like Parquet or a time-series database, split by symbol and date, because that is fast to read for backtests and compresses the large volumes well.
What happens to my data if the bot crashes?
The in-memory data is gone, so on restart the bot should backfill recent bars from history before it starts trading again.
Do I need tick data or are candles enough?
Candles are enough for most bar-based strategies. Tick data is only needed if you model fills precisely or trade on microstructure, and it is much larger to store.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Error Handling
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.