Data Validation

Quick answer: Data validation is the automated gatekeeping layer that verifies market data against explicit schema, range, continuity and cross-source checks before it is allowed into a backtest or a live system, so that corrupt data is rejected rather than silently traded on.

In simple words

Data validation is the set of automatic tests your data must pass before your strategy is allowed to touch it — the equivalent of a pre-flight checklist. It asks: is every field the right type and present, is every value in a plausible range, is the series continuous, and do independent sources agree? Anything that fails is blocked or flagged, so a bad feed stops the system instead of quietly poisoning a live trade.

Purpose

Validation exists as the last line of defence between imperfect data and real money: because data errors are silent and their downstream effects catastrophic, the only safe posture is to prove data is sound before trusting it, every time.

Professional explanation

Schema validation: structure and types

The first layer checks that data has the shape it claims. Every expected field is present, of the correct type (a price is a number, a timestamp is a timezone-aware datetime, a symbol matches a known instrument), and non-null where required. Schema validation catches whole classes of ingestion failure — a shifted column, a missing field after a vendor API change, a string where a number should be, a null volume — before any value-level logic runs. It is cheap, deterministic and belongs at the very edge of the pipeline, because everything downstream assumes the schema holds. In production a schema violation should typically halt ingestion rather than proceed on malformed input.

Range and sanity checks

The second layer checks that values are individually plausible. Prices must be positive and within a sensible multiple of recent levels; a Nifty spot of 2,50,000 or 0 is rejected. Bar-to-bar returns must be within a bound calibrated to the instrument (a liquid index moving 30% in a minute is almost certainly an error). Volume must be non-negative; OHLC bars must satisfy high >= max(open, close) and low <= min(open, close); quotes must not be crossed; prices must respect tick size. Circuit limits give India a natural range check — NSE caps many stocks at 5%, 10% or 20% daily bands, so a move beyond the applicable band is provably impossible and flags corrupt data instantly.

Continuity and completeness checks

The third layer checks the series over time. Is the expected number of bars present for the session (375 one-minute bars on a full NSE day)? Are timestamps monotonic, unique and inside trading hours per the calendar? Are there unexpected gaps within a session for a normally liquid instrument, or duplicate bars from a feed reconnect? Continuity checks catch missing-data and duplication problems that per-value checks cannot see, because each individual value is fine — the fault is in what is absent or repeated. They require a trading calendar and an expected-liquidity model to know what 'complete' means for each instrument on each day.

Cross-source and cross-field reconciliation

The strongest validation compares independent views that should agree. Two data vendors' closes for the same symbol should match within a tolerance; a bar series rebuilt from ticks should reproduce the vendor's bars; a stock's price move should be consistent with its index's move and its sector; an adjusted series should reconcile with raw prices times the known adjustment factors; your recorded fills should match the exchange's trade file. Disagreement between sources that ought to agree is a powerful, hard-to-fool error detector, because a single feed can be confidently wrong but two independent feeds rarely share the same error. This is how subtle corruption that passes every single-source check is finally caught.

Where validation runs, and failing safe

Validation is not a one-time batch job; it runs at every boundary — on historical data before a backtest, on each live tick or bar before it reaches the strategy, and on reference data (corporate actions, calendars) before it is applied. The critical design decision is what to do on failure, and the answer for live trading is to fail safe: block the suspect data, and if the failure is severe or persistent, halt trading or trigger the kill switch rather than act on data you cannot trust. A validation layer that merely logs and proceeds is theatre. Because a corrupt feed can otherwise drive a live algorithm into disastrous orders, validation is properly understood as a risk control that sits alongside the risk engine, not as mere data hygiene.

The four validation layers

Practical example

Advantages

• Blocks corrupt data before it reaches a backtest or, critically, a live order
• Layered checks (schema, range, continuity, cross-source) catch different, complementary error types
• Cross-source reconciliation catches subtle corruption that no single-source check can see
• Exchange rules like circuit bands give exact, tuning-free validation thresholds
• Turns silent data failure into a loud, actionable stop

Limitations

• Thresholds too tight reject genuine extreme moves; too loose let errors through — calibration is hard
• Cross-source checks need a second independent feed, which costs money and adds complexity
• Validation itself can have bugs, giving false confidence if not tested
• Continuity checks require an accurate trading calendar and liquidity model per instrument
• No validation catches every error; some novel corruption always slips through

Common mistakes

• Building a validation layer that logs failures but proceeds anyway, so it never actually protects a live trade
• Using fixed absolute thresholds that reject genuine high-volatility days or miss errors on quiet ones
• Validating historical data before a backtest but not validating the live feed to the same standard
• Relying on a single source, so a confidently wrong feed passes every check unchallenged
• Omitting continuity checks, so missing or duplicated bars pass because each individual value looks fine
• Never testing the validators themselves, so a broken check gives false assurance the data is sound

Professional usage

Mature trading operations run validation as a first-class, tested subsystem at every data boundary, with layered checks — schema, range, continuity, cross-source reconciliation — and explicit fail-safe behaviour that can block data or halt trading. They calibrate thresholds to each instrument's volatility rather than using flat constants, exploit exchange-defined limits (circuit bands, tick sizes) as exact rules, and maintain a second independent feed specifically so cross-source disagreement can surface subtle corruption. Crucially they treat the validators as production code that is itself tested, and they wire severe validation failures into the same kill-switch machinery as risk breaches. The philosophy is that unvalidated data is untrusted by default, and trust is earned per batch, per tick, every time.

Frequently asked questions

Voice search & related questions

Natural-language questions people ask about Data Validation.

Sources & references

Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.