Data Validation

Quick Answer

Data validation is the automated gate that proves market data is sound before a backtest or live system may use it. It runs four layers — schema and types, range and sanity, continuity, and cross-source reconciliation — and blocks or flags anything that fails. NSE circuit bands give a free range check, and disagreeing feeds catch subtle corruption.

Definition: Data Validation

Data Validation is the automated gatekeeping layer that checks market data against schema, range, continuity and cross-source rules before a backtest or live system may use it.

Key takeaways: Data Validation

• Validation is the automated gate that proves data is sound before a backtest or live trade uses it
• Layer the checks: schema, range, continuity and cross-source reconciliation each catch different errors
• Cross-source agreement and exchange rules (circuit bands, tick size) are the hardest-to-fool checks
• For live trading, fail safe — block or halt on failure rather than trade on data you cannot trust

Data Validation at a glance

Data Validation in simple words

Data validation is the set of automatic tests your data must pass before your strategy is allowed to touch it — the equivalent of a pre-flight checklist. It asks: is every field the right type and present, is every value in a plausible range, is the series continuous, and do independent sources agree? Anything that fails is blocked or flagged, so a bad feed stops the system instead of quietly poisoning a live trade.

What Data Validation is for

Validation exists as the last line of defence between imperfect data and real money: because data errors are silent and their downstream effects catastrophic, the only safe posture is to prove data is sound before trusting it, every time.

Data Validation — professional explanation

Schema validation: structure and types

The first layer checks that data has the shape it claims. Every expected field is present, of the correct type (a price is a number, a timestamp is a timezone-aware datetime, a symbol matches a known instrument), and non-null where required. Schema validation catches whole classes of ingestion failure — a shifted column, a missing field after a vendor API change, a string where a number should be, a null volume — before any value-level logic runs. It is cheap, deterministic and belongs at the very edge of the pipeline, because everything downstream assumes the schema holds. In production a schema violation should typically halt ingestion rather than proceed on malformed input.

Range and sanity checks

The second layer checks that values are individually plausible. Prices must be positive and within a sensible multiple of recent levels; a Nifty spot of 2,50,000 or 0 is rejected. Bar-to-bar returns must be within a bound calibrated to the instrument (a liquid index moving 30% in a minute is almost certainly an error). Volume must be non-negative; OHLC bars must satisfy high >= max(open, close) and low <= min(open, close); quotes must not be crossed; prices must respect tick size. Circuit limits give India a natural range check — NSE caps many stocks at 5%, 10% or 20% daily bands, so a move beyond the applicable band is provably impossible and flags corrupt data instantly.

Continuity and completeness checks

The third layer checks the series over time. Is the expected number of bars present for the session (375 one-minute bars on a full NSE day)? Are timestamps monotonic, unique and inside trading hours per the calendar? Are there unexpected gaps within a session for a normally liquid instrument, or duplicate bars from a feed reconnect? Continuity checks catch missing-data and duplication problems that per-value checks cannot see, because each individual value is fine — the fault is in what is absent or repeated. They require a trading calendar and an expected-liquidity model to know what 'complete' means for each instrument on each day.

Cross-source and cross-field reconciliation

The strongest validation compares independent views that should agree. Two data vendors' closes for the same symbol should match within a tolerance; a bar series rebuilt from ticks should reproduce the vendor's bars; a stock's price move should be consistent with its index's move and its sector; an adjusted series should reconcile with raw prices times the known adjustment factors; your recorded fills should match the exchange's trade file. Disagreement between sources that ought to agree is a powerful, hard-to-fool error detector, because a single feed can be confidently wrong but two independent feeds rarely share the same error. This is how subtle corruption that passes every single-source check is finally caught.

Where validation runs, and failing safe

Validation is not a one-time batch job; it runs at every boundary — on historical data before a backtest, on each live tick or bar before it reaches the strategy, and on reference data (corporate actions, calendars) before it is applied. The critical design decision is what to do on failure, and the answer for live trading is to fail safe: block the suspect data, and if the failure is severe or persistent, halt trading or trigger the kill switch rather than act on data you cannot trust. A validation layer that merely logs and proceeds is theatre. Because a corrupt feed can otherwise drive a live algorithm into disastrous orders, validation is properly understood as a risk control that sits alongside the risk engine, not as mere data hygiene.

Worked example: Data Validation

The four validation layers

Advantages of Data Validation

• Blocks corrupt data before it reaches a backtest or, critically, a live order
• Layered checks (schema, range, continuity, cross-source) catch different, complementary error types
• Cross-source reconciliation catches subtle corruption that no single-source check can see
• Exchange rules like circuit bands give exact, tuning-free validation thresholds
• Turns silent data failure into a loud, actionable stop

Limitations of Data Validation

• Thresholds too tight reject genuine extreme moves; too loose let errors through — calibration is hard
• Cross-source checks need a second independent feed, which costs money and adds complexity
• Validation itself can have bugs, giving false confidence if not tested
• Continuity checks require an accurate trading calendar and liquidity model per instrument
• No validation catches every error; some novel corruption always slips through

How professionals treat Data Validation

Mature trading operations run validation as a first-class, tested subsystem at every data boundary, with layered checks — schema, range, continuity, cross-source reconciliation — and explicit fail-safe behaviour that can block data or halt trading. They calibrate thresholds to each instrument's volatility rather than using flat constants, exploit exchange-defined limits (circuit bands, tick sizes) as exact rules, and maintain a second independent feed specifically so cross-source disagreement can surface subtle corruption. Crucially they treat the validators as production code that is itself tested, and they wire severe validation failures into the same kill-switch machinery as risk breaches. The philosophy is that unvalidated data is untrusted by default, and trust is earned per batch, per tick, every time.

Common mistakes with Data Validation

• Building a validation layer that logs failures but proceeds anyway, so it never actually protects a live trade
• Using fixed absolute thresholds that reject genuine high-volatility days or miss errors on quiet ones
• Validating historical data before a backtest but not validating the live feed to the same standard
• Relying on a single source, so a confidently wrong feed passes every check unchallenged
• Omitting continuity checks, so missing or duplicated bars pass because each individual value looks fine
• Never testing the validators themselves, so a broken check gives false assurance the data is sound

Data Validation: frequently asked questions

Voice search: how people ask about Data Validation

Natural-language questions people ask about Data Validation.

Sources & references

• Ernest P. Chan, “Quantitative Trading: How to Build Your Own Algorithmic Trading Business”, 2nd ed., Wiley, 2021
• Marcos López de Prado, “Advances in Financial Machine Learning”, Wiley, 2018