Home / Market Data / Data Cleaning
Data qualityIntermediate

Data Cleaning

Data cleaning is the systematic detection and correction of errors in market data — bad prints, outliers, duplicates and spurious spikes — so that a strategy is built and traded on the true price history rather than on artefacts of the feed.

Quick answer: Data cleaning is the systematic detection and correction of errors in market data — bad prints, outliers, duplicates and spurious spikes — so that a strategy is built and traded on the true price history rather than on artefacts of the feed.

In simple words

Raw market data is never perfectly clean: feeds glitch, prices get mis-typed, records get duplicated, and occasional impossible values slip through. Data cleaning is the housekeeping that finds and fixes these before they reach your strategy. It is unglamorous and easy to skip, but a single uncaught bad price can quietly make an entire backtest lie.

Purpose

Data cleaning exists because every downstream number — every indicator, signal and backtest metric — inherits the errors in its input, and those errors, being silent, are far more dangerous than a crash.

Professional explanation

The taxonomy of data errors

Errors fall into recognisable categories, and naming them helps you hunt them. Bad prints are single erroneous trades — a fat-finger, a feed glitch, a decimal misplacement — that show an impossible price. Outliers are values far outside the plausible range that may be real (a genuine gap) or errors, and telling them apart is the hard part. Duplicates are the same record ingested twice, common when reconnecting a feed. Spikes are transient jumps that revert within one or two ticks, often a stale or crossed quote. Gaps (missing data) are a related but separate problem. Each type needs a different detector; a single generic rule will miss most of them.

Robust statistical detection

Naive outlier detection using the mean and standard deviation is itself fragile, because a single huge bad print inflates the standard deviation and hides other errors — the very outliers you seek corrupt the yardstick. Robust statistics avoid this: the median and the median absolute deviation (MAD) are barely affected by extreme values, so a rule like 'flag any price more than k MADs from a rolling median' catches bad prints without being fooled by them. Similarly, checking bar-to-bar log returns against a robust volatility estimate flags spikes: a return of, say, more than 8 robust standard deviations in one minute on a liquid instrument is almost certainly an error, not a move.

Structural and cross-field checks

Beyond statistics, market data has internal logic that must hold. Within an OHLC bar, the high must be at least the open, close and low, and the low at most all three; volume must be non-negative; a bid must not exceed the ask (no crossed quote). Timestamps must be monotonically non-decreasing and fall inside a valid session. Prices must respect the instrument's tick size (a Nifty option quoted at 100.037 is impossible if the tick is 0.05). These structural checks are cheap, deterministic and catch a large class of errors that statistical methods would miss because the value is individually plausible but internally inconsistent.

Correct, drop or flag — the remediation decision

Detecting an error is only half the job; deciding what to do with it is a judgement with consequences. Options are to correct it (interpolate, or replace with the previous valid value), drop the record, or flag and quarantine it for review. Each choice biases the data in some way: interpolating a spike removes the error but invents a value; dropping a bad bar creates a gap that a later step must handle; silently correcting hides the frequency of feed problems you might need to know. The safest default in research is to flag and log every correction so its effect is auditable, and never to clean live and historical data by different rules — that mismatch reintroduces bias.

How uncleaned data silently corrupts strategies

The reason cleaning matters is that its failures are invisible. A single bad high of 2,50,000 on a Nifty bar will not crash your backtest; it inflates that day's ATR, widens Bollinger Bands for the next twenty bars, and may trigger a breakout entry or a volatility filter — all producing plausible trades on a fiction. Duplicated bars double-count volume and distort VWAP. A stale crossed quote invents an arbitrage that never existed. Because these outputs look reasonable, the strategy can be optimised, deployed and lose real money before anyone traces the fault back to one uncleaned row. Cleaning is therefore a risk control, not mere tidiness.

Practical example

Illustrative example (Indian market)

Suppose your one-minute Nifty feed contains a single bad print: one bar shows a high of 26,850 while every neighbouring bar sits near 25,000. A mean/standard-deviation filter over the day might not flag it strongly because that one value has inflated the standard deviation. A robust filter does: the rolling median is ~25,000 and the MAD is small, so 26,850 sits perhaps 40 MADs away and is flagged instantly. You then check the structural rule (is high >= open, close, low? yes, so it is not a bar-internal error) and the return rule (a +7% one-minute jump reverting immediately is implausible). You quarantine the bar, replace its high with the max of its open and close, log the correction, and re-run — the phantom ATR spike and the false breakout it would have caused both disappear.

NSE instruments have defined tick sizes (for example 0.05 for many option premia). A quoted price that is not a multiple of the tick size is provably an error, giving a cheap, exact structural check that catches decimal-shift and feed-corruption bugs no statistical method would flag.

Advantages

  • Prevents single bad values from silently distorting indicators and triggering phantom trades
  • Robust methods (median, MAD) detect errors without being fooled by them
  • Structural and tick-size checks are cheap, deterministic and catch internally inconsistent values
  • Logged corrections make data quality auditable and reproducible

Limitations

  • Genuine large moves and errors can look identical, risking the deletion of real data (over-cleaning)
  • Every remediation choice (interpolate, drop, flag) biases the data in some way
  • Cleaning live and historical data by different rules reintroduces the bias you tried to remove
  • Over-aggressive filters can smooth away real volatility, flattering a strategy's risk profile
  • No cleaning process is perfect; some errors always survive, so downstream validation is still needed

Common mistakes

  • Using mean and standard deviation for outlier detection, so one big bad print inflates the yardstick and hides others
  • Interpolating or deleting genuine large moves as if they were errors (over-cleaning), removing real risk
  • Cleaning historical data but not applying the same rules to the live feed, creating a train/live mismatch
  • Not de-duplicating after a feed reconnect, so bars are double-counted in volume and VWAP
  • Skipping cheap structural checks (high >= low, tick-size multiples, monotonic timestamps) that catch obvious errors
  • Silently overwriting bad values with no log, so the true frequency of feed problems is hidden

Professional usage

Professional data pipelines treat cleaning as a versioned, logged transformation with a battery of detectors — robust statistical outlier tests, structural OHLC and quote consistency checks, tick-size and session validation, and duplicate detection — run in a fixed order, with every correction recorded to an audit trail. The same cleaning logic runs on historical and live data so there is no train/serve skew. Quants deliberately keep the raw data immutable and derive cleaned versions, so any over-cleaning can be reverted and re-examined. The philosophy is to flag generously, correct conservatively, and make every decision reproducible rather than to silently sanitise.

Key takeaways

  • Clean data is a precondition for trustworthy indicators, signals and backtests, not optional tidiness
  • Use robust statistics (median, MAD) so the errors you seek do not corrupt the detector
  • Combine statistical detection with cheap structural checks like OHLC bounds and tick-size validity
  • Log every correction and clean live and historical data identically, or you reintroduce bias

Frequently asked questions

What is data cleaning in algorithmic trading?
It is the systematic process of finding and fixing errors in market data — bad prints, outliers, duplicates and spurious spikes — before it feeds a strategy. The goal is to trade and backtest on the true price history rather than on feed artefacts.
Why is data cleaning so important?
Because every downstream number inherits its input's errors, and those errors are silent. A single uncaught bad price does not crash anything; it distorts indicators, triggers phantom signals and produces a plausible but wrong backtest, which makes cleaning a risk control.
What is a bad print?
A bad print is a single erroneous trade or quote — from a fat-finger, a decimal shift or a feed glitch — showing an impossible price. It is one of the most common data errors and can severely distort range-based indicators if not caught.
Why shouldn't I use standard deviation to find outliers?
Because a single large bad value inflates the standard deviation itself, hiding other outliers behind a corrupted yardstick. Robust measures like the median and median absolute deviation are barely affected by extremes and detect errors far more reliably.
What is the median absolute deviation and why use it?
The MAD is the median of the absolute differences from the median — a robust measure of spread. Because it ignores extreme values, a rule like 'flag prices more than k MADs from a rolling median' catches bad prints without being distorted by them.
What structural checks should I run on OHLC data?
Confirm the high is at least the open, close and low; the low is at most all three; volume is non-negative; timestamps are monotonic and inside a session; and prices are multiples of the instrument's tick size. These are cheap and catch internally inconsistent values.
Should I correct, drop or flag bad data?
It depends, and each biases the data differently — interpolating invents a value, dropping creates a gap, silently correcting hides the error rate. The safest research default is to flag and log every correction so its effect is auditable.
What is over-cleaning?
Over-cleaning is removing or smoothing genuine large moves as if they were errors. Since a real gap and a bad print can look identical, an aggressive filter can delete true volatility, flattering the strategy's risk profile and hiding real tail risk.
How do duplicates get into market data?
Most often when a feed reconnects and replays recent messages, or when two ingestion processes overlap. Duplicated bars double-count volume and distort VWAP and any volume-based signal, so de-duplication on a unique key is essential.
Why must live and historical cleaning use the same rules?
Because if you clean historical data one way and the live feed another, the strategy sees systematically different inputs in backtest and production — a train/serve skew that reintroduces the very bias cleaning was meant to remove.
How does a tick-size check catch errors?
Every instrument trades in fixed price increments (a tick size). A quoted price that is not a whole multiple of the tick size is provably impossible, so a simple modulo check catches decimal-shift and corruption errors that are individually plausible in magnitude.
Can data cleaning ever remove real information?
Yes — that is its central risk. Because errors and genuine extreme moves can be indistinguishable, any cleaning process may delete real data. This is why corrections are logged and the raw data kept immutable, so over-cleaning can be reviewed and reversed.
What is a spike versus an outlier?
A spike is a transient jump that reverts within a tick or two, often from a stale or crossed quote; an outlier is any value far from the plausible range, which may or may not revert. Spikes are usually errors; standalone outliers may be genuine moves.
Is cleaned data guaranteed correct?
No process catches every error, so some always survive. That is why cleaning is followed by separate data validation before trading, and why keeping raw data lets you re-clean when a new error type is discovered.

Voice search & related questions

Natural-language questions people ask about Data Cleaning.

What does cleaning market data mean?
It means finding and fixing mistakes in your price data — wrong prices, duplicates and glitches — before your strategy uses them, so you aren't building on bad numbers.
Why is one bad price such a big deal?
Because it won't crash anything, it just quietly distorts your indicators and can trigger fake trades, so your backtest looks fine but is actually wrong.
What's the best way to spot bad prices?
Use robust measures like the median and median absolute deviation, because a single huge error won't fool them the way an average would.
Can cleaning data cause problems too?
Yes. If you clean too aggressively you can delete real big moves, which hides genuine risk and makes your strategy look safer than it is.
Should I clean live data the same way as history?
Absolutely. If your live and historical cleaning differ, your strategy sees different inputs in testing and in real trading, which quietly ruins your results.
How do duplicate records get in?
Usually when a data feed drops and reconnects and replays recent messages. The duplicates double-count volume, so you need to remove them using a unique key.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Corporate Actions
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.