Home / Market Data / Missing Data
Data qualityIntermediate

Missing Data

Missing data is the absence of expected observations in a time series — gaps from holidays, trading halts, illiquidity or feed failures — and how you fill, drop or interpolate those gaps changes indicators and backtest results, often introducing subtle bias.

Quick answer: Missing data is the absence of expected observations in a time series — gaps from holidays, trading halts, illiquidity or feed failures — and how you fill, drop or interpolate those gaps changes indicators and backtest results, often introducing subtle bias.

In simple words

A market data series is supposed to have a value for every expected period, but reality leaves holes: markets close for holidays, trading halts on circuit breakers, thin instruments simply do not trade some minutes, and feeds occasionally fail. Missing data is these holes, and the tricky part is not spotting them but deciding what to do with them — because every choice quietly changes your numbers.

Purpose

Handling missing data matters because time-series indicators and portfolio logic assume regularly spaced observations, and filling the gaps carelessly can inject look-ahead bias, fake continuity or distorted returns without any visible sign.

Professional explanation

Distinguishing the kinds of gaps

Not all missing data is the same, and treating them alike is a mistake. An expected calendar gap — weekends, exchange holidays, the daily overnight break — is not missing data at all; it is simply a period with no session, and your calendar should encode it so you never try to fill it. A within-session gap on a liquid instrument (a minute with no bar when there should be one) usually indicates a feed problem. A within-session gap on a thin instrument (an illiquid option strike that did not trade) is genuine absence of trading. A halt — a circuit breaker or a stock suspension — is a real period during which no price existed. Each demands different handling, and the first step is always classification against a trading calendar.

Forward-fill and its hidden dangers

Forward-filling carries the last known value across a gap. It is convenient and often necessary to align instruments, but it is dangerous in specific ways. It creates artificial flat periods that suppress measured volatility, so an ATR or standard-deviation estimate over a forward-filled series understates risk. More seriously, forward-filling can smuggle in look-ahead bias if done carelessly across the boundary of when data actually arrived, or can make an illiquid instrument look continuously tradeable when in reality you could not have transacted at that carried price. A backtest that assumes fills at forward-filled prices for an instrument that was not trading is executing at prices that never existed.

Dropping rows and the survivorship trap

Dropping missing observations seems clean but has its own biases. Dropping bars where an instrument did not trade quietly removes exactly the illiquid, hard-to-trade periods — which are often the riskiest — flattering the strategy. In a cross-sectional or portfolio context, dropping any row where any instrument is missing (listwise deletion) can shrink and skew the sample toward the most liquid names and continuous periods, a cousin of survivorship bias. Dropping also breaks the regular time grid that many indicators assume, so a 20-period moving average computed after dropping rows spans a different, uneven amount of real time than intended.

Interpolation: inventing data between the gaps

Linear or spline interpolation fills a gap with a smooth path between the known endpoints. Its cardinal sin in trading is look-ahead bias: interpolating between a point before the gap and a point after it uses future information (the gap's end) to fill the present, which is impossible in live trading and inflates backtest performance. Interpolated prices are also unnaturally smooth, again understating volatility and, worse, presenting a tradeable-looking price at times when no market existed. Interpolation may be acceptable for some non-price reference series, but for prices used in a causal backtest it is almost always wrong.

How missing-data handling silently biases results

The through-line is that every gap-handling choice is an assumption that the backtest cannot see through. Forward-filling suppresses volatility and can fake tradeability; dropping removes the hardest periods and skews the sample; interpolation injects look-ahead. None of these throws an error — they produce a complete, tidy series that quietly misrepresents what a trader could actually have done. The discipline is to classify each gap by cause, choose a handling rule justified by that cause, apply the same rule in backtest and live, and never let a data library's silent default (many forward-fill by default) make the decision for you.

Gap-handling methods and their bias

MethodWhat it doesMain risk
Forward-fillCarry last valueFakes tradeability, hides volatility
DropRemove the gapRemoves hard periods, breaks time grid
InterpolateSmooth between endsLook-ahead bias, false smoothness
Calendar-awareMark as no-sessionNone if calendar is correct

Practical example

Illustrative example (Indian market)

Suppose you backtest a pairs strategy on two NSE stocks, one liquid and one thin. On several minutes the thin stock does not trade. If your data library forward-fills its price by default, the spread between the pair looks smooth and you generate mean-reversion signals — but at those minutes you could not actually have traded the thin leg at the carried price, so the backtest books fills that were impossible. If instead you drop those minutes, you remove exactly the illiquid moments when the spread was widest and the 'opportunity' looked best, flattering the strategy differently. The honest approach classifies the gaps as genuine illiquidity, excludes those minutes from tradeable signals (not just from the maths), and applies the identical rule live — at which point much of the apparent edge, which depended on untradeable prices, disappears.

NSE observes numerous trading holidays and can halt a stock on circuit limits or a scrip-specific suspension. A daily backtest must use the exchange holiday calendar so that a Diwali-week gap is treated as no-session, not as missing data to be filled — filling across a holiday would fabricate returns for days the market never traded.

Advantages

  • Correctly classifying gaps prevents fabricated returns and impossible fills
  • A proper trading calendar makes holiday and session gaps a non-issue rather than an error
  • Explicit, cause-based handling keeps volatility and risk estimates honest
  • Consistent live and backtest handling removes a whole class of train/serve mismatch

Limitations

  • Every fill method biases the data; there is no neutral choice, only a justified one
  • Distinguishing a feed failure from genuine illiquidity can be ambiguous
  • Forward-fill and interpolation can make untradeable prices look tradeable
  • Dropping rows breaks the regular time grid indicators assume and can skew the sample
  • Data libraries often forward-fill by default, making a silent decision on your behalf

Common mistakes

  • Interpolating price gaps in a backtest, which uses the gap's end (future data) to fill the present — look-ahead bias
  • Forward-filling an illiquid instrument and then booking fills at prices that never traded
  • Filling across exchange holidays instead of encoding them as no-session in a calendar
  • Dropping untraded minutes, silently removing the riskiest, least liquid periods and flattering results
  • Letting a data library's default fill behaviour decide, rather than choosing a rule per gap cause
  • Handling gaps one way in research and another live, creating a hidden train/serve mismatch

Professional usage

Robust pipelines carry an authoritative trading calendar (holidays, half-days, session times) so calendar gaps are never mistaken for missing data, and classify every remaining gap by cause before choosing a rule. Prices are generally not interpolated for causal backtests; instead, periods without a genuine tradeable price are marked untradeable so the strategy cannot transact in them, and the same masking applies live. Forward-fill, where used for alignment, is bounded and flagged so it cannot silently fake long stretches of tradeability. The guiding rule is that a gap's handling must reflect why the data is missing, and must be identical in simulation and production.

Key takeaways

  • Classify gaps by cause — calendar, feed failure, illiquidity, halt — before deciding how to handle them
  • Forward-fill hides volatility and can fake tradeability; interpolation injects look-ahead bias
  • Dropping untraded periods quietly removes the hardest, riskiest moments and flatters the strategy
  • Encode exchange holidays in a calendar and apply identical gap rules in backtest and live

Frequently asked questions

What is missing data in a market time series?
It is the absence of an expected observation — a gap from a holiday, a trading halt, an illiquid instrument not trading, or a feed failure. The challenge is less in spotting gaps than in deciding how to handle them without biasing results.
Are weekends and holidays missing data?
No. They are expected periods with no trading session and should be encoded in a trading calendar, not treated as gaps to fill. Filling across a holiday fabricates returns for days the market never traded.
What is forward-filling and when is it dangerous?
Forward-filling carries the last known price across a gap. It is dangerous because it creates artificial flat periods that understate volatility, and it can make an illiquid instrument look continuously tradeable at a carried price you could not actually have transacted at.
Why is interpolating price gaps a problem?
Because interpolation uses the value after the gap to fill the present — future information you would not have in live trading. This is look-ahead bias, and it inflates backtest performance while producing unnaturally smooth, untradeable prices.
What is wrong with just dropping missing rows?
Dropping removes exactly the illiquid, often riskiest periods, flattering the strategy, and in a portfolio context it skews the sample toward liquid names — a cousin of survivorship bias. It also breaks the regular time grid many indicators assume.
How do I tell a feed failure from genuine illiquidity?
Context and cross-checks. A missing minute on a normally liquid instrument, while other instruments have data, suggests a feed problem; a missing minute on a thin option that rarely trades is likely genuine absence. Classification against expected liquidity guides the handling rule.
What is the safest way to handle price gaps in a backtest?
Classify each gap by cause and, for genuine no-trade periods, mark them untradeable so the strategy cannot transact there, rather than inventing a price. Avoid interpolation for causal price series, and apply the identical rule in live trading.
Do trading halts count as missing data?
A halt is a real period during which no tradeable price existed, so it should be treated as untradeable rather than filled. Fabricating prices across a circuit-breaker halt would let a backtest 'trade' when the market was frozen.
Why do data libraries forward-fill by default?
Because forward-filling produces a tidy, gap-free series that is convenient for computation. The danger is that this silent default makes a consequential modelling decision for you, so you should set the behaviour explicitly rather than accept it.
How does missing-data handling create train/serve mismatch?
If you fill gaps one way in research and another way (or not at all) in live trading, the strategy sees systematically different inputs in the two environments. Results diverge, and the backtest no longer predicts live behaviour.
Can forward-fill ever be acceptable?
Yes, for bounded alignment — for example, carrying a slow-moving reference value briefly to line up instruments — provided it is limited in length, flagged, and does not imply tradeability at the carried price. The key is that it is a deliberate, bounded choice.
How do gaps affect indicators like moving averages?
If you drop rows, a fixed-period moving average spans an uneven amount of real time; if you forward-fill, the flat values pull the average toward the last price and suppress its responsiveness. Either way the indicator no longer means what its period suggests.
Should missing data be handled before or after other cleaning?
Usually after error detection but as its own explicit step, because dropping a bad bar during cleaning itself creates a gap that the missing-data logic must then handle. The two steps interact and should be ordered deliberately.
Why is a trading calendar essential for handling gaps?
Because it tells you which periods are supposed to have data. Without it you cannot distinguish an expected no-session (holiday, overnight) from a genuine gap, and you risk fabricating returns for periods the market never traded.

Voice search & related questions

Natural-language questions people ask about Missing Data.

What is missing data in trading?
It is a hole in your price data where a value should be — from a holiday, a halt, an illiquid stock not trading, or a feed glitch. The hard part is deciding what to do with the hole.
Is a market holiday missing data?
No. A holiday is just a day with no trading, so you mark it in a calendar rather than filling it. Filling across a holiday would invent returns for a day nothing traded.
Why is forward-filling risky?
Because it carries the last price forward, which hides real volatility and can make an illiquid stock look tradeable at a price you could never actually have got.
Why can't I just interpolate the gaps?
Because interpolation uses the price after the gap to fill the present, which is future information you wouldn't have live. That's look-ahead bias and it makes backtests look better than reality.
Is it okay to just delete the missing rows?
Often no. Deleting untraded periods removes the hardest, riskiest moments, which makes your strategy look safer and better than it really is.
How should I handle gaps then?
Figure out why the data is missing, mark genuine no-trade periods as untradeable instead of inventing prices, and handle it the same way in testing and in live trading.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Data Cleaning
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.