Home / System Architecture / Failover & Redundancy
ComponentAdvanced

Failover & Redundancy

Failover and redundancy are the design properties that let a trading system survive the failure of a component, machine or connection, by having backups, recoverable state, and safe restart, without losing track of positions or duplicating orders.

Quick answer: Failover and redundancy are the design properties that let a trading system survive the failure of a component, machine or connection, by having backups, recoverable state, and safe restart, without losing track of positions or duplicating orders.

In simple words

Things break, machines crash, networks drop, processes die. Failover is the plan for when they do: a backup ready to take over, a way to recover exactly where you were, and a restart that does not double your orders or lose your positions. The goal is that a single failure is an inconvenience, not a disaster.

Purpose

A trading system runs in a hostile, unreliable environment with real money at stake, so failure is a certainty, not a possibility. Redundancy and recoverable state turn an inevitable failure into a survivable one rather than a catastrophic one.

Professional explanation

Single points of failure

A single point of failure (SPOF) is any component whose failure takes down the whole system: one server, one network link, one process, one data feed, one broker connection. Identifying SPOFs is the first step in failover design, you map the system and ask, for each part, what happens if this dies right now. Some SPOFs can be made redundant (two feeds, two network paths); some are irreducible (there is one broker, one exchange) and must instead be handled by failing safe when they go down. The discipline is deliberate: every SPOF is either given a backup or given a defined safe response, never left as an unexamined assumption that it will not fail.

Redundancy patterns

Redundancy means having more than one of something so a failure has a fallback. Active-passive (hot standby) keeps a backup instance ready to take over when the primary fails; it is simpler and avoids both instances acting at once, which for trading is important because two active instances could both send orders and double positions. Active-active is more complex and rarely appropriate for the order-sending path precisely because of that duplication risk. Redundancy applies at many layers: redundant data feeds (a backup vendor), redundant connectivity (a second internet path), redundant compute (a standby host), and geographic redundancy for disaster scenarios. For most retail systems, a hot standby plus a backup data feed is the pragmatic level.

State recovery

Failover is only safe if the backup, or the restarted process, can recover the exact state: what positions are held, what orders are in flight, what the P&L and limit counters are. This is why the OMS and portfolio engine persist state durably and reconcile against the broker, on takeover or restart, the system rebuilds its picture from persisted state plus the broker's authoritative record, rather than starting blind. State recovery is the hard part of failover: a backup that takes over without knowing the current positions and open orders is more dangerous than no backup, because it may re-trade or contradict what is already in the market. Reconciliation-on-startup is the linchpin that makes recovery trustworthy.

Restart safety and idempotency

A system must be safe to restart at any moment, because crashes happen at the worst times. Restart safety rests on the same properties that make the OMS reliable: durable state so nothing in flight is forgotten, idempotent order submission so a restart mid-send does not duplicate an order, dedupe so replayed events do not double-count, and mandatory reconciliation before resuming trading. The test of restart safety is brutal and simple: kill the process at a random instant and confirm it comes back to a correct, reconciled state with no duplicated or lost orders. If that is not true, the system is not safe to run unattended, because an unplanned restart is a matter of when, not if.

Failover mechanics and split-brain

Automatic failover needs a way to detect the primary's failure (heartbeats, health checks) and promote the standby, ideally coordinated so exactly one instance is active. The classic danger is split-brain: the primary is not actually dead (just unreachable), the standby also goes active, and now two instances trade the same account and duplicate orders, which for a trading system is a nightmare. Guards against split-brain include a single source of truth for who is active (a lease or lock the active instance must hold and renew), fencing the old primary so it cannot send orders after being superseded, and preferring to fail safe (halt) over risking two active order senders. For many retail setups, a manual, verified failover is safer than a fragile automatic one that risks split-brain.

How it fails

Failover fails by having an unnoticed SPOF that takes everything down, by a backup that was never tested and does not actually work, by state that cannot be recovered so the failover is blind, by split-brain duplicating orders, or by a restart that loses or duplicates in-flight orders. A particularly insidious failure is a backup that has silently drifted out of sync (stale config, stale credentials) and fails or misbehaves the moment it is promoted. Defences: enumerate and address every SPOF, test failover regularly by actually failing the primary, ensure recoverable durable state with startup reconciliation, and guard hard against split-brain with a single active-owner mechanism.

Observability and testing

Redundancy that is not tested is assumed broken. Regularly exercise failover, fail the primary, on purpose in a safe window and confirm the standby takes over correctly and reconciles. Monitor the health and sync status of standbys (a standby with stale state is a false safety), monitor which instance is active, and alert on any ambiguity about ownership. Log every failover event and every restart with the reconciliation result. Chaos-style drills, deliberately killing components, are how mature operations gain confidence that a real failure will be survivable. The combination of enumerated SPOFs, tested redundancy, recoverable state and split-brain guards is what lets a system claim to be resilient rather than merely hopeful.

Practical example

Illustrative example (Indian market)

A Nifty bot runs active-passive: a primary on one host and a hot standby on another, with the primary holding a short-lived lease it must renew every few seconds to remain the active order-sender. The primary's host loses network; it can no longer renew the lease, so it fences itself and stops sending orders, while the standby, seeing the lease expire, acquires it and becomes active. On takeover the standby does not trust its own memory: it reconciles against the broker, finds the account short 2 lots with one resting order, adopts that state, and resumes. Because order submission is idempotent and state was durable, no order is duplicated and no position is lost. The team validates this monthly by pulling the primary's network in a paper environment and confirming a clean, reconciled takeover, once catching a standby running stale credentials before it could matter live.

A common retail SPOF is a single home internet connection and a single broker session; even without full server redundancy, having a mobile-data failover for connectivity and a defined safe response (halt and flatten) for a broker-session drop is a meaningful improvement, because for Indian intraday F&O an unrecoverable disconnection with open positions near a volatile window is exactly the scenario that turns a small problem into a large loss.

Advantages

  • A single failure becomes an inconvenience rather than a disaster
  • Recoverable durable state means a crash resumes correctly, not blindly
  • Redundant feeds and connectivity remove common single points of failure
  • Restart safety is what makes unattended operation acceptable

Limitations

  • Redundancy adds cost and complexity, and complexity has its own failure modes
  • Automatic failover risks split-brain, which duplicates orders, a severe trading danger
  • Some SPOFs (one broker, one exchange) are irreducible and can only be failed safe
  • A backup that is never tested or has drifted is a false sense of safety

Why it matters in practice

  • The difference between a survivable outage and a runaway is often failover design
  • Restart safety and state recovery are prerequisites for trusting a system unattended

Common mistakes

  • Leaving an unexamined single point of failure that takes down the whole system
  • Never testing failover, so the untested backup fails when finally needed
  • Automatic failover without a split-brain guard, so two instances trade and duplicate orders
  • A standby with stale config or credentials that misbehaves the moment it is promoted
  • Restarting without durable state and reconciliation, losing or duplicating in-flight orders
  • Treating one broker connection as if it cannot drop, with no safe response defined

Professional usage

Resilient trading operations enumerate every single point of failure and give each a backup or a defined safe response, run order-sending components active-passive to avoid duplicate orders, and guard against split-brain with a single active-owner lease and fencing of the superseded instance. They rely on durable, reconcilable state so any restart or takeover rebuilds truth from the broker, and they test failover on a schedule by actually failing components. The philosophy is that failure is certain, so the system is engineered to survive it and is regularly proven to do so, rather than assumed to hold.

Key takeaways

  • Enumerate every single point of failure and give each a backup or a safe response
  • Prefer active-passive for order-sending and guard hard against split-brain
  • Make state durable and reconcilable so restarts and takeovers recover, not guess
  • Test failover for real on a schedule; an untested backup is assumed broken

Frequently asked questions

What is failover in a trading system?
It is the ability to survive the failure of a component, machine or connection by switching to a backup and recovering state, so that a single failure does not lose positions, duplicate orders, or take down the whole system.
What is a single point of failure?
It is any component whose failure takes down the entire system, such as one server, one network link, one data feed or one broker connection. Failover design starts by identifying these and giving each a backup or a defined safe response.
What is the difference between active-passive and active-active redundancy?
Active-passive keeps a standby ready to take over when the primary fails, so only one instance acts at a time. Active-active runs multiple instances simultaneously, which for the order path risks both sending orders and duplicating positions, so trading systems usually prefer active-passive.
What is split-brain and why is it dangerous in trading?
Split-brain is when a primary is wrongly presumed dead and a standby also goes active, so two instances trade the same account and duplicate orders. It is especially dangerous in trading because duplicated real-money orders can double exposure instantly.
How does a trading system recover state after a failure?
By persisting order and position state durably and reconciling against the broker's authoritative record on takeover or restart, so it rebuilds an accurate picture of positions, in-flight orders and P&L rather than resuming blind.
What is restart safety?
It is the property that the system can be killed at any instant and come back to a correct, reconciled state with no lost or duplicated orders. It relies on durable state, idempotent order submission, dedupe and mandatory startup reconciliation.
How do I prevent duplicate orders during failover?
By ensuring only one instance is active via a lease or lock, fencing the superseded instance so it cannot send orders, and using idempotent order submission with client order ids so any overlap does not create duplicate orders.
Should retail traders bother with redundancy?
At least at a basic level. Even without full server redundancy, a backup internet connection and a defined safe response, halt and flatten, for a broker-session drop meaningfully reduce the risk of an unrecoverable disconnection with open positions.
Why must failover be tested?
Because an untested backup is assumed broken. Backups drift out of sync, credentials expire, and code changes break the failover path silently, so the only way to trust it is to fail the primary on purpose in a safe environment and confirm a clean takeover.
Can every single point of failure be made redundant?
No. Some, like having one broker or one exchange, are irreducible. Those cannot be duplicated, so they are handled by failing safe, halting and protecting positions, when they go down, rather than by a backup.
What is a hot standby?
It is a backup instance kept running and ready to take over immediately when the primary fails, as opposed to a cold standby that must be started up first. A hot standby minimises downtime during failover.
How does failover relate to the OMS and portfolio engine?
Those components provide the durable, reconcilable state that makes failover safe. Without their persisted order and position state plus startup reconciliation, a backup or restarted process cannot recover an accurate view and failover is blind.
Is automatic failover always better than manual?
Not necessarily. Automatic failover reacts faster but risks split-brain if not carefully coordinated. For many retail systems a manual, verified failover is safer than a fragile automatic one that could put two instances live at once.
What happens to open positions if my system crashes?
They remain live at the broker, which is why restart safety and reconciliation matter: on restart the system must discover those positions from the broker and resume managing them, and until it does, a dead-man's switch may cancel resting orders as protection.

Voice search & related questions

Natural-language questions people ask about Failover & Redundancy.

What is failover in a trading system?
It is the backup plan for when something breaks, a spare ready to take over and a way to pick up exactly where you left off without losing positions or doubling orders.
What is a single point of failure?
It is any one thing that, if it dies, takes your whole system down, like a single server or a single internet connection. You either back it up or plan a safe response.
Why is split-brain dangerous for a trading bot?
Because if the main and the backup both think they are in charge, they both send orders and you end up with double the position you wanted.
What happens to my trades if my bot crashes?
The positions stay open at the broker. A safe bot, on restart, checks with the broker to find them and takes over managing them again.
Do I need a backup if I trade from home?
At least a backup internet connection and a rule to halt and close out if the broker connection drops, so a disconnection with open trades does not become a disaster.
Why test my backup if it is just sitting there?
Because backups quietly go stale or break. If you never test it, you only find out it does not work on the day you desperately need it.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Kill Switch
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.