Home / System Architecture / Monitoring
ComponentIntermediate

Monitoring

Monitoring is the cross-cutting component that continuously observes the system's health through metrics, heartbeats and health checks, raises alerts on anomalies, and includes a dead-man's switch that halts trading if the system itself stops responding.

Quick answer: Monitoring is the cross-cutting component that continuously observes the system's health through metrics, heartbeats and health checks, raises alerts on anomalies, and includes a dead-man's switch that halts trading if the system itself stops responding.

In simple words

Monitoring is the system watching itself. It tracks whether every part is alive and behaving, shows it on a dashboard, and alerts you when something is wrong, often before it costs money. The most important piece is the dead-man's switch: if the system goes quiet, that silence itself triggers a safe shutdown.

Purpose

An automated system fails silently by default, a frozen feed, a crashed process, a stuck order look the same as nothing happening. Monitoring makes failure visible and actionable, turning silent breakage into a loud, early alert.

Professional explanation

Single responsibility

Monitoring answers is the system healthy and behaving as expected, right now. It collects metrics and health signals from every component, evaluates them against expectations, surfaces them on dashboards, and fires alerts when something deviates. It does not fix problems (that is failover, error handling and the kill switch), but it is the detection layer that triggers those responses. The crucial framing for trading is that the default failure mode is silence: a dead process, a frozen feed and an idle-but-fine system can look identical, so monitoring must distinguish healthy quiet from broken quiet.

Health checks and heartbeats

A health check is an active probe: is the process up, is the broker connection authenticated, is the data feed fresh, is the clock in sync. A heartbeat is a periodic signal each component emits to prove it is alive and progressing, not just running. The distinction matters: a process can be running (up) but stuck (not progressing), and only a heartbeat tied to actual work, the feed handler emitting a beat each time it processes ticks, catches a stuck component. Missing heartbeats are one of the strongest signals of trouble, because they catch the frozen-but-alive state that a simple up/down check misses.

Metrics and dashboards

Monitoring is built on metrics: quantitative signals sampled over time, such as feed tick-rate, order latency, fill rate, error rate, queue depth, P&L, exposure versus limits, and CPU and memory. Dashboards turn these into an at-a-glance picture so an operator can see the system's state without reading logs. The most useful dashboards foreground the safety-critical signals, last-tick age per symbol, current loss versus the daily limit, margin used, reconciliation status, so that the things that would trigger a halt are the things most visible. Metrics also give history, letting you see a slow degradation (memory creeping up, latency rising) before it becomes a failure.

Alerting

Alerts convert a detected anomaly into a human (or automated) response. Good alerting is disciplined: alert on conditions that need action, route them to a channel that reaches you (push, SMS, a messaging app) with enough context to act, and set thresholds to catch problems early without crying wolf. Alert fatigue is a real failure mode, if everything alerts, nothing does, so tier alerts by severity and reserve the loudest for safety-critical conditions (loss nearing the limit, feed down, reconciliation break, kill switch fired). For an unattended system, some alerts should trigger automated action rather than only notify, because a human may not be watching at 10:30 on a volatile morning.

The dead-man's switch

A dead-man's switch inverts the usual logic: instead of acting when something bad is detected, it acts when the expected all-is-well signal stops arriving. The trading system emits a regular heartbeat to an independent watcher; if the watcher stops receiving heartbeats for a set interval, it assumes the system has failed and takes a safe action, typically cancelling open orders and flattening or blocking new trades. This is vital because the most dangerous failure is the one where the system dies but its orders live on: a crashed process cannot cancel its own resting orders, so an external, independent switch that reacts to silence is what protects you. It must run separately from the system it watches, or it dies with it.

How monitoring fails

Monitoring fails by watching the wrong things (green dashboards while the system loses money because no metric tracks the real risk), by alert fatigue (real alerts lost in noise), by the monitoring sharing fate with the system (same host, same process, so it dies together and reports nothing), and by heartbeats that prove liveness but not correctness (a component beats while producing garbage). The dead-man's switch specifically fails if it is not truly independent, if it runs in the same process it monitors, its silence when that process dies means no heartbeat and, if wired wrong, no action either. Robust monitoring runs on separate infrastructure and is itself monitored.

Observability and testing

Monitoring is the observability layer, but it too must be verified. Test that alerts actually fire and reach you, deliberately kill a component to confirm the heartbeat and dead-man's switch respond, and simulate a stale feed to confirm the staleness alert triggers. Review after every incident whether monitoring caught it and how early. Combine metrics (the what and when), logs (the why) and traces (the path through components) so detection leads quickly to diagnosis. An alerting path that has never been tested is as unreliable as a smoke detector with no battery, so exercising it is part of the design, not an extra.

Practical example

Illustrative example (Indian market)

A Nifty bot emits a heartbeat every 5 seconds to an independent watcher running on a separate host. Its dashboard foregrounds four tiles: age of the last Nifty tick, running P&L versus the 15,000-rupee daily limit, margin used, and reconciliation status. One morning the feed handler freezes: the process is still up, so a naive up/down check stays green, but the tick-age tile climbs past 5 seconds and a staleness alert fires to the operator's phone within seconds. Separately, the heartbeat stops; the watcher, seeing no beat for 20 seconds, invokes the kill path to cancel resting orders as a precaution. Because the watcher runs independently, it would have acted even if the whole trading process had crashed, precisely the case where the system could not have protected itself.

For an unattended intraday F&O bot, a dead-man's switch is especially important around forced square-off and high-volatility windows: if the bot crashes near the open with resting orders, only an independent watcher can cancel them, because the dead process cannot, and an uncancelled resting order in a fast-moving Bank Nifty option can fill at a wildly different price than intended.

Advantages

  • Turns silent, invisible failures into loud, early, actionable alerts
  • Heartbeats catch the frozen-but-running state a simple up/down check misses
  • Dashboards foreground safety-critical signals for instant situational awareness
  • A dead-man's switch protects even when the system cannot protect itself

Limitations

  • Only catches what it measures; an unmonitored failure mode stays invisible
  • Alert fatigue can bury real alerts among noise if thresholds are poor
  • Monitoring on the same host or process can die with the system it watches
  • Heartbeats prove liveness, not correctness; a component can beat while malfunctioning

Why it matters in practice

  • The difference between a caught glitch and a costly runaway is often just monitoring
  • For unattended systems, monitoring plus a dead-man's switch is what makes running safe

Common mistakes

  • Relying on a process up/down check while a frozen-but-running component goes undetected
  • Running the monitor or dead-man's switch in the same process it is meant to watch
  • Over-alerting until real alerts are lost in noise (alert fatigue)
  • Monitoring technical metrics but not the safety-critical ones like loss versus limit
  • Never testing that alerts actually fire and reach a human
  • Assuming a heartbeat proves correctness when it only proves the component is running

Professional usage

Serious trading operations run monitoring as independent infrastructure: metrics and heartbeats from every component flow to a separate monitoring stack with dashboards and tiered alerting, and an external dead-man's switch reacts to loss of heartbeat by flattening or blocking trading. They foreground safety-critical signals, tier alerts so the loudest are reserved for real danger, and regularly test the alerting and kill paths by injecting failures. The governing principle is that the monitoring must not share fate with the system it watches, and that silence is treated as a failure, not as calm.

Key takeaways

  • Treat silence as failure: distinguish healthy quiet from broken quiet with heartbeats
  • Foreground safety-critical metrics (loss vs limit, feed age, reconciliation) on the dashboard
  • Tier alerts to avoid fatigue and route the loudest to reach you fast
  • Run the monitor and dead-man's switch independently so they survive the system's death

Frequently asked questions

What is monitoring in a trading system?
It is the component that continuously observes the system's health through metrics, health checks and heartbeats, shows it on dashboards, and raises alerts on anomalies so failures become visible and actionable rather than silent.
What is a heartbeat in system monitoring?
It is a periodic signal a component emits to prove it is alive and making progress. Missing heartbeats catch the dangerous state where a process is still running but stuck, which a simple up/down check would miss.
What is a health check?
It is an active probe of whether a component is functioning: is the process up, is the broker connection authenticated, is the data feed fresh, is the clock in sync. It answers whether the system is in a fit state to trade.
What is a dead-man's switch in trading?
It is a safety mechanism that acts when an expected all-is-well signal stops arriving. The system sends heartbeats to an independent watcher, and if they stop, the watcher assumes failure and cancels orders or halts trading.
Why must the dead-man's switch be independent?
Because the failure it protects against includes the whole system crashing. If it ran in the same process, it would die too and take no action, so it must run on separate infrastructure to react to the system's silence.
What metrics should a trading system monitor?
Safety-critical ones first: feed tick-age per symbol, current loss versus the daily limit, margin used and reconciliation status, plus operational ones like order latency, fill rate, error rate, queue depth and resource usage.
What is alert fatigue and why does it matter?
It is when so many alerts fire, many of them noise, that real alerts get ignored. It matters because a buried critical alert is as bad as no alert, so alerts should be tiered by severity and thresholds tuned to catch real problems.
How is monitoring different from logging?
Monitoring watches live metrics and health to detect problems now; logging records detailed history to diagnose them later. Monitoring answers what and when; logging answers why. They work together.
Should alerts trigger automated action?
For an unattended system, some should. A human may not be watching during a volatile morning, so safety-critical conditions like a feed outage or a loss limit approaching may need automated halting, not just a notification.
How do I detect a frozen but running component?
With a heartbeat tied to actual work, so a component that has stopped processing stops beating even though its process is still up. A plain process-alive check cannot tell running from stuck.
What should a monitoring dashboard show?
The safety-critical signals foregrounded: last-tick age, loss versus limit, margin used and reconciliation status, so the conditions that would trigger a halt are the most visible, alongside operational metrics for context.
How do I know my monitoring actually works?
By testing it: deliberately kill a component to confirm the heartbeat and dead-man's switch respond, simulate a stale feed to confirm the staleness alert fires, and verify alerts reach you. Untested alerting is unreliable.
Can monitoring itself fail?
Yes, by watching the wrong metrics, by alert fatigue, or by sharing a host or process with the system so it dies together. Robust monitoring runs on separate infrastructure and is itself monitored.
Does a heartbeat guarantee a component is working correctly?
No. A heartbeat proves the component is alive and progressing, not that its output is correct. A component can beat while producing garbage, so correctness needs additional checks like data validation and reconciliation.

Voice search & related questions

Natural-language questions people ask about Monitoring.

What does monitoring do for a trading bot?
It watches the bot for you, checks every part is alive and behaving, and alerts you the moment something is wrong, often before it costs money.
What is a dead-man's switch in trading?
It is a safety that acts when the bot goes silent. If the bot stops checking in, an outside watcher assumes it crashed and cancels the orders.
Why is a silent trading bot dangerous?
Because a crashed bot can leave live orders sitting in the market that it can no longer cancel. Silence is the warning, so something has to watch for it.
How do I know if my bot froze?
Use a heartbeat. If the bot stops sending its regular check-in signal, you know it is stuck even though the process still looks like it is running.
Why do I get too many alerts from my bot?
Probably every little thing is set to alert. Tier your alerts so only the serious ones, like nearing your loss limit, are loud.
Where should my monitoring run?
On separate infrastructure from the bot, so if the bot's machine dies, the monitor is still alive to notice and react.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Logging
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.