Home / System Architecture / Logging
ComponentIntermediate

Logging

Logging is the cross-cutting component that records a structured, append-only, timestamped account of every decision, order and event so the system's behaviour can be reconstructed and audited after the fact.

Quick answer: Logging is the cross-cutting component that records a structured, append-only, timestamped account of every decision, order and event so the system's behaviour can be reconstructed and audited after the fact.

In simple words

Logging is the system's flight recorder. When something goes wrong, and in trading it will, the logs are how you find out what the system saw, decided and did. Good logs are structured, never overwritten, and detailed enough that you can replay the story of any trade from them.

Purpose

You cannot watch an automated system every second, and you cannot debug real-money incidents from memory. Comprehensive, structured logs are the only way to reconstruct what happened, prove what the system did, and fix it.

Professional explanation

Single responsibility and what makes it different in trading

Logging records what happened, as it happened, in a form you can query later. In trading it is not optional developer convenience: it is an audit trail for real-money decisions, a debugging tool for incidents you cannot reproduce, and often a compliance requirement. The distinguishing demands are that logs be append-only (you never rewrite history), precisely timestamped (ideally to milliseconds, in a single timezone), and complete enough to reconstruct any decision. A trading log answers, months later, exactly why the system bought 2 lots at 10:47:03 and what it knew at that instant.

Structured logging

Free-text logs are hard to search and impossible to aggregate. Structured logging emits each entry as machine-readable key-value data (typically JSON) with consistent fields: timestamp, level, component, event type, and event-specific fields like order id, symbol, price, quantity, and a correlation id linking related events. Structure lets you query (show every order for this symbol today), aggregate (count rejects per hour), and trace a single logical action across components via a shared correlation id. The correlation id, threaded from a signal through sizing, risk, OMS and fill, is what lets you reconstruct one trade's full journey from interleaved logs.

Append-only and auditability

Logs must be append-only and tamper-evident: you add records, you never edit or delete them. This matters because the log is evidence, of what the system did, for debugging, for reconciliation, and potentially for regulators or dispute resolution with a broker. Practically this means writing to append-only files or an append-only store, rotating (not truncating) files, retaining them per a policy, and ideally shipping them off the trading host so a crash or a compromise cannot erase them. An audit log that can be quietly rewritten is worthless as evidence.

Log levels and discipline

Levels let you separate signal from noise: DEBUG for fine-grained developer detail, INFO for normal significant events (order placed, filled, position changed), WARN for recoverable anomalies (a retry, a reconnect, a rejected order), ERROR for failures needing attention, and CRITICAL for events that threaten the system (kill switch fired, reconciliation break). In production you typically log INFO and above to keep volume manageable, with the ability to raise verbosity when investigating. Discipline matters: over-logging at high levels buries real signals, while under-logging leaves gaps exactly where you need detail. The rule of thumb is that anything you would need to explain a trade or an incident must be logged at INFO or above.

What to log in a trading system

Log the decision chain, not just the outcome. For each trade: the signal and the state that produced it (key inputs, indicator values), the sizing computation, the risk decision and which checks ran, every order state transition, every fill, and the resulting position and P&L. Also log system-level events: startups and shutdowns, config loaded (and its version), connections and disconnections, reconciliation results, and every warning or error with context. What not to log: secrets (API keys, tokens) must never be written, and personally sensitive data should be avoided. The test is whether, from the logs alone, you could reconstruct exactly what the system saw and did.

How logging itself fails

Logging can undermine the system it observes. Synchronous logging on the hot path can add latency to order placement; the fix is asynchronous logging via a queue so the trading thread is not blocked. Unbounded logs fill the disk and crash the process, so rotation and retention are mandatory. Logs lost on crash defeat the purpose, so critical events should be flushed durably, and ideally streamed off-host in near real time. And a logging failure should never take down trading, but a silent logging failure (disk full, log shipper down) means you are flying blind, so the health of logging itself should be monitored.

Observability

Logging is a foundation of observability but benefits from being observable itself. Monitor log volume and error/warn rates as metrics, because a sudden spike in WARN or ERROR is often the earliest sign of trouble, sometimes before a limit is breached. Centralise logs from all components so a single query spans the system, and index them so retrieval during an incident is fast, not a grep through gigabytes. Alert on CRITICAL log events directly. The relationship is symbiotic: monitoring watches metrics for the what, logs provide the why, and together they let you both detect and diagnose.

Practical example

Illustrative example (Indian market)

A Nifty bot logs each trade as a chain of structured JSON records sharing a correlation id. At 10:47:03.120 it writes an INFO signal event with the EMA values and current position; at 10:47:03.121 a sizing event with budget, stop and computed lots; at 10:47:03.122 a risk event listing the checks passed; then an OMS event with the client order id, an acknowledgement, and a fill at 10:47:04.010. Weeks later, investigating why that trade happened, an operator queries the correlation id and reads the entire story, inputs, decision, order, fill, in order, without guessing. Because the logs are append-only and shipped off-host, they survive the later crash that took down the trading process, and the WARN spike five minutes before the crash, visible in the log-rate metric, turns out to have been repeated feed reconnects.

For an unattended F&O bot, precise millisecond timestamps in IST (stored unambiguously, for example as UTC with an offset) are what let you line your logs up against the broker's order-time and the exchange trade-time when reconciling or disputing a fill, so a vague or timezone-ambiguous timestamp can make a log useless exactly when you need it most.

Advantages

  • Reconstruct any decision or trade after the fact from a complete record
  • Structured, correlated logs make the whole system searchable and traceable
  • Append-only logs serve as trustworthy audit evidence
  • Log-rate and error-rate metrics give early warning of trouble

Limitations

  • Synchronous logging on the hot path adds latency if not made asynchronous
  • Log volume can be large and costly to store, index and retain
  • Logs are only as useful as the discipline behind what and how you log
  • Logs on the trading host can be lost in the same crash unless shipped off-host

Why it matters in practice

  • Without good logs, a real-money incident is often impossible to diagnose
  • Logs are frequently the deciding evidence in a broker or exchange dispute

Common mistakes

  • Writing free-text logs that cannot be queried or aggregated when it matters
  • Logging secrets such as API keys or tokens into files that persist
  • Rewriting or truncating logs, destroying the audit trail
  • Logging synchronously on the order hot path and adding latency
  • No log rotation or retention, so the disk fills and the process crashes
  • Logging only outcomes, not the inputs and decisions, so trades cannot be explained

Professional usage

Professional trading operations treat logs as a durable, centralised, structured asset: every component emits JSON with correlation ids to a central log store, timestamps are precise and unambiguous, and logs are append-only and retained per policy for audit and compliance. They log the full decision chain so any trade is explainable, ship logs off the trading host in real time, and alert on error-rate spikes and critical events. The mindset is that if it is not logged, it did not happen, and reconstructing behaviour is a routine capability rather than a scramble.

Key takeaways

  • Log the full decision chain, not just outcomes, so any trade can be reconstructed
  • Use structured, correlated, precisely timestamped logs you can query and trace
  • Keep logs append-only and ship them off-host so they survive as audit evidence
  • Log asynchronously off the hot path, rotate and retain, and never log secrets

Frequently asked questions

Why is logging important in a trading system?
Because you cannot watch an automated system every second or debug real-money incidents from memory. Logs are the record that lets you reconstruct exactly what the system saw, decided and did, serving debugging, reconciliation and audit needs.
What is structured logging?
It is emitting each log entry as machine-readable key-value data, usually JSON, with consistent fields like timestamp, level, component, event type and order id. Structure makes logs searchable, aggregatable and traceable, unlike free text.
What does append-only logging mean?
It means you only ever add log records and never edit or delete them, so the log is a tamper-evident historical record. This is essential for logs to serve as trustworthy audit evidence.
What log levels should a trading system use?
Typically DEBUG for developer detail, INFO for normal significant events, WARN for recoverable anomalies, ERROR for failures needing attention, and CRITICAL for system-threatening events like a kill switch firing. Production usually logs INFO and above.
What should I log in a trading system?
The full decision chain: the signal and its inputs, the sizing computation, the risk decision, every order state transition and fill, and the resulting position and P&L, plus startups, config versions, connections, and reconciliation results.
What is a correlation id in logging?
It is a shared identifier threaded through all the events of one logical action, from signal through sizing, risk, order and fill, so you can query it and reconstruct that single trade's full journey from otherwise interleaved logs.
Should I log API keys or secrets?
Never. Secrets like API keys and tokens must never be written to logs, because logs persist and are often shipped off-host, so logging a secret is a serious security exposure.
How do I stop logging from slowing down trading?
Log asynchronously: the trading thread writes to an in-memory queue and a separate thread persists the entries, so order placement is not blocked by disk or network writes on the hot path.
How long should trading logs be retained?
Long enough to satisfy debugging, reconciliation and any regulatory or dispute needs, which often means months. This requires rotation and a retention policy so logs do not fill the disk while still being available when needed.
Why ship logs off the trading host?
Because a crash or compromise on the trading host could otherwise destroy the logs exactly when you need them. Streaming logs to a central store in near real time ensures they survive the incident they document.
What is the difference between logging and monitoring?
Logging records the detailed why of what happened for later reconstruction; monitoring watches live metrics and health for the what and now. They are complementary: monitoring detects, logs diagnose.
How precise should log timestamps be?
Ideally to the millisecond and in a single, unambiguous timezone, because you often need to line logs up against broker order times and exchange trade times, where coarse or ambiguous timestamps make reconciliation impossible.
Can logging failures affect trading?
A logging failure should never take down trading, but a silent one, such as a full disk or a down shipper, leaves you blind. So the health of logging itself should be monitored and alerted on.
How do logs help with reconciliation?
They record every order and fill with precise timestamps and ids, so when the system's position disagrees with the broker's you can trace exactly which orders and fills were applied and find the missed or duplicated event.

Voice search & related questions

Natural-language questions people ask about Logging.

Why does a trading bot need logs?
Because when something goes wrong you cannot ask the bot what it was thinking. The logs are the recording of what it saw, decided and did.
What is structured logging in simple terms?
It means writing logs as neat labelled data instead of plain sentences, so you can search and filter them easily later.
What should my trading bot log?
The whole story of each trade, the signal, the size, the risk check, the order and the fill, plus start-ups, errors and reconnects.
Should I ever log my API key?
No, never. Anything secret should stay out of the logs because logs get saved and copied around.
Why keep logs I never edit?
So they stay a true record. If you could rewrite them they would be useless as proof of what actually happened.
How do logs help when a trade goes wrong?
You follow the trade's id through the logs and read exactly what the bot knew and did at each step, so you can find the mistake.

Sources & references

    Last reviewed 11 July 2026. Educational content only — not investment advice. Markets and rules change; verify current conventions with SEBI, NSE/BSE and your broker.

    ← Previous
    Order Management System (OMS)
    Educational content only — not investment advice. Examples use illustrative numbers and simplified models. Algorithmic trading and derivatives involve substantial risk. See our Risk Disclosure and SEBI Disclaimer.