From tick stream to trained model to a system that paused itself

A market-data and ML platform, an autonomous trading system, and the controls that governed it — built by directing AI coding tools against written specifications. The repositories stay private; what is described here is the architecture and the operating discipline, not the trading logic.

What it is. Three connected systems: NovaHub, a market-data and ML platform; NovaQuant, an autonomous trading system; and novaquant2, its event-driven successor. Together they ingest a live trade stream for several hundred symbols, run thirty scheduled data feeds, keep a billion-row training store on on-demand GPUs, train both neural-network and gradient-boosted models, and place orders behind a gatekeeper, a risk governor and a kill switch. The money at risk was my own, which is the most demanding product review there is.

How it was built. By directing AI coding tools (Cursor, Windsurf and Claude) against written specifications: numbered PR prompts, a design document for every model pipeline, a coding standard the tools were required to read first, architecture decision records, and a prompt log. About 680,000 lines of code, 540 test files and three CI workflows in roughly ten weeks — and, more to the point, the specifications and decision logs that make that volume auditable.

Architecture

flowchart TB
  subgraph feeds["Data feeds"]
    ws["Trade tick stream<br/>(WebSocket, ~675 symbols)"]
    flow["Options-flow stream"]
    sched["30+ scheduled feeds<br/>OHLCV · options · macro · fundamentals · news · filings"]
  end
  subgraph hub["Data & ML platform"]
    sm["Stream manager<br/>(single WebSocket owner)"]
    bars["1s → 1m bars<br/>+ dollar / imbalance bars"]
    ts[("Time-series DB<br/>(primary)")]
    ch[("Columnar analytics DB")]
    feat["Feature engine<br/>(every 60 s)"]
    api["REST + WebSocket API"]
    dq["Data-quality scan · freshness checks · watchdogs · catch-up"]
  end
  subgraph gpu["On-demand GPU (rented)"]
    vol[("Training store<br/>~1B rows, schema-mirrored")]
    tf["Transformer<br/>(multi-stock returns)"]
    gb["Boosted cascade<br/>regime → viability / MFE → stop"]
    exp["Experiment runner<br/>cached features"]
  end
  subgraph trade["Autonomous trading system"]
    gate["Data-quality gate<br/>strict data mode"]
    sig["Signal + scoring"]
    gov["Universal Governor<br/>FULL_AUTO / HITL / MANUAL / EMERGENCY"]
    risk["Gatekeeper + risk governor"]
    exec["Broker adapters<br/>(paper / live, guarded)"]
    ledger[("Reconciliation ledger")]
  end
  ws --> sm --> bars --> ch
  flow --> ch
  sched --> ts --> feat --> ch
  ch --> api
  ch -- "nightly sync" --> vol --> tf & gb & exp
  tf & gb -- "models, same schema" --> api
  api --> gate --> sig --> gov --> risk --> exec --> ledger
  dq -.-> ts & ch

The parts worth explaining

A trade stream that never touches disk. One process owns the WebSocket for a universe of about 675 symbols and consumes trades only. It derives one-second and one-minute bars in memory and, in parallel, builds dollar bars — bars that close when a fixed amount of dollar turnover has traded rather than when a clock ticks — and imbalance bars that close when signed buying-versus-selling flow crosses a threshold. Thresholds come from each symbol's trailing 20-day turnover, computed without look-ahead, targeting about 150 bars per regular session. Raw ticks are discarded once bars form; the design document makes that a locked constraint, because storing them would have cost more than they were worth. The build was phased on purpose: one symbol for one day, then a week, a month, four years across the universe, and only then the live consumer, tested by replay.

Thirty feeds with a conscience. End-of-day and intraday jobs for bars, options chains, options analytics, fundamentals, transcripts, news, filings, order-flow, macro series and corporate actions, on their own queues. A freshness check runs every five minutes during market hours; if a feed is late, an end-of-day catch-up runs automatically; a daily data-quality scan and a weekly provider-coverage check close the loop. Data contracts are enforced as integration tests at the database boundary, not just in application code.

A schema contract between training and production. A billion rows (about 45 GB compressed) were synced from the analytics database to an on-demand GPU store in four hours, then nightly. The platform's first rule is that the training store must mirror the production schema exactly, so a model trained on the GPU side runs unchanged where inference happens. That rule is boring and it is the reason the models could be deployed at all.

Two model families, honestly scored. A Transformer encoder that predicts returns for many symbols jointly, trained on T4 GPUs; and a gradient-boosted cascade — a daily regime gate feeding three parallel models (short-horizon viability, and expected favourable excursion at two horizons) that in turn feed a stop-strategy model. The cascade design forbids the parallel models from seeing each other's outputs, lags the regime by a day, and holds out the last five months of data untouched for simulation only. Hyperparameters were tuned on GPU against a selection-level objective. The results are recorded as they were: the fifth model version scored an AUC of 0.55 — essentially random — and the repository documents why (too few features, too little data, no volatility or options context) and what the sixth version changed. Later runs reached AUC in the high 0.50s with useful lift only at the top of the ranking. No version is described as profitable, because none was proven to be.

Controls that fail closed. The trading system runs in strict data mode — it stops rather than degrades when data is missing — and a data-quality gate sits before any signal. Every change to a position's state goes through a single write path so state can be replayed after a restart. Orders pass a gatekeeper and a risk governor, then a guarded broker adapter, and land in a reconciliation ledger. Point-in-time determinism — the guarantee that a backtest only sees what was known at the time — is a test in CI, not a promise.

The system paused itself. In January 2026 an architecture review of the second-generation system found gaps in how operating modes were enforced across its runners and stopped it: "system paused — do not enable until fixed." The fix was a Universal Governor: one service that owns the operating mode, a permission matrix (fully automatic, human-in-the-loop for entries, manual, paused, emergency kill switch), a registry mapping every runner to a category, and fail-closed defaults so anything unknown is blocked. The review and the fix are both in the repository. The daily system I run today grew out of this estate, with those controls carried forward.

What this shows

Directing AI-assisted engineering at real scale, with the specifications, prompts and decisions preserved. Real-time data engineering under an explicit cost constraint. An ML platform whose central idea is a contract, not a model. Measurement that is allowed to disappoint. And the operational instinct to stop a live system when the review says so.

The repositories hold trading logic and broker integration and remain private. Vendor names, symbols, thresholds and infrastructure details are omitted here on purpose. Redacted screenshots and the design documents are available on request.