Finding the edge between P and Q, and retiring the system that got me there

A clean-sheet rebuild around one question: where does our conditional distribution of terminal prices diverge from the one the option market is quoting? Q comes out of the chain by Breeden–Litzenberger; P is a lookup into frozen, measured history with no live fitting; the edge is the difference, priced in dollars per contract. It inherits its predecessor's knowledge, data and measured results, and refuses to inherit a line of its code. Repository: compass, private — 47 programme documents.

1. Why I built it

Two reasons, and the second is the interesting one.

The modelling reason. The predecessor ranked candidates well but its edge estimate was assembled from signals and heuristics rather than derived from a distribution. I wanted the sharper formulation: an option is a claim on the terminal price, the market quotes a full risk-neutral distribution across every strike, and if I can produce a conditional physical distribution that is genuinely different from that one, the difference is expressible in dollars per contract — not as a score. Crucially, this framing is self-invalidating: if my cone merely reproduces the market's implied cone, I have no edge and the system says so.

The engineering reason. The predecessor worked and had become expensive to change. It carried three generations of interface in one tree and design documents recording superseded decisions beside live ones. It had run daily against live data with real money on the outcome, so whatever else was true, its measured results were earned. The question was how to keep those and leave everything else.

2. What I built

A charter-first rebuild with an explicitly gated lifecycle — Requirements → Design → Architecture → UI model → Build — where each phase ends at a human decision and the build phase starts with the smallest end-to-end slice rather than the most interesting component. Three modelling pillars, one grading spine, and a migration apparatus that decided asset by asset what deserved to survive.

The three pillars.

1. Identify the state. One market-state posterior over a component field — participation, dispersion, leadership, damping, fuel — read as measured cell frequencies. Entry and exit are framed as a quickest-detection problem: identification is the exit.
2. Put an interval around it. One conditional distribution, graded as a distribution, honest about its own calibration rather than its own accuracy.
3. Price the surface. Expected value under that conditional distribution, minus the market's price, in dollars, across strike × expiry × name.

3. How it works

flowchart TB
  subgraph p["P — our conditional cone"]
    FEAT["Physical feature vector<br/>per symbol-minute"]
    CELL["Discretise into a CELL<br/>edges frozen on the fit era"]
    POST["Posterior = recency-weighted<br/>empirical frequency of horizon<br/>outcomes in that cell<br/><b>no live fitting</b>"]
  end
  subgraph q["Q — the market's cone"]
    CHAIN["Live option chain<br/>recorded at 1-minute cadence"]
    BL["Breeden–Litzenberger<br/>strike-by-strike recovery"]
    RISKN["Risk-neutral density<br/>carries variance premium<br/>and crash skew"]
  end
  subgraph edge["The price map"]
    THEO["THEO = expected payoff at close<br/>under the recalibrated cone"]
    SPAN["Only strikes inside the<br/>cone's 5–95% span,<br/>two-sided quotes only"]
    DOLLARS["EDGE = THEO − market price<br/>published every 15 minutes"]
  end
  subgraph grade["The grading spine"]
    CLAIM["Every claim hash-chained<br/>when published"]
    VERDICT["RIGHT · WRONG · MIXED<br/>· UNGRADEABLE(reason)"]
    CAL["Calibration, not confidence<br/>distributions scored as distributions"]
  end
  FEAT --> CELL --> POST
  CHAIN --> BL --> RISKN
  POST --> THEO
  RISKN --> THEO
  SPAN --> THEO --> DOLLARS
  DOLLARS --> CLAIM --> VERDICT --> CAL
  CAL -. "recalibration" .-> POST

P is a lookup, not a fit. For every symbol and minute the system computes a small feature vector and discretises it into a cell whose boundaries were frozen on the fit era. The posterior for that cell is the recency-weighted empirical frequency of what actually happened, across every historical minute that sat in the same cell, pooled across the fleet. There is no live parameter estimation anywhere in the path. This is a deliberate trade: it gives up adaptiveness and buys the thing that matters more, which is that the model cannot quietly overfit to the last three weeks. The model is measured history.

Q is recovered, with its biases named. The market's terminal density comes strike-by- strike out of the live chain. The documentation is explicit that Q is risk-neutral and therefore embeds a variance premium and crash skew — fatter-tailed and more negatively skewed than the physical distribution. Naming those biases in the design document is what stops the difference between P and Q being read as pure alpha when part of it is a risk premium you are being paid to bear.

The price map is denominated in dollars. At each completed fifteen-minute bar, for every same-day-expiry contract with a two-sided quote whose strike sits inside the cone's 5–95% span, the system publishes a theoretical value and the edge against the market price. Not a score, not a rank — a dollar figure you can check at the close.

4. What is interesting and unique

Inheritance is a per-asset decision with a written record. Before any code was written, every asset in the predecessor estate was catalogued in a reuse census: one row per design document, law, research record or measured result, with columns for maturity (was-live / ratified / research-frozen / benched), the slot it would occupy in the new system, the mode (port now, port later, rebuild clean, or refuse), the effort, and the reason. Nothing crossed by default. A thing survived because someone wrote down why, in a row that names the alternative.

There is a refusal register, and it is the more useful artefact. It records what was deliberately not carried forward, and why. The reasoning behind a rejection evaporates much faster than the reasoning behind a decision: six months on, an undocumented absence is indistinguishable from an oversight, and somebody helpfully reintroduces the thing that was removed on purpose. The register turns "we don't do that" from folklore into something citable — and it makes the rebuild auditable, because a reviewer can ask why something is missing and get an answer.

Patterns were rebuilt; files were not imported. The census explicitly separates the two. Several of the predecessor's strongest assets — including a calibration corpus of hundreds of thousands of pairs and a change-detection routine — are listed as patterns to rebuild clean from their description rather than files to copy. That is the whole thesis of the rebuild in one operational rule.

Ungradeable is a first-class verdict. The grading vocabulary is RIGHT · WRONG · MIXED · UNGRADEABLE(reason), and the fourth carries a required reason. Most scoring systems quietly drop what they cannot score, which biases the record toward the claims that happened to be measurable. Making unmeasurability explicit, with a reason attached, keeps the denominator honest.

Distributions are graded as distributions. Where the output is a cone it is scored against a proper scoring rule rather than collapsed to a point estimate and scored on accuracy — which would reward confidence instead of calibration, exactly backwards for anything sizing off the interval.

5. Challenges I had to solve

Separating edge from risk premium. P minus Q is not automatically alpha. Q is risk-neutral; some of the gap is compensation for bearing variance and tail risk. The design handles this by decomposing the two cones explicitly and stating the known direction of Q's biases, so the divergence that matters is the conditional, non-symmetric part where the state model genuinely knows something the chain does not — not the static premium that is always there.

Not fitting on the era you trade. Cell edges are frozen on the fit era and validation bars were frozen before the state-model design document existed. Pre-registering the bars before the design is the only reliable defence against the researcher's version of look-ahead, which is choosing the test after seeing the candidate.

Grading something that publishes every fifteen minutes. The spine's own design document is blunt that this became overdue: the price map was publishing hash-chained verdicts every fifteen minutes and nothing was scoring them. The answer was one shared vocabulary and one append-only ledger, with grading rules versioned so a re-grade under a new rule cannot silently rewrite history.

Refusing to be reassured by a parallel run. The charter specifies a single day running alongside the old system and calls it exactly what it is — a plumbing smoke test, nothing more. It does not claim equivalence. It is far easier to write "validated against the incumbent", and that sentence would have been worth nothing.

6. What I learned

A self-invalidating thesis is worth more than a flexible one. "If our cone reproduces the market's cone we have nothing" is a statement that can fail, and building around it means the system can tell me it has no edge today. A framework that can always find something is a framework that will.

Lookup beats fitting when the sample is thin and the regime moves. Giving up live adaptation to guarantee the model is measured history was the right trade. Most of the damage I have seen in this domain comes from models that fit beautifully to the period they were built in.

Record refusals as carefully as decisions. This generalises well past trading. Every system I have built accumulates things that were removed for good reasons nobody wrote down, and every one of them eventually comes back.

Retire on schedule, not at collapse. The predecessor was working when it was retired. That is the only point at which you have the time and the clarity to do the census properly — and the census is what made the rebuild an inheritance rather than a restart.

The repository is private. Tuned parameter values, cell definitions, vendor identities and infrastructure detail are omitted; the methods, contracts and decision records are what is described here. Redacted design documents are available on request.