jvm.chisel.db.observe
The feedback half of the scheduler loop — the “monitor execution, reconfigure the next query” arc from the essay, built from pieces that already exist.
estimate/cardinalities ─▶ schedule/run-plan ─▶ actual-cardinalities ▲ │ │ correction-factors │ │ └──────────── make-cost-fn ◀────────────────┘
The cost model estimates per-node cardinalities from a uniform-domain assumption; real data is skewed, so the estimate is systematically off. One observed run of a query shape yields per-kind correction factors — plain data, persistable — that price the next query of that shape honestly. The factors map is deliberately the shape a learned model will later fill: same key space (resource kinds), same multiplication point (estimate/estimate-cost’s 2-arity), no control-plane changes.
Nothing here touches hardware; run-plan is the golden reference, so the observations are exactly what the composed datapath would produce.
actual-cardinalities
(actual-cardinalities run)Per-node observed output cardinality, read from a run-plan result’s :env.
calibration
(calibration plan input)Per-node estimate vs observation for one (plan, input) run: {node-id {:est e :actual a :ratio a/e}} — the human-facing report (“you predicted 0.03 rows here, 4 came out”). :ratio is nil where the estimate is zero. correction-factors is the machine-facing fold of the same run.
correction-factors
(correction-factors plan input)Fold one observed run into per-kind multipliers. For each filtering node the factor is its selectivity error, (actual-out/actual-in) ÷ (est-out/est-in): charging the correction to the node that caused it, so an upstream mis-estimate is not re-charged to the pass-through nodes below it. Factors are averaged per resource kind; nodes with a zero estimate or zero observed input contribute nothing. Feed the result to make-cost-fn.
make-cost-fn
(make-cost-fn factors)Wrap correction factors into a drop-in :cost-fn for schedule/schedule or cluster/admit. (make-cost-fn {}) is the uncorrected estimate.