MESSAI collab · database work · measured 2026-11-16

How a number gets from a paper into a prediction, and how far to trust it

Every value a partner page shows was read from a paper by a model, given a canonical name and an SI unit, judged against physical bounds, and either admitted to the modelling pool or held out with a reason. This page shows each of those steps, the counts at each step on the date they were measured, what has reached production, and the three evals that score the result. Nothing here is a result; it is the state of the machinery.

Asks for reviewCounts are from the staging database, measured read-only on 2026-11-16T14:15Z UTC. Production figures are quoted from the promotion runbooks, not measured here.

What we ask

Tell us where a step would let a wrong number through, or where a count makes you doubt the one you see on your workstream page.

  1. Comment on the line you mean: click any number or sentence on this page, write, and post. Comments need a free account.
  2. A dated count is a measurement, not a claim: the query that produced it is named under the table with its hash, and a re-run will differ as the corpus grows.
  3. “Held” never means deleted. A row that fails a step keeps its value, its quote and its page; it is marked with a reason and left out of the pool.

What happens to a row

Eight steps run inside the same transaction that inserts a row, so a value is never visible half-processed. The replay scripts apply the same rules corpus-wide after a rule changes.

  1. Canonical slug. The parameter name as the paper wrote it (Faradaic efficiency (FE) for Kolbe products) is mapped to one snake_case name (faradaic_efficiency) by one mapper, shared with the replay. A name the mapper does not know keeps its raw text and no slug.
  2. Catalog link. The slug links the row to its parameter definition, the entry the catalog pages and the priors read.
  3. SI value. The reported value and unit are converted to the one unit stored for that quantity (mW/m², mol/L, …). A unit the normaliser does not recognise is held with the reason unit_not_convertible.
  4. Modelable flag. One decision function, the same in TypeScript and Python, says whether the row may feed a model.
  5. Bounds gate. A value outside the physical range for its quantity (a pH of 40) loses the flag and gets a bounds: mark.
  6. Verifier. A heuristic check that the value appears in its own quote and sits inside the slug’s bounds. Rows the verifier fails stay visible.
  7. Off-corpus gate (3 November 2026). A paper that is not about a bioelectrochemical system, because its header or its record says so, has every row held with off_corpus_paper. Slug and SI value are kept, so a Kolbe current density stays comparable on the Kolbe page and never feeds a BES prior.
  8. Modelling-paper gate (16 November 2026). A modelling or simulation paper states model inputs and simulated results, not measurements, so every row is held with modeling_paper. The rows stay visible and comparable; the parameter pages hide them until a reader asks for them.

The pool the fitters read is narrower still: it also drops duplicate facts, rows on hidden, review or modelling papers, physics contradictions and patents. On 2026-11-16 it held 27,474 rows across 233 parameters from 1,765 papers.

The funnel

Live extracted rows at each step, on three dates. “Live” means not archived; nothing is ever deleted.

Step24 Oct26 Oct2026-11-16Change since 26 OctWhy it moved
Live rows204,076219,023214,201-4,8224,918 rows whose unit was not a unit were archived on 28 October; the Kolbe smoke added 77.
With a canonical slug92,141112,009112,022+13Unchanged by the archive: those rows never had a slug.
With a catalog link85,650110,295110,308+13
With an SI value52,42977,12671,572-5,5542,403 of the drop are the archived rows; 3,162 are not yet explained.
Modelable31,47547,69744,488-3,2092,814 rows on off-corpus papers were held on 3 November.
Modelable and verified6,40311,69411,518-176The held rows that had passed the verifier.
Held as off-corpus—82,838+2,830New step; 8 on the Kolbe smoke, then the whole corpus.
Served pool——27,474 / 233—The same before and after the hold: every held row was already out through another rule. A 45,095 figure quoted on 26 October does not reproduce on this query.

15,424 live papers; 4,256 of them marked off-corpus; 1,765 with at least one served row.

Query scripts/collab/measure-pipeline-funnel.sql (sha256 8281dccf8840), run read-only on staging at 2026-11-16T14:15Z UTC.

Modelling and simulation papers

A paper that simulates a reactor reports what its model computed, not what an experiment measured. Since 16 November 2026 the database keeps the two apart. This section says what happened, starting with ten papers from the COMSOL Conference, and how to work with the result.

What happened

  1. 7 October 2026: the audit. A request to add the COMSOL Conference papers back found them already in the database, as broken records: 55 rows for ten papers, with file names as titles, placeholder entries, and DOIs guessed from citations in the PDF text.
  2. 23 October: the repair, on staging. Each of the ten papers was checked against the first page of its PDF and its proceedings page on comsol.com, and became one entry with its real title, authors, venue and abstract. 43 rows (posters, slides, abstracts and copies of the same paper) were merged into their entry, and two placeholders with no document were archived. Every row was copied to an archive table first; nothing was deleted. The guessed DOIs were removed.
  3. What the ten papers are. Two are about microbial electrochemical systems and are visible: Torrents 2010 (pH and carbonate buffering in a lactate microbial fuel cell) and Oyetunde 2013 (bioelectrochemical systems for wastewater treatment). The other eight (enzymatic biofuel cells, electroporation, a micro mixer, flow electrolysers and others) are kept, marked as not bioelectrochemical, and hidden from the paper list.
  4. 16 November: the two papers were read again, and a rule followed. The second reading gave 19 values for Torrents 2010, all of them results of the model. One, a simulated pH of 2.06, came back marked as usable for modelling, as if it had been measured. So the rule became general: every value of a paper typed as modelling or simulation is held with the reason modeling_paper. On staging this held 395 rows, and the pool the fitters read went from 27,726 to 27,474 rows.
  5. 16 November: models against measurements. Held values are not put out of sight. Each parameter page now sets the values that modelling papers state beside what experiments measured, so a model can be checked against the measured range.

Where it stands

Counted on staging on 2026-11-19.

WhatCount
COMSOL Conference papers, one entry each10
Of these, about a microbial electrochemical system and visible2
Duplicate and partial records merged into their entry43
Live papers typed as modelling or simulation569
Values held with modeling_paper396
Model values that pass every other check253 on 27 papers, across 54 parameters
Example, voltage: measured1,904 values from 383 papers; median 0.36 V; 10th to 90th percentile 0.0327 to 0.9 V
Example, voltage: stated by models27 values from 8 papers; median 0.7 V; 5 below, 15 inside and 7 above the measured range

Query scripts/collab/measure-modelling-papers.sql (sha256 bcb0cbad0edd), run read-only on staging at 2026-11-19T12:42Z UTC.

How to work with it

  1. On a parameter page

    Open a parameter, for example /parameters/voltage, and go to “Papers reporting this parameter”. Values from modelling papers are left out of the list until you tick “Show n values from modelling papers”.

  2. The “Models against measurements” card

    In the same section, a card places every model value below, inside or above the 10th to 90th percentile of the measured values, and names the paper and system type each one comes from. A parameter with fewer than five measurements gets no range and no position.

  3. On a paper page

    A modelling paper still lists every value read from it. Each one is marked “Excluded: modelling or simulation paper: model values, not measurements”.

  4. Through the API

    /api/parameters/voltage/model-comparison returns both sides and the position of each model value. /api/parameters/voltage/sources?modeling=hide lists the papers for a parameter without the modelling papers.

  5. When the type looks wrong

    The rule looks at the paper, not at the single value. A paper typed as modelling that also reports its own measurements has those held too; an experimental paper that also reports a simulation keeps all its values. If you see either, comment on the value. The fix is to correct the paper’s type, not to make an exception in the pool.

Limits

  • Staging only. The repair, the second reading and the hold have run on staging (messai.ai). On production (messai.io) the ten papers are still the earlier broken records and the hold has not been replayed; both are pending rows in the table below.
  • One measured range for all system types. A model of one system type is compared with measurements from all of them. Each model value shows the system type of its paper, so a mismatch is visible.
  • Some papers typed as modelling are not about a microbial system. A megawatt fuel-cell power unit appears under voltage. Their type or their verdict needs correcting; the card shows what the database holds.
  • The model itself is not captured. Geometry, mesh, solver settings and boundary conditions are not extracted, only the values a paper states.

What reached production

Production is never written by hand: a change is a committed, ref-gated script that ran on staging first, was verified there, and was then replayed on production after a backup. The pending rows are replays no runbook records as run.

WhatStateWhenSize
Pipeline overhaul: slugs, SI values, harmonise at landing, additive copy of staging rowsOn production26 Oct+19,053 rows; modelable 32,004 → 47,895
Full-text search vector backfillOn production26 Oct23,598 papers, 0 left empty
Not-a-unit quarantineOn production31 Oct4,918 rows archived
Microbe kinetics: growth constants, paper–organism links, electron-transfer rowsOn production31 Oct and 1 Nov+1,522 rows
UFZ collab corpus (Kolbe and MES cohorts)On production3 Nov64 papers, 434 values, fingerprints equal
Per-paper processing statePendingnot recorded15,361 rows on staging
Parameter display names (English and German)Pendingnot recorded0 empty on staging
Three reviewed BES verdicts (28 Oct manifest)Pendingnot recorded3 papers
Off-corpus hold (replay of the new gate)Pendingafter reviewabout 2,800 rows
Modelling-paper hold (replay of the new gate)Pendingafter review395 rows on staging
COMSOL Conference papers: repair of the records, and the second reading of the two about microbial systemsPendingafter review55 rows into 10 entries on staging

26 October, 15:50. After a day of write load the production database answered a probe query in 49.7 s against 0.59 s on staging; the paper list timed out. The compute tier was raised and one sampler query was cut from 9,972 ms to 46 ms. Rule since: run the probe before any bulk job, one stage at a time, never beside a deploy.

29 to 31 October. A migration adding display-name columns was merged but not applied on production; parameter pages answered 500 for about two days. Rule since: a merged migration is not an applied migration; runbooks name the checkout they run from.

Evals and benchmarks

Three committed cards, each rebuilt offline from the repository, score the machinery from three sides: is the extracted value in the paper, does the model match what the paper reported, and are the predictive intervals honest. The numbers below were copied from the cards on 2026-11-03T15:43Z UTC.

Extraction: is the value in the paper?

Precision is the share of extracted values the paper text supports; recall is the share of the gold values the extractor found. The gold set is 20 papers, 6 parameters, 290 truth rows; no person has checked it yet (0 of 290).

Arm or slicePrecisionRecall
CMA agent v3 (gold set)97 %78.4 %
v2 ext families, arm B (gold set)100 %47.3 %
v2 flat values, legacy arm A (gold set)60.3 %56.2 %
v1.2 orchestrator, retired (gold set)78.3 %7.2 %
Live rows, primary papers95.8 %88.3 %
Live rows, v2 extractor99.3 %54.7 %
Live rows in the modelling pool100 %40.6 %

The pool is precise (nothing served is unsupported in these 20 papers) and misses more than half of what the papers report. Reviews score zero because every review value is held as cited by design.

Published anchors: does the model match the paper?

A preset or an API call is scored against the value its paper published. Green is within 25%, amber within 50%, red further. Only the headline rows were not tuned on; agreement with the 51 calibration presets is an in-sample check.

SetRowsGreenAmberRedRefused or no outputMedian error
Headline, lab sweep (13 papers)17728—27.4 %
Headline, API and chat17405851.6 %
All rows, lab sweep84611013—12.4 %

Of the 9 API answers on the headline rows, 6 reported values sat inside the stated interval. The 7 refusals are the predictor declining for want of inputs, which is the designed behaviour.

Priors: are the intervals honest?

Coverage is the share of held-out values that fall inside the 95% predictive interval of the hierarchical priors (v2-holdout-validation-2026-10-28). 95% is honest; higher is wide, lower is biased.

MeasureValue
Held-out papers339
Held-out values5,269
Observed coverage at nominal 95%96.1 %
95% interval on the coverage95.5 % – 96.6 %
Misses low / high122 / 85
Verdictcalibrated

Slightly wide, not biased: 122 values fell below their interval and 85 above, of 5,269. The science gate that blocks a push on a new failure lists 111 known ones and 0 new since 26 October.

What is still weak

Known, named and not fixed on 2026-11-16. Each is a place a scientist could find a wrong number or a missing one.

  1. A pool figure quoted on 26 October does not reproduce

    The same query gave 27,726 rows on 3 November and gives 27,474 today, now that modelling papers are left out; the 45,095 quoted then was measured another way. Re-measure before quoting any pool number.

  2. 3,162 SI values lost beyond the quarantine

    The archive explains 2,403 of the drop between 26 October and 3 November; the rest needs a diff against the 26 October backup.

  3. 12 papers carry a BES class but are flagged not-BES, with no reviewed verdict

    They are out of the pool until a verdict is recorded. One, an 82-row bio-photovoltaic paper, looks mis-flagged.

  4. Six replays not recorded on production

    Processing state, display names, three BES verdicts, the off-corpus hold, the modelling-paper hold and the COMSOL Conference repair. The pending rows above.

  5. Nine of eleven Kolbe PDFs not yet read

    The parameter set is a proposal for Max Pohl to verify first; reading the rest costs about one US dollar.

  6. Five Kolbe quantities have no catalog definition

    Acid concentration, chain-length composition, charge passed, product amount and residual acid land with their raw names and no link.

  7. The paper classifier is not a pipeline stage

    A paper extracted through the wrapper has no class until someone runs the taxonomy sync. The landing gate covers the first insert; the corpus replay still needs the class.

  8. The pool misses more than half of what papers report

    Recall 40.6% on the gold set at 100% precision. The extractor reads conservatively by design; the cost is coverage.

  9. The gold set has not been checked by a person

    290 truth rows were scored by machine. One person-day would turn the extraction card into a benchmark.

  10. 2,924 papers have no fit or hold-out role

    A corpus-wide split and a refit on both environments is a maintainer decision.

  11. The paper page does not show the trail

    A row’s raw name, SI conversion, extractor, run and verifier reasoning are stored but not shown on /research/paper/…; the exclusion reason is a tooltip. Showing them is the next page change.

Where to check

Everything on this page can be re-run from the repository.

  1. Funnel: scripts/collab/measure-pipeline-funnel.sh rewrites the snapshot from staging, read-only.
  2. Eval cards: pnpm eval:extraction, pnpm eval:anchors, and the priors validation under apps/web/public/data/computed/research/; scripts/collab/build-data-evals.ts copies them here.
  3. Modelling papers: scripts/collab/measure-modelling-papers.sh rewrites that snapshot from staging, read-only; the record of the COMSOL Conference repair is docs/handoffs/2026-10-07-comsol-conference-papers.md.
  4. The pool predicate: libs/data-access/database/src/modelable-pool.ts and its Python twin, kept equal by a test.
  5. Promotion runbooks: docs/handoffs/2026-10-11-collab-corpus-prod-promotion.md, docs/handoffs/2026-10-30-microbe-kinetics-prod-promotion.md.
  6. Kolbe parameter set: docs/extraction/2026-11-03-kolbe-extraction-spec.md, the sheet Max Pohl verifies.

Staging measured 2026-11-16T14:15Z UTC; eval cards copied 2026-11-03T15:43Z UTC. Numbers drift as the corpus grows; re-measure before quoting any of them.