WithDave · accuracy receipts
Every AI company says their product is accurate. We publish the measurements instead — including the ones that caught us being wrong. Real runs, real numbers, preserved in version control.
Receipt №001
August 1, 2026
The setup. We built Shore & Crane LLP — a fictional six-person Boston law firm — and onboarded it through our real product: 67 documents (engagement letters, demand letters, wills, leases, memos, spreadsheets) uploaded through the actual ingestion path, chunked (~1,800 chars, 200 overlap), embedded (voyage-3, 1024-dim), and indexed in pgvector. Then we wrote the exam before trusting anything: 24 client questions, each with the matter whose documents the answer must cite.
The failure. First run: 9 of 24 passed. The other 15 didn't cite the wrong documents — they cited nothing. The AI answered generically with zero sources, and no error was raised anywhere. That's the worst kind of failure: silent.
The diagnosis. Instead of guessing, we re-ran the failing queries with the similarity threshold at zero and looked at what retrieval actually ranked. The ranker was right every single time — the top hits were the correct matter's documents. They scored 0.31–0.44, and the pipeline's hard floor was 0.45. Conversational client phrasing (“my mother's estate”) simply embeds farther from formal legal prose (“Estate of Miriam Alvarez, Docket SU26P0847EA”) than whoever picked 0.45 assumed. The measurements:
“What's the status of my mother's estate?”
should cite: Estate of Miriam Alvarez (26-EST-001)
| cosine sim | top-ranked chunk | right matter? | ≥ 0.45 floor? |
|---|---|---|---|
| 0.427 | 26-EST-001 draft inventory | ✓ | discarded |
| 0.419 | 26-EST-001 letter to co-heir | ✓ | discarded |
| 0.388 | 26-EST-001 engagement letter | ✓ | discarded |
| 0.366 | 26-EST-001 letter to co-heir | ✓ | discarded |
| 0.360 | 26-FAM-001 disclosure memo | — | discarded |
“They offered $12,000. What happens if we sue instead?”
should cite: Ironsides v. Hardwick (26-BUS-004)
| cosine sim | top-ranked chunk | right matter? | ≥ 0.45 floor? |
|---|---|---|---|
| 0.418 | 26-PI-001 status memo | — | discarded |
| 0.416 | 26-BUS-004 opposing counsel's response | ✓ | discarded |
| 0.368 | 26-BUS-004 strategy memo | ✓ | discarded |
| 0.359 | 26-PI-001 demand letter | — | discarded |
| 0.352 | 26-BUS-004 strategy memo | ✓ | discarded |
“Kevin says he'll pay half. Should I take it before the hearing?”
should cite: Tran contempt matter (26-FAM-003)
| cosine sim | top-ranked chunk | right matter? | ≥ 0.45 floor? |
|---|---|---|---|
| 0.442 | 26-FAM-003 hearing-prep memo | ✓ | discarded |
| 0.425 | 26-FAM-003 engagement letter | ✓ | discarded |
| 0.344 | 26-LT-001 status memo | — | discarded |
| 0.329 | 26-FAM-003 engagement letter | ✓ | discarded |
| 0.314 | 26-BUS-006 lien strategy memo | — | discarded |
“Do we become the tenant's landlord automatically at closing?”
should cite: Lindqvist purchase (26-RE-001)
| cosine sim | top-ranked chunk | right matter? | ≥ 0.45 floor? |
|---|---|---|---|
| 0.422 | 26-RE-001 engagement letter | ✓ | discarded |
| 0.403 | 26-RE-001 P&S rider | ✓ | discarded |
| 0.392 | 26-RE-001 closing checklist | ✓ | discarded |
| 0.373 | 26-RE-001 P&S rider | ✓ | discarded |
| 0.372 | 26-RE-001 engagement letter | ✓ | discarded |
The fix. One constant, recalibrated from the measurements: the similarity floor moved from 0.45 to 0.30, with the calibration data documented in a comment at the constant so nobody “tidies” it back. The citation-discipline prompt (cite only sources that inform the answer) handles the weak tail below the old floor.
The verification. Same 24-question battery, same production path, later that afternoon: 24/24 grounded with citations to the right matter's documents — including the flagship “What's the status of my mother's estate?”, which answered with the estate's exact inventory value and cited the draft inventory, the engagement letter, and the attorney's letter to the co-heir.
What we took from it. Three things. One: retrieval thresholds fail silently — the product keeps answering, just ungrounded, and nothing pages you. Two: diagnose with measurements, not vibes; the instinct “the retriever must be bad” was wrong — the ranker was perfect and the gate was miscalibrated. Three: every new corpus gets a question battery before it gets trusted. Ours now runs as a script against the real ask path, and its JSON reports live in the repo.
Receipt №002
August 2, 2026
The method. After the 24/24 run, we took six of the answers — one per practice area — and graded them the slow way: every factual claim in every answer, traced by hand to the cited document in the firm's corpus. Because the corpus is fictional and we wrote its ground truth, there is no ambiguity about what's correct. (Grading was done by the system's builders — an independent re-grade is the obvious next step, and we say so rather than pretend otherwise.)
| client question | claims traced | figure errors | notes |
|---|---|---|---|
| “What's the status of my mother's estate?” | 14 of 15 | 0 | one procedural inference not in the file — see below |
| “The landlord wants a 24-month personal guaranty. How bad is that?” | all | 0 | caught the planted trapdoor clause — see below |
| “They offered $12,000. What happens if we sue instead?” | all | 0 | disambiguated the matter before answering; treble math exact |
| “What do I owe tenants for the missed deposit interest?” | all | 0 | six tenancies, deposits, and rate all match the tracker |
| “They keep saying I was a manager so no overtime applies…” | all | 0 | stated the exemption test; flagged the lookback period for attorney confirmation instead of guessing |
| “My stepmother says a 2022 paper makes her the trustee…” | all | 0 | execution-defect table matches the file; hedged exactly where the file hedges |
The find we're proudest of. The lease in the corpus deliberately contains a defect a careless reader misses: the personal guaranty advertises a 24-month cap, but a buried subsection makes the cap evaporate on any default — precisely when it would matter. The AI's draft found it, cited it, and explained it to the client like this:
“A cap that evaporates the one time you are in trouble is not a cap. It is a trapdoor.”
The miss we're publishing anyway. In the estate answer, one sentence — that the co-heir “receives a copy the same week” the inventory is filed — is a reasonable procedural inference but is not stated anywhere in the firm's file. Fourteen of fifteen claims traced; that one didn't. This is exactly the class of sentence the attorney review queue exists to catch, and in the live product every one of these drafts waited for a partner before any client saw it.
The behavior worth noticing. Where the corpus ran out, the drafts didn't improvise — they wrote inline flags like [REVIEWER: confirm current lookback period] for the human to resolve. An AI that knows what it doesn't know, and says so in the draft, is the property everything else here is built on.
More receipts as we measure them. No cherry-picking: misses get published with the fixes.
See the product these receipts are about