Fig. 3
Docwise
AI extraction of supplier invoices, with every value audited.
A 90-second walkthrough with narration and captions. Sound plays only when you press play.
The problem
Small finance teams type supplier invoices into the books by hand. AI extraction is faster, but a wrong total that looks certain is worse than no help. Docwise reads the invoice, checks every value the way an auditor would, and approves only what it can vouch for. Everything else waits for a person, with the reason shown.
For ops and finance teams that process supplier invoices, and for developers who want the same thing as an API.
The product
Hard problems
Confidence that means something
Models are poor judges of their own certainty, so Docwise doesn’t ask for a score. A value is high confidence only if no rule implicates it, the model didn’t list it as uncertain, and, for a PDF with a text layer, the value is actually printed in the document. Numbers and dates match by value, so 1.234,56 € matches 1234.56. Anything else goes to a person, with the reason beside the value.
The evaluation is part of the product
Sixty invoices are generated from a seed, each with its true values: five layouts with their local number and date formats, scans, phone photos, two-page invoices, handwritten notes, and three that don’t add up on purpose. A third were held out and scored once, after tuning ended. The eval runs the app’s own functions, and CI re-scores every saved model answer on every push, so a change that moves a reported number fails the build.
A model that never writes to the database
Claude answers through structured outputs, and Zod parses every answer. A malformed one goes back once with the exact errors; a second failure is reported, never stored. A reviewer’s corrections are kept beside the model’s values, so the log can always tell what the model said from what a person fixed.
Evaluation
| Header fields correct | 100% |
|---|---|
| Line-item F1 | 100% |
| Invoices fully correct | 20 of 20 |
| Sent to a person | 1 of 20, the invoice that doesn’t add up |
| Escaped errors | 0 |
| Model cost per invoice, median / p95 | $0.0115 / $0.0170 |
| Time per invoice, median / p95 | 3.1 s / 5.6 s |
One prompt change, before and after
The prompt tells the model how to write dates, amounts and currency codes. Without those rules, on the 40 tuning invoices, Claude still got every value right, but copied dates and amounts as printed. Every first answer failed validation and needed the retry.
| Measure | Without the rules3 | With the rules4 |
|---|---|---|
| Model calls per invoice | 2 | 1 |
| Median cost | $0.0251 | $0.0111 |
| Median time | 6.3 s | 3.1 s |
| Sent to a person | 12.5% | 5% |
Twenty invoices is a small sample: no escaped errors means none occurred here, not that the rate is zero. The invoices are generated, so real ones, with odd layouts, stamps and poor scans, will be harder.
Every run, with each answer the model gave, is in docs/measurements.
How it’s built
- Next.js 16 (App Router, Server Actions)
- TypeScript
- Claude Sonnet 5.5 with structured outputs, PDF and image input
- Zod
- Postgres on Neon, Drizzle ORM
- Cloudflare R2, private with signed URLs
- Vitest with PGlite, Playwright
- Vercel, GitHub Actions
- Browser → R2Sends the invoice straight to a private bucket through a signed URL.
- AppChecks the stored file’s type, size and pages, and queues it. An API client sends the file with a Bearer key instead.
- App → ClaudeIn the background, sends the PDF or image to Claude Sonnet 5.5, which answers in a strict JSON schema.
- AppZod parses the answer. A malformed one goes back once with the exact errors.
- AppRules check the dates and the arithmetic. Each field’s confidence comes from the rules, the model’s own doubt and the PDF’s text.
- App → PostgresStores only validated values. Every field high: approved. Anything low: waits for review.
- Reviewer → AppCorrects values beside the original. Corrections are stored next to the model’s values, never over them.
- App → CSV, APIApproved invoices go out as CSV, one row per line item, or as JSON.
- npm run evalRuns the same functions on the test set. CI re-scores the saved answers on every push.
More decisions and their trade-offs are in the README.
Replies within a working day.
Sources
- Docwise held-out split: 20 generated invoices, scored once after tuning ended, with prompt v1 on Claude Sonnet 5.5. Header fields 100% correct, line-item F1 100%, 20 of 20 fully correct, review rate 5% (1 of 20), 0 escaped errors. Model cost per invoice at list prices: median $0.0115, p95 $0.0170. Time per invoice: median 3.1 s, p95 5.6 s. Source: docwise-extract/docs/measurements.
- Docwise tests: 155 Vitest tests (unit, database tests on PGlite, and a replay of every saved eval run) and 8 Playwright tests on a production build, green in CI. Source: docwise-extract PROGRESS.
- Docwise tuning split, 40 invoices, the bare prompt (v1 without its rules for values) on Sonnet 5.5: 40 of 40 fully correct after the retry, 2 model calls per invoice, median cost $0.0251, median time 6.3 s, review rate 12.5%. Calls per invoice are in the JSON record. Source: eval-03-bare-sonnet-5-5-tuning.json.
- Docwise tuning split, 40 invoices, prompt v1 on Sonnet 5.5: 40 of 40 fully correct, 1 model call per invoice, median cost $0.0111, median time 3.1 s, review rate 5%. Calls per invoice are in the JSON record. Source: eval-02-v1-sonnet-5-5-tuning.json.