Skip to content
All work

Case study

Multi-Tenant Contract Extraction

Secure multi-tenant API extracting ~50 fields from hand-filled, negotiated purchase agreements using a vision LLM.

Role
Sole author — prompt, API, security and database
Domain
Data & Document Intelligence · Generative AI & LLMs · Backend & APIs · Cloud & Production
Stack
Python · FastAPI · Anthropic Claude · Azure AI Foundry · SQLAlchemy
Result
~88% field extraction accuracy
Still: a hand-filled agreement with crossed-out values, where the final accepted value of each field is extracted into a structured schema.

Overview

An API that reads hand-filled, renegotiated real-estate purchase agreements and returns the final agreed value of around fifty fields as strict JSON — built multi-tenant and security-first, for a law firm processing them daily.

The task is not reading the form. It is knowing which of four crossed-out prices is the real one.

What makes these documents hard

On the pageValues crossed out and rewritten across rounds of counter-offers.
How extraction handles itThe prompt teaches the model to ignore struck-through text and take the last value standing.
On the pageSignature stamps printed over the text they sign.
How extraction handles itThe model is instructed to read through overlays rather than stop at them.
On the pageAround fifty fields — names, money, dates, times, choices.
How extraction handles itA strict schema with a normalisation rule per field, then a separate formatter that builds the nested response.
On the pageA model that occasionally returns broken JSON.
How extraction handles itOne automatic retry with a stricter instruction, then a clean failure.

Around the model

The API accepts documents from strangers' URLs, so most of the engineering is not about the model at all.

Zero-trust fetching
HTTPS only; private, loopback and cloud-metadata addresses blocked; the host re-checked after every redirect; a hard byte cap.
Content checks
The declared file type must agree with the file's actual bytes, or the request is refused.
Tenant isolation
Every job is tenant-scoped, and API keys are compared in constant time.
Cost and failure
Tokens, latency and cost recorded per job — and a database outage degrades the audit, not the request.

My Role

Role

Sole author — prompt, API, security and database

Contribution

  • Engineered the system prompt that isolates the final non-crossed-out value across up to eight rounds of counter-offers, reading through signature stamps
  • Kept extraction and API contract separate via a pure post-processor, so the schema can evolve without touching extraction
  • Hardened the service: zero-trust URL fetching with SSRF blocking and redirect re-validation, magic-byte content sniffing, constant-time API-key auth
  • Built an async job and polling pattern with a provider abstraction and fallback
  • Added a Postgres audit trail of per-job tokens, latency and cost that degrades gracefully during database outage

Team Context

Sole author.

Decisions

A vision model instead of OCR plus rules.

Strikethroughs and stamps defeat positional rules — the right value is not where the form says it should be. A vision model can read intent from the page; rules cannot.

  • Instead of
  • OCR and positional rules
  • A trained layout model

Keep the raw model output; format separately.

The API contract stays stable while the prompt changes, and every response can be traced back to exactly what the model said.

  • Instead of
  • Let the model produce the final response

Results

~88%

field extraction accuracy

~50

fields per document

~15s

per document

Limits

The accuracy figure is my own aggregate across evaluated documents; the labelled set and scoring script are not in the repository. The model's self-reported confidence is not calibrated against correctness, so it is shown but not relied on. Extracted results are held in memory, so they do not survive a restart — an accepted trade-off at proof-of-concept stage, since only the audit metrics persist.

What I'd Improve

  • Commit a labelled set and scoring script so accuracy is reproducible, field by field. First fix.
  • Calibrate confidence against real correctness, so low-confidence fields can be flagged for review with meaning.
  • Persist results, and give reviewers a field-level view of what was extracted from where.
Back to all work