An AI copilot that reads regulation so analysts do not have to

The challenge

The product's value was a corpus of 40,000 regulatory documents, but the only way in was keyword search. Analysts routinely spent half a day answering questions the corpus already contained. An earlier LLM prototype had been shelved because it invented citations, which in a compliance product is worse than useless.

Our approach

We treated citation accuracy as the product requirement rather than a nice-to-have, and built the evaluation suite before the feature. Every answer is grounded in retrieved passages and refuses rather than guesses when retrieval is weak. We ran the eval suite on every pull request, so a regression in grounding failed CI the same way a broken test would.

What we delivered

• Retrieval pipeline over 40,000 documents with hybrid semantic and keyword search
• Answer generation with mandatory inline citations linked to source passages
• Evaluation suite of 400 graded questions, run in CI on every change
• Refusal behaviour and confidence signalling when retrieval quality is low
• Analyst feedback loop that feeds directly into the evaluation set
• Per-query cost tracking and monthly spend dashboards