Most retrieval systems are built on an assumption nobody writes down: that you are allowed to send the document somewhere. Remove that and a surprising amount of the standard architecture goes with it.
A compliance team's job is to read contracts and policies and find the clauses that create exposure. It is slow, it is expensive, and it is the kind of pattern-matching work that language models are genuinely good at. So the pitch writes itself, and every compliance product on the market makes it.
The problem is the second sentence of every procurement conversation. The documents a compliance team works on are the confidential asset — a master services agreement under negotiation, a data processing addendum, an incident report. A legal team cannot paste those into a hosted model endpoint, and if you tell them the provider has a zero-retention policy, they will ask you to put that in writing and then still say no. The retention policy is a promise. Their obligation is a contract.
We built Sovereign for the case where that answer has to be architectural rather than contractual: the documents never leave the customer's network, because there is nowhere else for them to go. It audits contracts against GDPR, HIPAA and SOC 2 and returns severity-ranked findings, each one citing the clause it came from, with inference running on the customer's own AMD Instinct GPUs.
These are the three decisions that shaped it, and the one we would revisit.
Self-hosting changes which problems you have
Calling a hosted model API is the easy path for a reason. Someone else owns capacity planning, batching, failover and the upgrade treadmill. Give that up and you inherit all of it — but you also stop having the conversation that was killing the deal.
Running on AMD Instinct rather than NVIDIA adds a second, less obvious cost. Almost every tutorial, Dockerfile and Stack Overflow answer in the LLM ecosystem assumes CUDA. The mainstream serving stacks support ROCm, but support is not the same as being the well-trodden path: you are off the default, and the failure modes you hit are the ones with three search results instead of three hundred.
Budget for that. The engineering cost of self-hosting is not the inference code — it is the hours spent on the parts of the toolchain that assume a vendor you are not using.
What you get back is a guarantee you can hand to a security reviewer without a caveat. For a product whose entire value proposition is confidentiality, that trade was not close.
Retrieval over the regulation, not just the contract
The obvious design is to embed the contract, retrieve the relevant clauses for a question, and ask the model whether they are compliant. That works in a demo and fails in review, because it asks the model to supply the regulation from memory.
“Is this clause GDPR-compliant?” is not one question. It is: which specific obligations does this clause engage, and does it satisfy each of them? A model answering from parametric memory will be roughly right about Article 28 and confidently wrong about the sub-clause that actually matters — and confidently wrong is the worst possible failure mode for a compliance tool, because it is indistinguishable from confidently right until someone checks.
So the regulation is a retrieval corpus too:
- Chunk and index the frameworks — GDPR, HIPAA, SOC 2 — at the granularity a lawyer argues at, which is the sub-article, not the article.
- Segment the contract at clause boundaries rather than by token count. A clause split across two chunks is a clause that gets assessed twice, badly, from two halves.
- For each clause, retrieve the obligations it plausibly engages, and assess the pair.
The unit of work is a clause–obligation pair, not a document. That is more retrieval calls and more inference, which on self-hosted capacity is a real cost rather than a line on someone else's invoice. It is also the difference between a tool that produces plausible prose and one that produces findings a reviewer can check.
Citations have to be structural, not requested
Every RAG system says it cites its sources. Most of them ask the model to, in the prompt, in bold, sometimes in capitals. That is a suggestion, and it holds until the run where it does not — usually the run where the retrieved context was thin and the model filled the gap.
A compliance finding with a hallucinated citation is worse than no finding at all. It sends a lawyer to a clause that does not say what the report claims, and the second time that happens they stop trusting the tool entirely. So we made it a property of the pipeline instead:
finding = {
clause_id: "<id from the parsed contract>",
obligation_id: "<id from the indexed framework>",
severity: "high" | "medium" | "low",
rationale: "<model output>",
}
# Both ids must resolve against what was actually retrieved
# for this pair. A finding that fails to resolve is dropped,
# not rendered with a warning.The model does not write the citation. It selects from the identifiers it was given, and anything it invents fails to resolve and is discarded before rendering. Structured output makes this cheap to enforce; the important part is not the JSON schema but deciding that an unresolvable finding is a bug rather than a caveat.
The useful reframe: do not ask the model to be trustworthy. Build the system so that untrustworthy output cannot reach the user.
Severity ranking is a product decision disguised as a model one
A thorough audit of a long agreement produces a lot of findings. Ship them unordered and you have built something that technically works and nobody uses, because the reviewer's actual job — decide what to escalate before the call at four — has not been done for them.
Ranking is not free. It requires an opinion about what makes exposure severe: the obligation engaged, whether the clause is silent or actively contradictory, whether remediation is a redline or a renegotiation. That opinion belongs to the compliance team, not the model, and getting it out of their heads is a design problem rather than an engineering one. It was the part of the build that most needed a human in the room, and the part we would schedule earlier next time.
What we would do differently
We built the evaluation set after the pipeline rather than before it. Not catastrophically late — there was a graded set before anything went near production — but late enough that some early architectural decisions were made on impressions rather than numbers. Retrieval quality in particular is very hard to judge by looking at outputs: a system retrieving mediocre context produces answers that read fine, and you cannot see the obligation it failed to surface.
If we started again, the first week would produce a graded set of clause–obligation pairs with known-correct answers, and every subsequent change would be measured against it. That is the advice we now give every client building on retrieval, and it is worth more than any individual architectural choice above.
Sovereign is live and public — you are welcome to open it and judge the engineering yourself. If you are building something with a similar shape and want a second opinion on the architecture, that conversation is free and we are happy to have it whether or not it goes anywhere.