Retrieve, Then Verify: How Ironclad’s AI Matches Playbook Rules to Long Contracts
By Arina Kazakova, Senior Staff Engineer, AI, Ironclad
Learn how Ironclad’s Contract Review Agent finds where each rule in a company’s negotiating playbook applies in a long contract, catching more of them than an AI model reading the whole contract, at 3-4× lower cost.
Want more content like this? Sign up for our monthly newsletter.
At Ironclad, we build the agentic contract management platform that legal, sales, and procurement teams use to negotiate and manage their agreements. Its Contract Review Agent (CRA, formerly Jurist) reviews a contract against the company’s playbook (its written rules for which terms it will accept, such as a ‘cap on liability’) and drafts the edits. Before it drafts anything, it has to work out which paragraphs of the contract each playbook rule applies to. When CRA misses one of those paragraphs, the company’s position never gets applied there, and a reviewer has to catch the gap by hand. This post covers how we made that step reliable on long contracts1, such as master service agreements (MSAs).
Playbook rules often apply in multiple places. In a pilot on long MSAs, nearly half (47%) of the rules that matched at least one paragraph matched two or more. A matcher that returns only one of those paragraphs leaves the rest for a reviewer to catch. Yet both direct methods for finding those paragraphs missed matches: given a rule and the full contract, a leading LLM identified only 54% of the matches in our test set, while a search limited to the five paragraphs closest to each rule missed 37% of the relevant paragraphs, because they ranked sixth or lower.
To find the relevant paragraphs both methods missed, we built a two-stage matcher we call retrieve-then-verify. A retriever we fine-tuned on contract language searches from rules to paragraphs and from paragraphs back to rules, and an LLM verifies each candidate it finds.
The task: matching playbook rules to contract paragraphs
The objective of CRA’s multi-agent system is to redline a contract so that every position in the company’s playbook is evaluated against the relevant terms in the contract. A naive approach would be to give a model the whole contract with one playbook rule at a time and ask which paragraphs the rule applies to.
The mapping runs both ways. A confidentiality rule can apply to the confidentiality clause and also to a survival clause (which says which obligations continue after the contract ends) near the end of the contract. One termination paragraph can trigger several different rules.
Rule matching runs once per rule, often a dozen or more times per contract, so its errors add up across the contract. A missed match means a rule never gets applied and the redline has a gap. A match on the wrong paragraph puts an edit in the wrong place.
Why reading the full contract misses matches
The naive approach fits easily within what current models can read: a prompt with one rule and the full contract averages about 15,000 tokens in our traffic, and the longest reach about 47,000.
Models don’t read long inputs evenly. Liu et al. (2023) found that models use information at the start and end of a long input more reliably than information in the middle. The drop is steepest when the relevant passage doesn’t share wording with the question: on the NoLiMa benchmark (Modarressi et al., 2025), 11 of 13 long-context models fell below half their short-context score at 32,000 tokens. Contracts hit that case often, since a survival clause can govern confidentiality without using the word. We saw the same pattern: full-contract matching found only 54% of the matches in our test set. Repeat runs of the same rule against the same contract could also return different paragraphs, most often on MSAs, our longest contract type.
Retrieval addresses both problems by cutting how much text a model reads for each decision.
Retrieve, then verify
Retrieve-then-verify splits the job into two stages.
First, a retrieval model scores rules against paragraphs and selects the top candidate pairs. Second, an LLM verifier reads each pair, one rule and one paragraph, and decides whether the rule applies. Accepted matches go on to redline generation.
Each stage has a narrower job than one full-contract call. Retrieval is deterministic, so the same contract produces the same candidates on every run. The verifier reads one paragraph per call, so nearly everything in its input is evidence for the decision it has to make.
A retriever trained on contract language
The retriever is an embedding model. It turns each rule and each paragraph into a vector, and the candidates are the pairs whose vectors are closest.
We started by comparing four off-the-shelf embedding models on our test set: OpenAI’s text-embedding-3-large, Qwen3-Embedding-4B, BGE-large and ModernBERT-large. Qwen3-Embedding-4B and text-embedding-3-large finished within a point of each other at the top. We picked Qwen3-Embedding-4B because its weights are open, which means we can fine-tune it and run it inside our own infrastructure.
We fine-tuned it with LoRA (low-rank adaptation), which trains a small set of added weights and leaves the original model unchanged. Positive examples come from hand-curated gold-standard redlines: pairs where a rule was applied to a specific paragraph. Negative examples include paragraphs from unrelated contracts and paragraphs from the same contract that look similar to the rule but don’t match it. The second kind teaches the model to separate clauses that share vocabulary. The test pairs were held out from training.
Fine-tuning raised the share of relevant paragraphs in the retriever’s top five results from 74.5% to 84.0%. The largest gain was on MSAs, where the retriever’s score improved by about 26% over the off-the-shelf model.
Searching in both directions
Take an invented playbook rule: confidentiality obligations must last at least three years after the agreement ends. The contract has a confidentiality clause in section 7 and, near the end, a survival clause in section 18: “Sections 7 and 9 survive termination of this Agreement for two years.”
Section 18 is the paragraph that matters, because it sets the two-year term the rule cares about. It mentions confidentiality only by section number, so when the retriever ranks paragraphs for the rule, section 18 falls outside the top five results. A search that runs only from rules to paragraphs never sends it to the verifier.
Backward search runs the other way. It starts from each paragraph and asks which rules it matches best. Section 18 and the rule both deal with how long obligations last after the agreement ends, and section 18 has little in common with rules about payment terms or governing law. So the confidentiality rule is among its closest matches and the pair goes on the shortlist. The verifier sees that two years falls short of three and accepts the match, and CRA can propose extending the survival period.
Backward search checks only the top three rules per paragraph, so it fails when other rules – such as general termination, notice, or survival rules – rank higher than the confidentiality rule, or when the clause and rule share no wording in either direction. Even with these limits, bi-directional search caught 88.1% of relevant paragraphs in our pilot, finding more matches than doubling the number of paragraphs retrieved per rule, at half the added cost:
| Search | Relevant paragraphs found | Added verifier calls |
|---|---|---|
| Top 5 paragraphs per rule | 62.7% | Baseline |
| Top 10 paragraphs per rule | 79.9% | +100% |
| Top 5 per rule, plus top 3 rules per paragraph | 88.1% | +51% |
A verifier that judges one match at a time
The verifier is gpt-oss-120b, an open-weight model from OpenAI. For each candidate it gets the rule and the paragraph and returns accept or reject.
Before choosing gpt-oss-120b, we compared it with frontier models as the verifier. Gemini 3.1 Pro scored an F1 of 0.788 against 0.776 for gpt-oss-120b (F1 combines how many correct matches are found with how many returned matches are correct), at about ten times the cost per call. Narrowing each decision to one rule and one paragraph is what lets an open-weight model reach near-frontier accuracy at a fraction of the cost.
When no paragraph is accepted for a rule, for example because the contract lacks the clause it calls for, CRA routes the rule to a full-contract check so it can flag a missing clause. Our cost figures include that path.
How we measured it
The main test set is contract and playbook pairs across MSAs, NDAs and mutual NDAs, held out from training. The labels started from hand-curated redlines and were reviewed and corrected before we used them for scoring. Test-set scores use forward retrieval only, so they don’t include the gain from bi-directional search. The bi-directional search experiments ran on a separate pilot of long MSAs. Matching numbers compare against Gemini 2.5 Pro, the model CRA used for full-contract matching when we started this work. We repeated the comparison with Gemini 3.1 Pro reading the full contract, and the two-stage matcher still came out ahead.
Attorneys on Ironclad’s in-house legal team also reviewed rule decisions directly and marked which rules applied. That check doesn’t depend on our reference labels.
For finished redlines, we ran CRA twice on each contract in a comparison set, once with full-contract matching and once with retrieve-then-verify, and changed nothing else. Three models from different labs (GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro) scored each redline, and the majority vote counted. Mixing model families follows Verga et al. (2024), who found that a panel of judges from different model families shows less self-preference bias than a single judge. Each judge scored redlines without knowing which matcher produced them.
No customer data was used to train the retriever.
Results
| What we measured | Retrieve-then-verify vs. full-contract matching | How it was measured |
| Applicable rules caught | 31% more | Attorneys on Ironclad’s in-house legal team reviewed matching decisions |
| Matching F1 (combines matches found and matches correct) | 11% higher | Held-out test set |
| Redline quality | Better on choosing the right playbook position, factual correctness, edit precision and coverage of playbook priorities; no measure got worse | Blind LLM-judge panel (majority vote) |
The matching gains carry through to the final redlines. The extra matches also hold up: in a second set verified by our attorneys, 93% of retrieve-then-verify’s matches were correct, against 95% for full-contract matching. Matching is faster and costs 3–4× less than full-contract matching.
What carries over
Many tasks that apply a long list of criteria to one long document have the same shape. If one model reads the whole document for every criterion, cost grows with document length times the number of criteria, and coverage depends on how evenly the model attends to a long input. Retrieving the relevant text first turns each decision into a short, focused one, and a model fine-tuned on your own documents can do that retrieval cheaply and consistently every time.
For legal teams, this changes what a first-pass AI review can catch. Retrieve-then-verify catches more of the playbook rules that apply to a contract, so fewer of a company’s positions go unapplied and there is less for a reviewer to catch by hand. The retriever returns the same candidates on every run, so a second pass on the same contract checks the same paragraphs. At 3-4× lower cost per contract, a team with a high-volume queue can run its full playbook on every agreement, including the long MSAs where the recall gains were largest.
1 In this post, a long contract is one over 30,000 tokens, roughly 20,000 words of English text.
Ironclad is not a law firm, and this post does not constitute or contain legal advice. To evaluate the accuracy, sufficiency, or reliability of the ideas and guidance reflected here, or the applicability of these materials to your business, you should consult with a licensed attorney.
