Verified intelligence · Expert-reviewed from public records, platform data, primary sources, and private research beyond AI search.
MODELSLegal servicesGLOBAL
AI models for legal work
Legal AI models should be compared as a work stack: frontier model, legal retrieval, OCR, embeddings, reranking, deployment controls, source permissions and lawyer review.
The colophon
- Landscape reviewed
- 2026-06-19
- Latest revision
- 2026-06-19
- Kinetics
- Kinetics: activeImpact 0.88 · momentum 0.87 · vector 0.778 balls · 58 px/s
Legal AI has moved from clever research chat to work systems that touch diligence, drafting, spreadsheets, matter knowledge, and client delivery. Model releases, retrieval benchmarks, and legal AI product features now change fast enough that a one-model policy ages badly.
- To watch
- 23 June 2026: European Commission high-risk AI consultation closes; scope could change which legal AI uses need AI Act controls 143.
- 2 August 2026: AI Act Commission enforcement powers for GPAI providers begin; EU model routes may need stronger documentation, copyright-policy or incident-reporting controls 142.
- Thomson Reuters v ROSS docket entries or appeal briefing; legal-RAG licensing economics may change 133.
- OpenAI, Anthropic or Google model-deprecation notices; approved legal routes or cost per accepted legal answer may change.
- Known unknowns
- Independent end-to-end legal-work benchmarks with multilingual and non-US matters, stale-law checks, bad OCR, spreadsheets, privilege review, realistic matter permissions, regulatory-change events and cost per accepted output.
Contents
- Executive summary
- What changed
- Model economics
- Throughput and rate limits
- Benchmark evidence
- Use-case route selection
- Retrieval, embeddings and reranking
- OCR and document models
- Deployment choices
- Risk controls
- Cross-border limits
- Regulatory gates
- Model shortlist
- Legal-political risk
- Operational signals
- AI agents for legal workflows
- Implementation cases
- Expert watchlist
- Known unknowns
- Vendor directory
- Source ledger
- Glossary
Executive summary
Legal model selection is stack approval: which model sees which sources, under which permissions, with which document layer, and for which legal task. GPT, Claude, Gemini, open-weight and Chinese models all have roles; none deserves the keys to the whole matter. A good model with the wrong source is still a bad legal answer. Use the strongest frontier models where reasoning, judgement and drafting quality matter. Use cheaper or private models where volume, sovereignty or latency matters. Use legal retrieval, reranking and OCR before asking any model to pronounce on a clause, case or filing.
The trap is to treat a model leaderboard as a buying decision. Legal work fails less often because a model is weak than because the system sent it the wrong source, a bad scan, a stale authority or a document it was not allowed to read; Stanford HAI found pervasive legal hallucination even in retrieval-augmented legal products 29,30.
For high-risk work, the generator is only the visible part of the system. Parser quality, source rights, jurisdiction metadata, currentness, retrieval recall, reranker precision, citation support and human sign-off decide whether the answer is defensible.
Model choice is a lane decision rather than a coronation. A research memo, invoice review, Chinese-language diligence, privilege triage and scanned lease schedule need different routes.
Risk decides the model lane
Start with the legal job, then route by risk, source need, document condition and jurisdiction.
Will this become advice, filing, privilege, citation or client text?
YesUse the strongest approved route, source checks and lawyer sign-off.
NoKeep screening the route.
Next: Q2 sourcesDoes it need authorities, clauses, matter files or local law?
YesUse legal retrieval, reranking and currentness checks first.
NoCheaper classifiers or summaries may be enough if evals pass.
Are files scanned, tabular, redlined, multilingual or schedule-heavy?
YesTest OCR and parsing before comparing models.
NoChoose by task difficulty, cost and data boundary.
Is it cross-border, sovereign, public-sector or country-of-concern work?
YesClear language, law, hosting, transfer and local-review gates.
NoPilot the best-priced route that passes private evals.
The tree turns model selection into a route decision, not a brand preference.
The route decision
| Decision | Buyer move | View |
|---|---|---|
| Premium reasoning | Route | Use the premium frontier lane where legal reasoning, drafting quality and review minutes matter more than raw unit price. Keep exact model versions in the dated shortlist. |
| Legal retrieval | Prioritize | Prioritize legal embeddings, reranking, source hierarchy, currentness, and citation verification before swapping answer models. |
| Open weights | Evaluate locally | Use Mistral, Llama, DeepSeek, Qwen, and similar lanes for private, sovereign, or high-volume work after local evals. |
| Document understanding | Test first | Treat OCR, layout, tables, redlines, exhibits, and page references as a model layer. Bad parsing makes good models look clever and wrong. |
| Final legal judgement | Human gate | Do not let any model own filings, privilege calls, client advice, or citation-bearing work without lawyer verification and an audit trail. |
What changed
The bottleneck moved from raw fluency to defensible work product. Legal AI has moved from clever answers to accountable work. The model race still matters, but it no longer explains most of the difference between a useful legal AI system and a dangerous one. Frontier models can draft, summarize and reason well enough to be productive. The harder question is whether the system knows the matter, finds the authority, preserves the evidence and returns work in a form a lawyer can check.
That shift changes procurement. Test the model's legal reasoning and the machinery around it: source access, retrieval, OCR, citations, permissions, evals and review workflow.
The market is splitting into layers. Model vendors push capability. Legal-source owners push authority. Workflow platforms push adoption. Procurement has to identify which layer is being bought and which layer still needs to be built.
The next advantage is not asking a larger model a longer question. It is reducing the number of ways the model can be confidently wrong.
What changes now
| Shift | Meaning | Action |
|---|---|---|
| Frontier models compressed | GPT, Claude, and Gemini are all strong enough for serious legal work in different ways. | Run task-level bakeoffs; do not standardize on one model for political neatness. |
| Workflow value expanded | The model is now only one part of drafting, diligence, research, Excel, DMS, VDR, and client-delivery systems. | Evaluate the surrounding system: source access, permissions, output formats, review UX, and audit trails. |
| Retrieval became visible | Legal embedding, reranking, chunking, and source hierarchy now have dedicated benchmarks and vendors. | Score retrieval separately from generation. A fluent answer from the wrong source is still wrong. |
| Benchmarks became less naive | LegalBench, BigLaw-style tasks, PRBench, LegalBenchmarks.ai, MLEB, and agent benchmarks now test different pieces of the stack. | Use benchmarks as a map of failure modes, not as a shopping basket. |
| Lawyers started building small agents | The practical frontier is not a giant legal oracle; it is clause miners, risk registers, authority checkers, and redline copilots. | Pilot narrow, source-linked workflows with explicit approval gates. |
Model economics
Price the finished legal task, not the token. The cheap model is cheap only when it finishes the legal job. Per-token price is a sticker. Legal cost is the whole route: prompt length, output length, retries, hidden reasoning tokens, cache hits, tool calls, queue time, reviewer minutes and the approved data boundary. A model that costs twice as much per token can still win if it writes the answer in half the tokens, needs fewer repair prompts and takes less lawyer time to verify.
Token mix changes the bill. A tool that reads a large bundle and returns a short answer is mostly an input-token problem. Translation, redlining, drafting and clause-by-clause issue lists can force the model to write almost as much text as it read, so high output-token prices can dominate the matter budget. 120,147
Artificial Analysis-style comparisons are useful because they put price beside intelligence, output speed, latency and context. Legal buyers need one more column: accepted work product. The unit is not a million tokens. It is a correct answer with source support, a reviewable redline, a complete issue register or a verified extraction.
The same rule applies to agent routes. Harvey's initial LAB snapshot put the then-highest all-pass route at about $50.90 per task and roughly 22 minutes; the later Opus 4.8 note improves the score to 10.4%, but it does not remove the need to price passed legal work rather than only the token stream. 195,197
Open and Chinese models can be excellent routes, but they do not guarantee savings. Self-hosting only becomes cheap when utilisation is high, serving is efficient, security and patching are paid for, and fallback capacity is planned. A cheap model running through a reseller, a low-utilisation GPU fleet or a review-heavy workflow may not be cheap at all.
The most useful procurement metric is cost per accepted output: model spend plus infrastructure, latency, retries, review time, failed-task rework and compliance overhead. Many legal AI pilots discover that cheaper tokens were only cheaper accounting.
Method: Artificial Analysis price rows grouped by a16z bands; compares cheapest blended 3:1 token price with estimated full-index run cost. Caveat: not a legal-quality score 119,149.
Sticker price is only the first cost signal. Lane choice determines whether the same workload is interactive, batched, cached, flexed, provisioned or delayed, which changes both the bill and the review burden.
The best discount for legal VDR review, litigation production, clause extraction and private eval reruns is often patience. Long extraction packets, eval reruns, classification sweeps and archive enrichment do not need a user waiting in the browser. Move that work to async or lower-priority lanes, keep deadline work on the fast path, and compare cost per accepted legal output.
The discount lane is part of the route
| Lane | How it works | Good for | Watch |
|---|---|---|---|
| OpenAI Batch API | Async jobs use a separate batch lane with a lower price and a 24-hour completion target. 172 | Large document extraction, eval reruns, classification, enrichment and non-urgent archive work. | Not an interactive path; queue time, file packaging, partial failures and output reconciliation become part of the route. |
| OpenAI Flex processing | Eligible calls use lower-priority capacity and are charged at Batch API rates for supported models. 182 | Long-document async work where cost matters more than immediate availability. | Treat it as a cheaper lane with lower priority, not as the path for a signing call, filing deadline or live customer answer. |
| Anthropic Message Batches | Bulk async requests can be priced below the standard synchronous lane and can be combined with prompt caching. 184,185 | Large non-interactive packs: extraction, summarization, evals, source tagging and overnight review queues. | Batch queue behaviour and cache checkpoint design decide whether the savings survive production. |
| Gemini / Vertex AI batch | Vertex batch prediction is the non-real-time lane; Google pricing separates standard, batch and cached-input treatment. 186,188 | Bulk document analysis, classification and back-office model calls that do not need immediate response. | Batch mode, provisioned throughput and cached context are separate mechanisms; do not assume one discount buys the others. |
| Amazon Bedrock batch inference | Bedrock batch inference runs async jobs over S3 inputs, with discounted batch pricing for supported models. 189,191 | Cloud-controlled bulk inference where the input/output artefacts can live in the approved AWS path. | Prompt caching and batch inference are not universally combinable; model, region and feature support decide the real lane. |
| Groq Batch and Flex | Groq's Batch API is a discounted async lane; Flex processing trades immediacy for access to available capacity rather than a lower price. 193,194 | Throughput-sensitive work where the buyer can separate background processing from live assistant use. | Low published developer limits can still make a clever route unusable until the account, plan or enterprise path changes. |
| Usage tiers and quota requests | Spend tiers, usage tiers, quota requests and account-specific limits can change the ceiling before any model changes. 171,173,174,175,177,179 | New projects that need to absorb a sudden file drop, launch backlog or review queue. | Tier language is not capacity evidence. Check the live project, account, subscription, region and deployment type. |
| Provisioned or reserved capacity | Azure PTUs and reservations, Vertex Provisioned Throughput and Bedrock Provisioned Throughput buy predictability rather than a smarter model. 176,178,181 | Recurring volume, deadline-sensitive work and routes where shared-capacity 429s would be business failures. | Commitments, idle capacity, regional availability, spillover rules and reservation terms belong in the price model. |
Prompt caching is easy to misunderstand. It is not model memory and it does not cache the answer. It reuses the provider's work from the repeated beginning of a prompt: instructions, tool schemas, output schema, playbook text, matter taxonomy or source bundle. If the next request starts the same way, the provider can bill and process that repeated prefix differently. If the route prepends a timestamp, request id or one-off user sentence before the stable material, the cache may miss.
Design the prompt like a production artefact. Put the durable, reusable context first; put the unique question last; record cache hits in the legal model route register. Cache savings are real only when the route keeps a stable prefix and still produces accepted legal output.
How to make caching work
| Design rule | Why it matters | Implementation |
|---|---|---|
| Put stable material first | Most cache mechanisms reward an identical or explicitly marked prefix. | Start with system instructions, tool schemas, output schema, jurisdiction rules, playbook text and reusable source packs. |
| Put variable material last | A unique timestamp, request id, user aside or matter-specific fact at the top can break a prefix match. | Append the specific question, run id, new facts, reviewer note and requested output after the cacheable prefix. |
| Keep tools and schemas stable | Tool definitions, JSON schemas and message ordering can be part of the cache key. | Version schemas deliberately; avoid rebuilding tool lists, whitespace, examples or column order on every call. |
| Warm the cache before fan-out | Parallel extraction can waste money if every worker pays to prefill the same long context. | Send one cacheable request for the source pack or playbook, then fan out clause, entity, citation or issue tasks. |
| Cache source packs, not shortcuts | The cache should preserve approved context, not hide missing retrieval or stale policy. | Attach source version, route version and expiry; rebuild the prefix when authority, playbook or source set changes. |
| Measure cached-token economics | A cache discount that never hits is procurement fiction. | Log cache reads, writes, cached tokens, hit rate, latency and accepted-output rate beside the normal token bill. |
| Respect the data boundary | Caching is still provider processing under the approved route. | Cache only context that is allowed inside the approved data boundary. Caching is a billing and latency control, not permission to reuse one client's facts in another client's work. |
Provider caching is not one thing
| Provider | Mechanic | Buyer read |
|---|---|---|
| OpenAI | Automatic prompt caching applies to repeated prompt prefixes above the documented threshold; usage surfaces cached tokens, and prompt-cache keys can help route related work. 183 | Keep the first part of the prompt identical: instructions, tools, schema and reusable source context before matter-specific text. |
| Anthropic | Claude uses explicit cache-control breakpoints over tools, system and message content, with separate cache write and read economics and documented cache durations. 185 | Design cache checkpoints deliberately. The best target is the long shared context, not the short one-off question. |
| Gemini / Vertex AI | Vertex supports context caching, including explicit cached content and implicit prefix caching; pricing separates cached input from ordinary input and batch treatment. 186,187 | Check whether the route uses explicit cache objects, implicit prefix reuse or batch prediction; each has different cost and expiry behaviour. |
| Amazon Bedrock | Bedrock prompt caching uses cache points with time-to-live controls and separate cache pricing, while batch inference has its own feature constraints. 189,190,191 | Do not assume a cached interactive prompt can simply move into batch. Confirm feature support by model and region. |
| Groq | Groq prompt caching is automatic for exact repeated prefixes above the documented threshold, with discounted cached tokens and separate treatment in rate-limit accounting. 192 | The route must keep a stable prefix. This matters most when rate limits are low and large token packets would otherwise queue. |
The practical question is whether the buyer can deliberately choose the cheap lane without breaking the work. A cheaper async run is useful for overnight extraction. It is dangerous for an answer needed during a signing call. A cache hit is useful when the prefix is stable. It is imaginary when every prompt is rebuilt from scratch.
What to prove before calling the route cheap
| Test | Proof | Pass condition |
|---|---|---|
| Interactive versus async split | Tag each workflow step as user-blocking, same-day, overnight or archive work. | Only non-blocking work moves to batch, flex or lower-priority processing. |
| Batch economics | Run the same packet through interactive and batch lanes, including failed files, retries, queue time and output handling. | The cheaper lane still produces accepted legal output inside the work window. |
| Cache hit rate | Log cache reads, writes, cached tokens or provider-equivalent fields by route version. | The actual hit rate is high enough to change the finished-task bill, not just the theoretical token price. |
| Tier and capacity path | Capture account tier, quota, regional capacity, provisioned-capacity option and support escalation. | The buyer knows which spend, quota or reserved-capacity lever unlocks the next workload level. |
| Accepted-output denominator | Combine model spend, queue time, cache hit rate, reviewer minutes, failed runs and rework. | The route is cheaper per accepted legal output, not merely cheaper per million tokens. |
Cost per accepted output
| Cost item | Token view | Accepted-output view | Measurement |
|---|---|---|---|
| Visible token price | Input and output prices. | Accepted legal output. | Cost per accepted memo, issue, redline, extraction or citation-checked answer. |
| Input-output mix | One blended token rate. | Scenario-specific bill. | Separate read-only Q&A, translation, redlining, drafting and issue-list scenarios. |
| Reasoning tokens | Simple input/output price. | Total billed cost. | Total billed cost and pass rate on the same private legal tasks. |
| Token efficiency | Price per 1M tokens. | Passed task cost. | Total tokens and repair prompts per passed task. |
| Speed and latency | Cheap slow output. | Usable lawyer workflow. | Time to first useful draft, output tokens per second and throughput under expected concurrency. |
| Self-hosting utilisation | Low marginal inference. | Fully loaded route cost. | Peak, trough and idle load plus serving stack, security, patching, fallback and support. |
| Provider route | Base model price. | Approved route economics. | Compare direct API, cloud marketplace, router, reseller, platform and self-hosted lanes under the same controls. |
| Review burden | Model spend. | Lawyer verification and rework. | Reviewer minutes, rejected outputs, corrected citations and escalation rate. |
Throughput and rate limits
Rate limits decide whether the legal model can survive a matter drop. Rate limits are production capacity, not a nuisance error. A model can be the best reasoning route and still fail a legal VDR, litigation production, lease pack, regulatory archive or privilege-review batch if the account cannot move enough tokens through it. The limiting number may be requests per minute, tokens per minute, input tokens per minute, output tokens per minute, batch-queue tokens, spend tier, project quota, regional capacity or provisioned-unit size.
This matters most when work arrives in a lump. A one-off demo does not prove that a route can process the approved workload by the production deadline. Low limits turn intelligence into waiting time, retries, partial runs and fallback risk.
Record throughput in the legal model route register. The approval question is not "is the model smart?" It is "can this approved account, region, deployment type, batch lane, provisioned capacity and fallback process the actual workload before the business deadline?"
The first proof is tier math: the same paid API can clear a large packet quickly at the top tier and stall at the entry tier. Treat the tier row as a production input, not procurement small print.
The second proof is contract capacity. Azure PTUs and reservations, Vertex Provisioned Throughput, Bedrock Provisioned Throughput, cloud quota requests, global or cross-region routing, Batch APIs, prompt caching and fallback routes all change both cost and operational risk.
Think of the approved file packet like a large print job waiting at a shared printer. The model may be ready, but the paid API tier decides how wide the paper feed is. The 750K-token packet is the scenario load for this stress test. At OpenAI's published 500K TPM entry tier, that packet waits about 1.5 minutes just to enter the lane; at the published 40M TPM top tier, it clears in about a second 171. The bars show that intake queue, not model intelligence or total job time.
Values checked on 27 May 2026. Bars show minimum intake wait for one 750K-token packet under public tier limits. Caveat: excludes output, retries, queueing, review and private capacity.
Lowest and top tiers are different products
| Provider | Named public route | Published RPM and token cap | Approval read |
|---|---|---|---|
| OpenAI | GPT-5.5 API | Lowest paid: Tier 1, 500 RPM / 500K TPM. Highest public: Tier 5, 15K RPM / 40M TPM. 171 | Use the live usage tier, long-context cap and shared-family cap. Do not use response-header samples as route evidence. |
| Anthropic | Claude Opus 4.x API | Lowest paid: Tier 1, 50 RPM / 500K ITPM / 80K OTPM. Highest public: Tier 4, 4K RPM / 10M ITPM / 800K OTPM. 173 | Input and output caps are separate; Opus 4.x traffic shares one family limit and custom limits require approval. |
| Groq | Groq API production route | No single public production TPM ceiling. Public docs expose plan/model rate-limit rows and say higher limits are available for select workloads and enterprise use cases. 174 | Do not approve from public base rows. Capture the live account or enterprise contract limit before using Groq as a production lane. |
| Azure OpenAI | gpt-5.5 GlobalStandard | Lowest nonzero paid: Tier 5, 10K RPM / 10M TPM. Highest public: Tier 6, 15K RPM / 15M TPM. Lower public tiers list 0 / 0. 175 | Quota is per model, deployment type, subscription and region; quota is not proof of deployable capacity. |
| Google Vertex AI | Gemini 2.5 Pro Standard PayGo | Lowest paid: Tier 1, 500K TPM. Highest public: Tier 3, 2M TPM. System cap: 30K RPM per model per region. 177 | The baseline is org-level shared capacity on the global endpoint; traffic smoothing and Provisioned Throughput still matter. |
| Amazon Bedrock | On-demand and Provisioned Throughput | No single comparable public default TPM row. AWS exposes model-specific token/RPM quota families and provisioned MUs. 179 | Check Service Quotas for the exact model and region; use cross-Region inference or Provisioned Throughput for the production lane. |
Rows are official paid API tier or quota rows captured on 31 May 2026. The comparison rule is lowest public paid tier plus highest public paid tier where both are published; otherwise the row names the missing public production ceiling.
The wait calculation answers only one question: how hard the public token cap bites before any workaround. The buyer still has to ask which lever is actually available for the approved account, region and data class.
Capacity levers change the contract
| Route | Capacity levers to check | Procurement caveat |
|---|---|---|
| OpenAI APIs | Usage tiers, org/project/model limits, response headers, Batch API, priority processing. | Exact limits remain account-specific. |
| Claude API | Spend tiers, RPM/ITPM/OTPM, prompt caching, Message Batches, Priority/custom limits. | Acceleration limits can still punish sudden spikes. |
| Groq API | Plan upgrade, per-model limits, cached-token treatment, Batch/Flex or enterprise workload path. | Public base rows are not a production ceiling; the buyer needs live account or enterprise limit evidence. |
| Azure OpenAI | Quota tiers, model/deployment/region pools, multi-region deployment, PTUs, spillover, reservations. | PTU quota does not guarantee deployable regional capacity. |
| Gemini / Vertex AI | Standard PayGo tiers, global endpoint, traffic smoothing, custom tier, Provisioned Throughput, caching or batch lanes. | PayGo 429s can reflect shared-resource contention, not a fixed quota breach. |
| Amazon Bedrock | Service quotas, quota increase request, cross-Region inference, Provisioned Throughput, commitment terms. | Model, region and account quotas decide the real production lane. |
Documented capacity mechanisms are not a quality score. The useful question is which lever the buyer can actually buy, reserve, request, batch or route around for the approved workload.
Once the tier and capacity levers are visible, approval should move from provider comparison to failure design. Name the first limiter, the owner of the workaround and the proof that the fallback can scale.
How rate limits break production work
| Bottleneck | Failure mode | Buyer move |
|---|---|---|
| Synchronous TPM, ITPM or OTPM | Large files queue behind the token budget even when the model can reason over them. | Measure tokens per accepted job; compare input, output and cached-token treatment separately. |
| RPM and burst shape | A filing drop, diligence batch or claims surge can hit request caps before token caps. | Run burst tests with realistic concurrency; smooth traffic and batch small tasks when latency allows. |
| Spend tier or usage tier | A new account or project may be stuck below the limits needed for a real matter. | Check the live account tier before procurement approval; pre-warm tiers or contract higher limits. |
| Batch queue | Offline work can move out of the synchronous lane, but turnaround and queue limits decide whether it fits the deadline. | Separate interactive lawyer work from overnight extraction, eval and classification runs. |
| Cloud quota and regional capacity | Quota by model, deployment type, region, project or subscription can block scale even after the model is approved. | Request quota early; verify region capacity and data-residency constraints before the launch window. |
| Provisioned or reserved capacity | Reserved units solve predictability but introduce commitments, idle capacity and capacity-allocation risk. | Size PTUs, GSUs, MUs or token reservations from real prompt and output distributions. |
| Fallback route | The backup may have different rate limits, data terms, quality, region, logging or source access. | Approve fallback by data class and workflow, not just by provider name. |
Before production, make the route pass concrete tests. A limit that is acceptable in a demo is not approved until the live account, region, batch lane and fallback path have been exercised together.
What to prove before the route scales
| Test | Proof | Pass condition |
|---|---|---|
| Large-file packet | Run a 750K-token packet plus expected output under target concurrency. | The route finishes inside the work window without 429 cascades, missing files or unreviewable partial output. |
| Deadline cluster | Replay the busiest hour of the production deadline window. | Primary and fallback routes stay inside latency and data-boundary limits. |
| Tier and quota evidence | Capture account tier, project limits, model limits, cloud quota, regional capacity and support escalation path. | Procurement can name the exact limiter that would fail first. |
| Reserved-capacity sizing | Convert request shape into provisioned units using average input, output, cache hit rate and RPM. | The buyer knows commitment cost, idle-capacity risk, spillover behaviour and scale-up constraints. |
| Fallback legality | Test whether fallback changes region, subprocessor, retention, source licence, audit log or model quality. | Fallback is allowed for that data class and reviewer instruction. |
Benchmark evidence
Use benchmarks to find questions, not to outsource judgement. A benchmark is a map. Procurement is still the journey. LegalBench and Vals cover broad legal reasoning 17,18. LawBench and LexEval cover Chinese legal tasks 58,59. KBL and JBE-QA cover Korean and Japanese legal knowledge 61,62.
Harvey LAB and PRBench get closer to long-horizon agent and professional-reasoning work 48,49. LegalBenchmarks.ai tests drafting reliability 151 and extraction reliability 151. MLEB tests legal retrieval embeddings 50.
The useful LegalBench signal is not the logo on the leaderboard. It is that task-level rows expose granular legal calls and can show when a contained task is near first-pass range; the July 2025 LegalEvalHub rows here are historical examples, not current frontier rankings 167.
The useful LegalBenchmarks.ai signal is different: no raw model clears 56% reliability on contract drafting 151. Usefulness splits from reliability because GPT-5.5 leads usefulness while sitting lower on drafting reliability 151. App wrappers can move hard-task results by double-digit points 151. That turns a leaderboard into a private-eval design brief, not a procurement answer.
The useful Harvey LAB signal points the other way: strict all-pass scoring on long-horizon legal-agent work still leaves the leading Opus 4.8 route at only 10.4% 197. That is not failure theatre. It is a better denominator for deciding what private evals must test.
GDP.pdf adds an input-layer warning: Surge's leaderboard puts the top current macro row at 30% on professional PDF QA 198. The benchmark spans 100 real-world prompts and PDFs across legal and other expert domains 199. Reducto's vendor-reported rerun says structured parse context can change macro acceptance 201. Parser evaluation belongs inside the model route, not in a pre-processing footnote.
The buyer question is not which model won a table. It is what failed: the wrong jurisdiction, the wrong passage, a missing schedule, a thin judging method. Those misses tell a firm what to test on its own matters.
Do not merge unlike scores into one leaderboard. Chart comparable scores inside one benchmark. Use tables for scope, caveats and blind spots.
Public benchmarks are a starting pack for private evals. Production proof has to use the firm's own matters, playbooks, document quality, source licences and review standards.
Ask vendors for the tasks behind the score, the rubric, the human baseline, judge agreement and examples of failures. If they cannot provide them, treat the score as a lead, not proof of fit.
Published numbers are shown one source at a time. Where the evidence is thin, prose or table caveats beat an invented score.
Chart renders public LegalEvalHub task rows checked in 2026, with runs dated 24 or 25 July 2025; it is task-shape evidence, not a current frontier ranking 167.
The next question is what those task rows do not see. A high score on a contained task does not prove retrieval, source rights, drafting workflow, agent action, multilingual competence or lawyer-review fit. Treat the July 2025 chart as task-shape evidence; use the gap map to decide what the private eval still has to test.
Scores still miss the work around the answer
Use each strong cell as the benchmark's job description and every weak or blank cell as private-eval backlog. A top legal-reasoning score does not cover retrieval, workflow, agent action or local-law proof.
The winner changes with the test
Vals LegalBench
Public LegalBench score; higher is better. Snapshot: 26 May 2026.
LegalBenchmarks.ai drafting
Reliability over 34 drafting tasks: percent where every checklist requirement passed. Snapshot: May 2026.
LegalBenchmarks.ai extraction
Reliability over 31 information-extraction tasks: percent where every requirement passed. Snapshot: May 2026.
GDP.pdf
Macro accuracy: share of professional PDF tasks where every rubric criterion passed.
Harvey BigLaw Bench support page
Vendor-reported BigLaw Bench scores from Harvey's model-comparison support page. Snapshot: 21 May 2026.
Harvey LAB initial holdout before Opus 4.8
All-pass percentage on Harvey's initial LAB holdout; higher is better.
PRBench legal hard subset
Hard legal subset score without web search; 1.0 is the maximum.
MLEB
Task-average NDCG@10 across ten expert-annotated legal retrieval datasets. Snapshot: 21 Oct 2025.
LawBench
Zero-shot overall average across memorization, understanding and application tasks.
LexEval
Zero-shot average across six legal cognitive ability levels.
KBL
Closed-book overall score across Korean legal knowledge, reasoning and bar-exam tasks.
JBE-QA
Zero-shot F1 on Japanese bar-exam question-answering.
LEXam
Open-question judge score on English and German law-school exam questions.
LexGLUE
Main-paper arithmetic mean macro-F1 across LexGLUE tasks.
COLIEE 2026
Task 3 statute-law QA accuracy; team runs, not model routes.
Each square keeps one benchmark's own scoring axis. Similar names across squares are a pattern; different winners are the point.
What each benchmark can see
| Benchmark | Signal | Limit |
|---|---|---|
| LegalBench / Vals LegalBench | Broad legal reasoning across 110 model rows and current provider comparison 18. | Not a full workflow, retrieval, or client-output benchmark. |
| LegalBench-RAG / Legal RAG Bench | Retrieval quality and legal RAG failure decomposition 19,20. | Narrower than production matter work and source governance. |
| BigLaw-style and Harvey LAB tasks | More realistic legal work, strict all-pass agent scoring, practice-area variance, behaviour traces and Opus 4.8's 10.4% LAB update 195,197. | Vendor-authored results still need public task audit, human baselines, judge agreement and local workflow evals. |
| PRBench legal | Hard professional reasoning with expert rubrics and low top-model scores 49. | Not an end-to-end production benchmark. |
| LegalBenchmarks.ai | Drafting reliability splits from usefulness 151,151. App wrappers can change hard-task outcomes 151. Information extraction has a separate reliability curve 151. | Still one task set; private evals must test local matters, sources, review burden and adoption. |
| GDP.pdf | Professional PDF question-answering across 100 real-world documents 199. Top current macro row is 30% 198. Reducto's vendor-reported rerun says structured parse context lifted aggregate macro accuracy by 9 points 201. | Cross-domain and partly vendor-adjacent; private legal evals still need local matters, file condition, source rights and reviewer acceptance. |
| MLEB / COLIEE | Legal retrieval and competition-system retrieval or entailment signals 50,51. | Search and competition signals still miss the full legal work system. |
| LexGLUE / LEXam | Legal classification and law-exam reasoning stress tests 52,53. | Useful task slices, not production matter approval. |
| LawBench / LexEval | Chinese legal tasks in original legal contexts 58,59. | Less comparable than English legal benchmark leaderboards; still not workflow proof. |
| KBL / JBE-QA / Japanese bar-exam writing | KBL covers Korean legal knowledge 62. JBE-QA covers Japanese legal QA 61. Japanese bar-exam writing adds open-ended legal reasoning 60. | Less mature and less comparable than English legal benchmark leaderboards. |
| MMLU-ProX / Global MMLU / KMMLU | KMMLU gives Korean language-risk signal 64. MMLU-ProX and Global MMLU add multilingual and cultural evaluation signal 65,66. | Useful language-risk screen; not a legal-work acceptance test. |
Scores need translation before buying
| Term | Plain meaning | Does not measure |
|---|---|---|
| Eval | A repeatable test set with prompts, expected answers or rubrics, scoring rules and failure review. | Production value unless it uses the buyer's own matters, sources, reviewers and risk thresholds. |
| LegalBench / Vals LegalBench | Legal-reasoning benchmark and public leaderboard used to compare answer models on legal tasks 17,18. | Retrieval, OCR, current source access, jurisdiction routing, privilege handling or workflow adoption. |
| LegalBench-RAG / Legal RAG Bench | Retrieval-augmented legal tests that ask whether the system finds and uses legal evidence 19,20. | The full quality of advice, drafting style, client posture or matter permissions. |
| MLEB | Massive Legal Embedding Benchmark; an embedding retrieval test using NDCG@10 to score whether relevant legal material appears near the top 50. | Whether the final answer is correct, cited, current or acceptable to a lawyer. |
| PRBench | Professional-reasoning benchmark with hard open-ended tasks and expert rubrics, including legal samples 49. | End-to-end production workflow, source licences, parser quality or client review. |
| GDP.pdf | Professional document question-answering benchmark over real PDFs and expert rubrics 198,199. | Legal-only workflow approval, matter permissions, privilege, source licences, reviewer time or production controls. |
| LawBench / LexEval | Chinese legal benchmarks that test PRC-law tasks and legal capability in Chinese legal contexts 58,59. | Hong Kong, Singapore, Taiwan or general Chinese-language common-law competence. |
| KBL / SuperLawyer | Korean legal-language and bar-exam signals 62,63. | A reproducible production score for Korean legal research, drafting or litigation workflows. |
| JBE-QA / Japanese bar-exam writing | Japanese legal question-answering and open-ended legal-reasoning signals 61,60. | Commercial deployment readiness, local source rights or lawyer-review cost. |
| OCR, parser and reranker | OCR and parsing preserve document structure 16. Rerankers reorder candidate sources before generation 23,25. | They do not replace legal judgement; they decide whether the model sees usable evidence. |
Use public benchmarks to build the private eval. Do not use them as the private eval.
Do not buy the leaderboard
| Use | Good | Bad |
|---|---|---|
| Shortlisting | Use several benchmarks by task family. | Buy the highest single row. |
| Vendor diligence | Ask for tasks, rubrics, human baselines, judge agreement, and failures. | Accept headline percentages. |
| Internal eval design | Borrow benchmark shapes and add firm-specific documents. | Assume public tasks resemble your matters. |
| Trajectory review | Score source coverage, validation, revision, tool fan-out and stop conditions. | Score only the final answer. |
| Regression monitoring | Re-run after model, parser, retrieval, and prompt changes. | Trust release notes. |
Use-case route selection
The right stack changes with the legal risk. Do not use the same model to write the memo, tag the privilege call and clean the invoice. Legal AI becomes useful when the work is routed by risk. The cheaper model belongs at the start of the funnel: intake, classification, rough summaries and repeatable extraction. The stronger model belongs where facts, law, money and reputation meet.
The workflow also decides the supporting models. Research needs authority retrieval and citation checking. Diligence needs OCR, clause extraction and defined-term handling. Litigation needs page-line evidence and privilege controls. Legal ops needs spreadsheet discipline.
Map the route to the lifecycle stage. Intake and onboarding tolerate cheaper models and lighter review. Analysis, drafting and contract review need stronger models, retrieval, and structured review. Action steps such as filing, signing, sending and billing belong behind explicit human approval regardless of model strength.
Routing is not only a cost tactic. It is a safety control. It prevents low-risk automation from quietly becoming high-risk legal judgement.
Choose the oversight mode from the action: human-in-the-loop where lawyer judgement is required, human-on-the-loop where the work is reversible and a sample is enough.
Route work by legal risk, evidence requirements and review burden.
Workflow approval map
| Workflow | Route | Oversight | Approval test |
|---|---|---|---|
| Case-law research | Premium reasoning model plus legal database retrieval, reranker and citation verifier. | Human-in-the-loop. | Jurisdiction, treatment, pin cites and quote support are verified. |
| Contract review and redline | Drafting model plus parser, clause library, client playbook and precedent retrieval. | Human-in-the-loop. | Defined terms, fallbacks, numbering and business owner positions survive review. |
| M&A diligence | OCR, entity extraction, legal retrieval and risk-register generation. | Human-in-the-loop on findings; human-on-the-loop on extraction. | Each issue has evidence, coverage status and missing-document flags. |
| Litigation record review | Private or VPC route with OCR, retrieval, privilege classifier and page-line evidence. | Human-in-the-loop on privilege; sampled review otherwise. | Privilege, Bates ranges, contradictions and adversarial-document controls are logged. |
| Regulatory monitoring | Source watchlist, jurisdiction classifier and change-impact summarizer. | Human-on-the-loop with escalation rules. | Effective date, binding status and affected product or process are confirmed. |
| Legal ops and Excel | Office model plus structured outputs and spreadsheet validation. | Human-in-the-loop before save or send. | Formula diff, assumptions and approval are captured before save or send. |
| Client-facing triage | Small model, approved-content RAG and escalation rules. | Human-on-the-loop with sampled review. | The bot stays inside approved content and hands off when advice starts. |
OCR and document models
Parse the legal file before running the model bakeoff. A legal model comparison is invalid if scans, tables, redlines, signatures, schedules, exhibits or page-line references were lost before prompting. Approve the parser route first, then score the generator on evidence it actually received.
GDP.pdf gives that rule a public denominator: hard professional PDFs can break frontier answer models before legal reasoning begins 198,199. Reducto's rerun is vendor-reported, but it points to the right acceptance test: compare final reviewed answers before and after structured parser context 201.
OCR, extraction, signature handling, redline recovery, VDR ingestion, table reconstruction, pricing and parser evals need their own acceptance tests before the answer model is scored.
The accepted document output should preserve file class, page anchors, table structure, redline state, signature evidence, missing-document flags and reviewer-ready source links.
Deployment choices
The channel often matters as much as the model. Approve the deployment route before comparing model features. The same base model creates a different legal risk when it runs through a direct API, cloud platform, legal SaaS workflow, model gateway, productivity suite or controlled open-weight lane. Approve the route by use case and data class before comparing features.
Consumer or personal accounts are not a legal-work tier. Public-law summaries, privileged diligence, regulated-sector advice, spreadsheet edits and cross-border investigations need different approved deployment paths and reviewer evidence.
Deployment diligence should cover licence, hosting, gateway policy, retrieval control, GPU economics, contract structure, recovery planning, support, region and audit evidence.
Risk controls
Fluent and wrong is worse than blunt and uncertain; controls must be testable. The risk is not that models err. The risk is that they err beautifully. Legal AI output can be fluent, structured and wrong. Stanford HAI documented pervasive legal hallucinations in general-purpose LLMs and continued hallucination risk in dedicated legal RAG products 29,30. The system must expose evidence, uncertainty, source limits and model version so a lawyer can verify the work.
This is where legal differs from coding and maths. Much public model testing rewards domains where the answer can be run, linted, unit-tested or checked against a known solution. A legal hallucination can look plausible to a tired human and still be difficult for a computer to falsify. The control stack therefore needs private evals, verifier models or AI-judge passes, citation checks and human gates for high-consequence output 42,48.
The worst failures are usually compound failures: a bad scan, weak retrieval, missing metadata, stale authority and a confident model. Governance needs to break that chain before the final answer.
Grounding helps; explanation does not. Retrieval-augmented generation reduces hallucination by tying outputs to a known source set; chain-of-thought traces look like reasoning but are statistical natural language, not faithful records of model decision-making 144. Treat citations and confidence scores as the trustworthy signals; treat the model's own narration as a hypothesis to verify.
A disclaimer is not a control. Controls are testable: citation verification, quote support, jurisdiction filters, prompt-injection tests, model versioning, matter permissions and human approval for final legal judgement.
Bias compounds across training data, algorithmic design, and feedback loops. Mitigation needs cross-group and cross-jurisdiction testing, requested model reasoning, and vendor evidence of bias measurement, not a single line in a model card.
The audit trail answers a simple question after the fact: why did the system say this, and who accepted the risk?
Legal AI failure modes to control
| Risk | Approval stance | Control |
|---|---|---|
| Hallucinated authority or quotation | Hard gate | Citation, quote, pin-cite, and treatment verifier plus lawyer review. |
| Untestable legal correctness | Hard gate | Private legal evals, verifier or AI-judge passes, citation checks, and human gates for high-consequence output. |
| Wrong jurisdiction or stale law | Hard gate | Jurisdiction filters, source hierarchy, effective-date metadata, and local counsel. |
| OCR or table failure | Control before scale | Parser evals, page-level citations, confidence flags, and review queues. |
| Retrieval miss | Hard gate | Hybrid retrieval, reranking, recall evals, and no-answer behaviour. |
| Prompt injection or adversarial document | Hard gate | Untrusted-content isolation, tool scopes, and adversarial tests. |
| Model drift | Control before scale | Version pinning, regression evals, and change logs. |
| Consumer-tool data leakage | Hard gate | Approved commercial channels, retention rules, DPAs, and matter-level permissions. |
| Bias against client groups, case types or jurisdictions | Control before scale | Cross-group eval sets, requested model reasoning, and documented vendor bias-measurement evidence. |
| False confidence from unfaithful chain-of-thought | Control before scale | Treat reasoning traces as hypothesis, not proof; rely on citations, source support, and verifier passes. |
| Training or retrieval data poisoning | Control before scale | Source provenance, corpus diffing, supplier attestation, and periodic re-evaluation after material data refreshes. |
| Model extraction and context exfiltration | Hard gate | Output filters, secret redaction, memory scoping, rate limits, and red-team probing for prompt and document leakage. |
Cross-border limits
A fluent answer can still route through the wrong legal system. Cross-border approval has three gates: original language, local authority and data path. LawBench and LexEval show that Chinese legal evaluation needs Chinese legal tasks, not translated common-law questions 58,59. Japanese and Korean work now has bar-exam and legal-language benchmarks 60,62. Russian public evidence is thinner, which is itself a procurement warning 67,68.
General multilingual benchmarks add a second warning. MMLU-ProX and Global MMLU show that models can move unevenly across languages and that translated benchmarks carry cultural assumptions 65,66. A strong score in English is not a licence to advise in Japanese, Korean, Mandarin or Russian.
Safety guardrails degrade in regional languages too. Most public red-teaming and bias research is English-first and Western-centric, so a model that refuses an adversarial prompt in English may comply with the same prompt in Vietnamese, Arabic, Thai, or a regional dialect. Cross-border deployments need adversarial evaluation in the languages the model will actually receive in production.
A leaderboard would overstate the evidence here. LegalBench, LexEval, KBL, Japanese bar-exam tasks, KMMLU, MMLU-ProX and Global MMLU use different datasets, metrics and scopes, so the honest format is a source map plus route controls.
False familiarity is the failure to fear: a model borrows a familiar concept from another legal system because the words look close enough. For sensitive matters, region, retention and source access are legal questions before they are technical ones.
Cross-border work changes the control burden
The breakpoint is not the country label. It is when the work becomes a legal conclusion, client-data movement, reusable corpus or external filing; those turn language convenience into route approval.
Read multilingual evidence correctly
| Evidence | What it says | Move |
|---|---|---|
| LegalBench / Vals | English legal-reasoning signal, not global legal proof. | Shortlist models; then test local sources and court hierarchy. |
| LawBench / LexEval | PRC-law signal, not Chinese-language common-law validation. | Separate Mainland China, Hong Kong, Singapore and Taiwan evals. |
| Japanese bar-exam work / JBE-QA | Useful Japanese-law signal, still early for production workflows. | Test Japanese issue spotting, citation support and unsupported-law hallucination. |
| KBL / SuperLawyer | Korean legal-language and bar-exam signal, with reproducibility limits. | Use Korean statutes, cases, bar tasks and lawyer-reviewed answers in pilots. |
| MMLU-ProX / Global MMLU / KMMLU | Language and cultural-risk screen, not a legal acceptance test. | Use to size language risk before building jurisdiction-specific evals. |
| Russian public evidence | Thin comparable Russian-law evidence is itself a risk signal. | Require official-source retrieval and local counsel review. |
Wrong-law failure gates
| Jurisdiction | Failure to avoid | Approval gate |
|---|---|---|
| US and UK common law | Wrong court hierarchy, stale treatment or imported US/UK priors. | Named source database, treatment check and local review. |
| EU and member states | GDPR, AI Act, data transfer and national law collapsed into one answer. | Member-state evals, supplier documentation and human-oversight logs. |
| Mainland China | Chinese fluency mistaken for PRC-law and data-clearance competence. | PRC corpus, onshore review and data-export assessment. |
| Japan and Korea | Language fluency without local authority hierarchy or exam-style issue spotting. | Original-language bar/legal tasks and local lawyer review. |
| Russian-law work | Generic Russian-language fluency substituted for Russian-law source grounding. | Official-source retrieval, sanctions check and counsel review. |
| Multilingual disputes | Translation erases defined terms, negation, privilege or governing-law boundaries. | Original-language retrieval, bilingual citations and locked glossary. |
Regulatory gates
Some model routes require legal clearance before use. Regulation can block the highest-scoring route. The approval question is practical: which model routes need a legal, security or procurement gate before pilot work begins.
Route matters more than nationality. A consumer chatbot app, a cloud API, a self-hosted open-weight model, a PRC data centre and a private evaluation environment can carry different legal profiles even when the model name is similar.
For legal work, the blocked route may be invisible to a benchmark. A model can score well, then fail approval because the matter contains export-controlled technical data, US government-related data, bulk health or financial data, a sanctioned beneficiary, PRC-important data, a Russian nexus or a public-sector restriction.
Regulatory routing is a go/no-go control before pilot work starts: data class, model owner, hosting region, subprocessors, customer sector, end user, sanctions status, export-control status and local filing obligations.
Regulatory routing gates
| Decision | Trigger | Required action |
|---|---|---|
| Route approval | US government-related or bulk sensitive data; covered persons in countries of concern; PRC important-data export. | Classify data before pilot. Approve provider, region, subprocessor, retention and transfer path. |
| Stop unless cleared | Entity List counterparty, sanctioned owner, Russia beneficiary, restricted reseller or payment chain. | Screen provider, affiliates, end user and beneficial owner. Geofence and get counsel sign-off. |
| Approved route only | Defence, dual-use, controlled software, technical assistance or client technical data. | Run matter-level export classification. Bar unapproved cloud, model and foreign-person access. |
| Sector gate | Government, critical infrastructure, connected vehicles, healthcare, finance or employment use. | Apply the sector policy before model selection. Keep a named exception owner and audit trail. |
| EU compliance evidence | EU high-risk use, prohibited-practice risk, GPAI provider obligations or systemic-risk model. | Demand provider documentation. Set use-case classification, logs, human oversight and incident route. |
| China local clearance | Mainland public-facing GenAI, public-opinion/social-mobilisation features or outbound important/personal data. | Separate internal R&D from public service. Confirm CAC filing, security assessment and data-export path. |
How to read public restrictions
| Signal | Wrong read | Correct read | Buyer move |
|---|---|---|---|
| DeepSeek actions in Australia and Taiwan | DeepSeek is banned everywhere. | Public-sector and critical-infrastructure policy can override model fit. | Separate government, regulated-sector and private-client routes before relying on a score. |
| DeepSeek privacy action in Italy and Korea | The issue was benchmark quality. | Prompt logs, telemetry, consent and cross-border transfer can stop the route. | Ask where prompts, files, embeddings, logs and support traces go; then approve by data class. |
| US connected-vehicle rule | The US has a general Chinese AI software ban. | Vertical supply-chain rules can block foreign-linked software. | Map the client sector and connected-system exposure before using foreign-linked model software. |
| US AI Diffusion Rule rescission | All advanced model weights are currently banned under that rule. | Check the current EAR, chip and model-weight rule before relying on headlines. | Use current export-control text, not stale policy headlines, for model-weight and technical-data decisions. |
| US and EU Russia service sanctions | Sanctions only block payments. | Model access, support, fine-tuning and cloud services may be restricted by beneficiary. | Screen provider, affiliate, reseller, end user, support path and beneficial owner. |
| Mainland China public GenAI rules | Every internal China test needs a public-service filing. | Public-facing mainland services and data export need separate local clearance. | Separate internal R&D, client work, public-facing service, important-data export and local hosting. |
Model shortlist
Route constraints decide which models are eligible. A shortlist is a route map, not a ranking. Start with the legal job, source set, document condition, data class, jurisdiction, deployment channel, review gate and throughput target. A model enters only if the approved route can see the evidence and the reviewer can audit the output.
Keep the durable shortlist at lane level: premium synthesis, independent control, lower-cost volume, retrieval and reranking, parser/OCR, and sovereign or local-language. Exact versions and prices belong in dated tables.
Benchmarks supply failure probes. Model docs supply context, tool and data constraints. Economics shows when the premium lane is overkill. Risk gates can remove a route even when the score is strong.
For each route, write down the job it is allowed to do, the sources it may read, the data boundary it must stay inside, the failure it is meant to catch, and the private eval that proves it works. Without those five fields, the row is only a model preference.
Shortlist routes by legal need
| Need | Starting route | Proof | Bad shortcut |
|---|---|---|---|
| Citation-bearing research or legal memo | Two premium frontier answer routes plus licensed legal retrieval, reranking and citation verification. | Current controlling authority, treatment, quote support and pin cites survive lawyer review. | Pick the highest legal-reasoning score and let it search the open web. |
| Diligence over scans, schedules, exhibits or VDR packets | Parser and OCR bakeoff first; answer model second; source pages and missing-document flags visible. | Tables, signatures, amendments, redlines, counterparts and folder relationships remain attached to findings. | Assume long context can repair broken extraction. |
| High-volume intake, classification or clause extraction | Lower-cost model or extractor with schema validation, confidence thresholds and escalation queue. | False negatives, rejected rows, reviewer minutes and cost per accepted output beat the premium route. | Send every token through the most expensive model as a safety blanket. |
| Privileged, regulated or sovereign matter | Approved enterprise, VPC or self-hosted route with retention, region, logging, support and fallback controls. | The route clears client policy, data-transfer, sanctions, export-control, retention and audit requirements. | Treat a strong model score as permission to process the matter. |
| Cross-border or local-law work | Original-language retrieval, jurisdiction metadata, local-law tests, local counsel review and controlled translation. | The route separates language fluency from governing law and authority hierarchy. | Use English benchmark strength or generic multilingual fluency as legal proof. |
| Spreadsheet, legal-ops or agentic workflow | Model or agent route with tool scopes, structured outputs, formula diffs and approval before save or send. | The system logs what changed, who approved it and how to roll back or correct it. | Evaluate only answer quality when the workflow can mutate records. |
Scenario-style shortlist examples
| Scenario | Shortlist shape | Deciding eval | Likely rejection |
|---|---|---|---|
| US litigation team drafting a research memo with citations | Two frontier answer routes, one legal database route, legal reranker, citation verifier and no-answer policy. | Twenty recent questions with controlling authority, bad-law traps, quote checks and partner review. | Any route that cannot show source chain, treatment or proposition support. |
| M&A team reviewing a messy VDR | OCR/parser lane, extraction lane, premium synthesis lane and cheaper triage lane for low-risk documents. | Hard sample with scans, schedules, amendments, duplicate agreements and missing-document flags. | A route that summarizes cleanly while losing packet structure or page evidence. |
| In-house team processing routine NDAs at volume | Contract app or extractor, low-cost model route, schema validator and premium exception route. | False-negative rate on fallback clauses, escalation accuracy and cost per accepted issue. | Premium-only route that improves prose but not throughput or review burden. |
| Cross-border investigation with PRC and Japanese documents | Original-language retrieval, approved hosting route, local-law eval set, controlled translation and local counsel gate. | Jurisdiction-specific questions using original sources, local authority hierarchy and data-transfer constraints. | Any route that collapses language, jurisdiction and data clearance into one score. |
| Legal-ops workflow that updates spreadsheets or matter records | Structured-output model, tool-scoped agent, spreadsheet diff, approval gate and rollback log. | Formula preservation, record mutation audit, exception handling and reviewer approval time. | A chat-only route that cannot prove what it changed. |
Legal-political risk
Court rulings, elections, policy posture, lobbying, and public sentiment can change the acceptable model route. A legal model route can fail outside the benchmark. Model comparisons usually rank quality, price, speed, context, and data terms. Legal work adds another screen: whether the route survives current litigation, copyright posture, agency rulemaking, lobbying pressure, party-policy shifts, public-sector procurement, and election-cycle risk.
The Thomson Reuters v ROSS ruling 133 shows why the distinction matters. Legal-source rights and fair-use assumptions can sit underneath retrieval, training, fine-tuning, and vendor indemnity. A model can be good at legal reasoning and still be the wrong route for a matter if source provenance, data class, geography, or public-sector policy does not clear.
Politics is not a substitute for law, but it changes the calendar. Elections affect oversight, appropriations, hearings, state-law experiments, and federal preemption. Polling and news salience are weak signals; they tell a team when to refresh legal analysis, not what the law is.
Not every political signal deserves the same weight. CourtListener, Federal Register, LDA, FEC, and official party or White House pages can support route-risk analysis. GDELT and polling trackers are directional context unless a primary record or pollster methodology supports the claim.
Model routing should treat a legal-political change as route drift: review approved-use lists, source rights, data boundaries, contract terms, and whether the route remains acceptable.
Events that reopen approval
| Scenario | Trigger | Route risk | Control |
|---|---|---|---|
| A litigation team uses legal RAG for cited research. | A court ruling questions source rights, fair use or competing-product use 133. | The answer model may still perform well, but the source path, training data, retrieval corpus or indemnity position may no longer clear. | Check source licences, training and fine-tuning provenance, permitted outputs, indemnity, and whether the legal database route is still approved. |
| A firm deploys a model route for EU-facing regulated or high-risk work. | AI Act classification, GPAI documentation, copyright-policy or incident-reporting obligations become active 81 82. | A generic model approval may not satisfy use-case classification, logging, human oversight, documentation or provider-evidence requirements. | Classify the use case, demand supplier documentation, record oversight logs, and define the incident and withdrawal path before production. |
| A national law firm wants one AI route across US states and public-sector clients. | Federal preemption proposals, state AI statutes, agency rules or procurement guidance move in different directions 138. | A single national policy may miss state-specific duties, public-sector restrictions or deployer documentation needs. | Keep a jurisdiction-by-use-case approval register and require local route clearance when the work enters regulated or public-sector channels. |
| A government or regulated-sector matter sits inside an election cycle. | Primaries, committee control, appropriations or agency leadership changes alter policy posture 137 139. | The same model route can become politically or commercially awkward before the legal text changes. | Schedule route reviews around election and transition windows; update approved-use lists, contract positions and client-risk notes. |
| A vendor's lobbying or coalition activity shifts toward copyright, liability, export controls or government sales. | LDA filings show new issues, clients, lobbyists, bills or spending jumps 108. | The vendor may be anticipating a legal bottleneck that has not yet appeared in product marketing or contract terms. | Refresh vendor questions on liability, source rights, data transfer, export controls, roadmap dependencies and termination rights. |
| AI becomes a public controversy around jobs, children, copyright, safety or data centres. | Polling, party messaging or news volume creates a low-confidence watch cue before a primary legal record exists 140 141. | Public salience can change procurement appetite, board posture or regulator attention, but it is weak evidence on its own. | Treat it as a monitoring flag only; require a legal text, agency record, docket item or pollster methodology before changing route rules. |
Read the table as examples of route drift. The event does not choose the model; it tells the buyer which approval question to reopen.
Operational signals
The dull records show whether a model is usable in production. Benchmarks show capability; operational signals show whether the stack can be bought, called, debugged, secured and trusted. A model shortlist needs records vendors do not polish into marketing: status pages, router catalogues, package registries, model hubs, issue trackers, security advisories, source-provider notices, regulator pages and archived trust pages. These sources do not rank models. They reveal context limits, price changes, supported tool parameters, release churn, dependency risk, reliability and the errors users keep hitting.
Reliability needs translation. A one-point availability gap is 0.01 x 365 days, or 3.65 days a year if spread evenly. If those hours land inside deadline-sensitive production windows, the business impact is not a small decimal. The route needs a fallback owner, a degradation mode and a retest trigger.
Use operational signals as deltas in the route register. The useful question is narrow: does this change the route, the control, the fallback, the eval, the contract or the reviewer instruction?
Worst-week downtime cannot be inferred from an aggregate uptime percentage. Incident timing, component scope, recovery cadence and customer tier decide whether a published decimal was harmless background noise or a production event.
Operational signals do not prove model quality. They identify the procurement questions that deserve renewed scrutiny.
Start with the decimal because it is the number procurement and risk teams usually see first. Use only observed status-page uptime here; contractual SLA floors are a different source class and do not belong in the same chart.
Values checked on 27 May 2026. Rows use observed status-page uptime for OpenAI and Claude. SLA floors are excluded because they are not comparable uptime history.
The zero-origin percentage view explains why the numbers look reassuring. The buyer question appears only after observed uptime inputs are translated into time that someone must cover with retries, fallback routing, client communication or manual work.
Method: (100 - observed uptime) / 100 x 365 days. Lower is better. Caveat: not an SLA result, user-specific downtime or quality score.
Annualized days still smooth the calendar. The harder operational test is whether official incidents clustered inside a period the buyer could not move.
Values checked on 27 May 2026. Rows sum the busiest relevant seven-day incident cluster from official status histories. Caveat: fallback sizing, not model-quality ranking.
Reliability is only one signal family. The same route-review discipline applies when catalogues, model cards, package registries, trust pages, security advisories or source-provider records change.
Signals that reopen approval
| Source | Signal | Use |
|---|---|---|
| Provider and cloud status pages | Published uptime windows, incident history, component degradation and recovery cadence. | Test fallback routing, degradation modes and client promises; do not treat unequal status-page windows as an SLA ranking. |
| Model-router catalogues | Model availability, context limits, price, tool support, structured outputs and provider routing. | Build a substitution matrix before committing sensitive workflows to one provider. |
| Model hubs and release notes | Downloads, licences, tags, task families, model cards, deprecations and update cadence. | Separate real route momentum from paper-only interest and stale model assumptions. |
| Package registries and issue trackers | SDK downloads, release churn, dependency changes, tool-call errors, rate limits and regression reports. | Design private evals around failures that users and integrators are already hitting. |
| Archived trust, privacy and pricing pages | Changed claims, removed limits, policy drift, data-use language and price packaging. | Ask why the route changed before renewing approval or expanding use. |
| Security advisories | CVEs, known exploited vulnerabilities and affected model-serving, parser, browser, notebook and agent dependencies. | Map the route's runtime stack to patch gates, compensating controls and reapproval triggers. |
| Regulator, procurement and source-provider records | New rules, access changes, public tenders, source-rights constraints and resilience expectations. | Attach policy, source-rights and operational-resilience changes to the route register. |
| Domain and certificate metadata | New subdomains, regional surfaces, developer endpoints, trust portals and deployment names. | Spot new surfaces that need DPA, security and data-residency review. |
| Grant databases | Funded research themes, institutions, labs and pre-company categories. | Track future model and workflow categories before the vendor market has names for them. |
| Court and lobbying records | Disputes, hallucination litigation, product claims, regulatory pressure, procurement, copyright, export controls and liability campaigns. | Add legal, policy and reputational risk to model-route monitoring. |
AI agents for legal workflows
The provider layer turns models into legal workflows. Legal AI providers compete on sources, workflow, permissions, review experience, deployment route and adoption. Harvey, Legora, CoCounsel, Lexis+ AI, Hebbia, Colabra and other legal platforms package models with legal sources, matter workflows and review UX. Microsoft, Google, AWS and cloud platforms package model access with deployment controls, tenant governance and integration surfaces.
Buy mature workflows when source rights, review UX and adoption are harder than model access. Build where proprietary data, client process, sovereignty or internal controls create the advantage.
Provider selection is an approval decision: what sources the system can use, what actions it can take, what the lawyer can verify and where the data sits.
Implementation cases
Implementation stories show the variables a model leaderboard cannot see. Use them to identify route variables, not to prove quality. The practical point is route order: a model starts to matter after the firm decides what sources it may see, who reviews the answer, which workflow owns it, and how the result is measured.
That changes a model shortlist. Legal research, diligence, internal helpdesk work and firmwide AI platforms can all use frontier models. They still need different source rights, confidentiality controls, training plans and acceptance tests.
Treat vendor-authored stories as pattern evidence until a customer page, filing, independent study or measured denominator corroborates them. Extract the route shape: user, workflow, source path, control, result and caveat.
Do not carry polished quotes into procurement unless they answer a denominator question: matter mix, time period, review burden, error rate, adoption rate or repeatable throughput.
Implementation signals that change model selection
Zarwin Baum
The firm announcement ties CoCounsel use to document review, research, drafting, client service, confidentiality, and oversight.
- Workflow
- Document review, legal research, drafting, and client-service support through CoCounsel.
- Stack
- CoCounsel Legal
- Deployment
- Responsible-adoption narrative with confidentiality, oversight, and legal-service framing.
- Result
- Useful model signal: legal output quality depends on review gates and source-handling controls, not only the generator.
- Caveat
- Vendor-adjacent story; it does not publish a reproducible benchmark or matter-level error rate.
Rupp Pfalzgraf
The firm reported a 10% increase in case workload after adopting Lexis+ AI 125; LexisNexis case material packages adoption around workflow speed and usage.
- Workflow
- Lexis+ AI for legal research, drafting, and matter throughput.
- Stack
- Lexis+ AI
- Deployment
- Firmwide adoption story around usage, case workload, and research/drafting speed.
- Result
- Useful model signal: compare cost per accepted matter output, not only model leaderboard score.
- Caveat
- The workload increase is reported by the firm/vendor ecosystem and needs independent denominator, time period, and matter-mix context.
Reed Smith
Harvey's Reed Smith spotlight and public discussion emphasize platform rollout, training, user adoption, and leadership support.
- Workflow
- Firm AI platform adoption, training, and repeat use of Harvey for legal work.
- Stack
- Harvey
- Deployment
- Public spotlight and video frame adoption as firmwide enablement rather than a standalone model swap.
- Result
- Useful model signal: route selection has to include enablement, training, and workflow ownership because these change realized quality.
- Caveat
- Public video and vendor spotlight are high-signal for adoption mechanics but weak for quantitative output quality.
A&O Shearman
The firm's public AI page reinforces that legal AI model routes have to fit practice-area governance and client-facing legal workflows.
- Workflow
- Firm-developed AI tools and practice-area expertise around legal AI and regulated AI matters.
- Stack
- Firm AI tooling plus legal AI expertise
- Deployment
- Public practice page connects AI to legal services, internal tooling, and regulated workflow needs.
- Result
- Useful model signal: the model route must be evaluated against practice-specific source rights, risk tier, and counsel accountability.
- Caveat
- The public page is descriptive rather than a measured case study; use it as workflow-shape evidence only.
LegalCo
CIPD's case study emphasizes adoption mechanics and internal expertise rather than model brand.
- Workflow
- AI legal assistant adoption and internal expert-led change management.
- Stack
- AI assistant implementation
- Deployment
- Independent case study frames adoption through culture, internal experts, and operating model.
- Result
- Useful model signal: training, culture, and internal ownership can decide whether a technically strong route gets used.
- Caveat
- Independent case study, but it is still a single-organization implementation rather than a controlled model comparison.
Expert watchlist
Follow the people producing tests, tools, and failure analysis. A useful legal AI feed is not a popularity contest. It is an early-warning system. The best follows publish artefacts: benchmarks, datasets, redlining tools, source infrastructure, evaluation rubrics, regulatory trackers, and sober post-mortems. They surface the work before it becomes vendor copy.
Treat named follows as a monitoring surface, not an endorsement badge. Affiliations, social handles and activity levels change. Replace anyone whose signal turns into generic AI commentary.
The inclusion test is simple: would following this person improve a buyer's eval design, source strategy, model routing, or legal workflow controls?
People to monitor
| Expert | Lane | Signal | Links |
|---|---|---|---|
Anna Guo Founder, Legal Benchmarks | EvalsSGPractitioner benchmarks | She is turning legal AI evaluation into concrete buyer rubrics and real-workflow benchmarks. Watch framework updates, benchmark methodology, scoring criteria, and procurement failure modes. | |
Elgar Weijtmans CTO, Legal Benchmarks | ProcurementNLTool evaluation | He brings a practical screen-to-pilot view of legal AI procurement and evaluation workflows. Watch screening criteria, pilot design, vendor scoring, and evaluation tooling. | |
Jamie Tso Founder, LegalQuants | BuilderHKLawyer-built tools | He shows what lawyers can prototype themselves when models make workflow software cheap. Watch redlining, tabular review, signature packets, contract simulation, and LegalQuants output. | |
Raymond Sun Founder, LegalQuants | RegulationAUAI regulation tracker | He pairs legal practice with working AI-regulation data products and lawyer-builder community work. Watch regulatory tracking, global coverage claims, APIs, and LegalQuants builder patterns. | |
Hou Fu Ang Senior legal counsel; Alt + Counsel | InfrastructureSGOpen legal tooling | He writes from practice and builds legal data and redlining infrastructure that other lawyers can inspect. Watch open-source redlines, legal-data infrastructure, and practical AI governance notes. | |
Troy Zhaoyi Song Disputes and technology lawyer | Source accessHKCaseThread and legal data | He focuses on jurisdictional source bottlenecks and lawyer-controlled analysis of judgments. Watch English-judgment data constraints, CaseThread, BYO-key tooling, and Hong Kong/China legal AI context. | |
Neel Guha Stanford CS and law | ResearchUSLegal benchmarks | His work connects legal reasoning, retrieval, benchmarks and AI governance with strong technical range. Watch LegalBench, legal retrieval benchmarks, law-following AI, and model leaderboard changes. | |
Julian Nyarko Stanford Law and liftlab | ResearchUSLegal judgement and evals | He is working on legal subjectivity, contract quality, legal personas, and AI model evaluation. Watch liftlab outputs, contract-risk assessment, legal-persona work, and legal benchmarking papers. | |
Megan Ma Stanford CodeX | Computable lawUSLegal knowledge to code | She works where legal knowledge, generative AI, and human-machine collaboration meet. Watch CodeX, legal education, computable-law projects, and responsible legal AI systems. | |
Daniel Martin Katz Illinois Tech; 273 Ventures | Legal NLPUSLegal data and agentic systems | He has long worked at the intersection of law, NLP, legal data, and applied AI systems. Watch legal NLP research, agentic AI in regulated industries, ALEA, KL3M, and Kelvin Legal DataPack. |
Public profiles and work links were current on 31 May 2026.
Known unknowns
What public evidence cannot decide. Public evidence can narrow the route, but it cannot approve legal production work. The missing pieces are firm-specific: matter types, source licences, document condition, reviewer standards, data boundaries, local law, client rules and risk appetite.
Use public benchmarks and vendor claims to choose what to test privately. Use the private eval to decide which route can move from pilot to approved use.
Some risks sit outside benchmark design. Archive drift, CVEs, exploited vulnerabilities, domain changes, certificate issuance, court disputes and lobbying filings can change the risk of a route even when the model score stays flat.
The strongest legal AI programmes will own their evaluation data. That is the only way to detect model drift, compare providers, test retrieval, measure cost per correct output and decide which tasks can safely move from pilot to production.
A vendor can sell capability. The legal team defines acceptable work product.
What public evidence still misses
| Gap | Why it matters | Move now |
|---|---|---|
| No trusted production legal benchmark | Public scores do not reflect your matters, sources, reviewers, or clients. | Build private evals by practice group. |
| Weak cross-border evaluation | Language and jurisdiction are still too often conflated, and Russian-law evidence remains especially thin. | Create original-language, jurisdiction-specific tests. |
| Regulatory routes change faster than benchmarks | Government-device bans, sanctions FAQs, data-transfer rules and export-control guidance can make a valid model route invalid. | Maintain a live route register by country, sector, data class and provider. |
| Sparse reranker evidence | Rerankers can decide whether the right authority reaches the model. | Test rerankers internally with legal snippets. |
| Legal OCR is under-tested | Redlines, tables, schedules, signatures, and transcripts break generic parsing. | Benchmark parsers before answer models. |
| Citation support is not enough | A citation can exist and still fail to support the proposition. | Add proposition-level verification. |
| Cost per correct output is opaque | Long-context and agent loops can make cheap tokens expensive. | Track cost per verified answer, not cost per token. |
| Model updates change behaviour | A model can improve overall and regress on one legal workflow. | Version-pin and regression-test. |
| Operational records are indirect | Archive drift, CVEs, domains, certificates, court records, and lobbying filings flag risk but do not measure legal quality. | Use them as triggers for renewed evals and vendor questions. |
| Manual hard sources remain fragmented | Bankruptcy, UCC, tax incentives, insurance, local board packets, app stores, and demo-video forensics lack one clean global API. | Create jurisdiction-specific collection lanes when the matter warrants it. |
Vendor directory
Main model providers and inference platforms represented by source-ledger rows in this report.
Model providers and inference platforms
- OpenAIFrontier LLM and reasoning models
- AnthropicFrontier LLM and reasoning models (Claude)
- Google / Vertex AIGemini models and cloud model platform
- Meta AIOpen-weight LLMs (Llama)
- Mistral AIOpen-weight and hosted LLMs
- CohereEnterprise LLMs, embeddings, and reranking
- DeepSeekOpen-weight Chinese frontier LLMs
- Alibaba Cloud (Qwen)Open-weight LLMs, embeddings, and rerankers
- Microsoft Azure OpenAICloud OpenAI deployment and quota route
- AmazonNova/Titan models and Bedrock platform
- Voyage AILegal-tuned embedding and reranking models
- IsaacusLegal embedding and reranking models (Kanon)
- GroqLow-latency LPU-based inference
- OpenRouterUnified router across model providers
- Hugging FaceOpen model catalogue and model metadata surface
Inclusion does not imply endorsement; classes reflect the ledgered source used for this report.
Source ledger
Primary sources and diligence anchors behind the analysis.
Primary source list
| Ref | Source | Verification | Evidence | Open source |
|---|---|---|---|---|
| 1 | OpenAI - introducing GPT-5.5 | Publisher: OpenAI; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | E1: OpenAI positions GPT-5.5 for professional work, tool use, online research, data analysis, documents, spreadsheets, and operating software. | E96: OpenAI's GPT-5.5 launch lists API pricing at $5 per 1M input tokens and $30 per 1M output tokens for gpt-5.5, and $30 per 1M input tokens and $180 per 1M output tokens for gpt-5.5-pro, with a 1M context window. | Open source |
| 2 | OpenAI API model docs - model catalogue | Publisher: OpenAI API model docs; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | Open source | |
| 3 | Anthropic - introducing Claude Opus 4.8 | Publisher: Anthropic; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | E2: Anthropic introduced Claude Opus 4.8 on 28 May 2026 as an Opus 4.7 upgrade with stronger coding, agentic, and professional-work performance, effort controls, dynamic workflows, unchanged regular pricing, and API ID claude-opus-4-8. | Open source |
| 4 | Google - Gemini 3.1 Pro announcement | Publisher: Google; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | E3: Google released Gemini 3.1 Pro for complex reasoning and agentic workflows across Gemini API, Vertex AI, and related developer products. | Open source |
| 5 | Google Cloud - Gemini 3 Pro on Vertex AI | Publisher: Google; Kind: Cloud documentation; Checked: 2026-06-19; Status: verified | E4: Vertex AI documents Gemini 3 Pro preview with 1M-token multimodal context, PDFs, function calling, grounding, code execution, and data-residency controls. | E99: Google's Vertex AI Gemini 3 Pro documentation treats the model as a preview offering with 1,048,576 input-token limit, PDF/multimodal support, data-residency controls, and warnings that it may occasionally guess when information is missing or misinterpret dense charts, tables, or complicated resources. Limitation: vendor-controlled source; use for route shape and test operating claims in diligence. | Open source |
| 6 | Anthropic model overview | Publisher: Anthropic; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | E97: Anthropic's model overview lists Claude Opus 4.8 as its most capable generally available model for complex reasoning, long-horizon agentic coding, and high-autonomy work, with 1M context, moderate comparative latency, 128K maximum output, and $5 per 1M input tokens / $25 per 1M output tokens. | Open source |
| 7 | AWS - Amazon Bedrock overview | Publisher: AWS; Kind: Cloud documentation; Checked: 2026-06-19; Status: verified | E5: Amazon Bedrock emphasizes multi-model choice, agent deployment, guardrails, customization, and privacy controls across foundation model providers. Limitation: vendor-controlled source; use for route shape and test operating claims in diligence. | Open source |
| 8 | Microsoft - Azure OpenAI data, privacy, and security | Publisher: Microsoft; Kind: Cloud documentation; Checked: 2026-06-19; Status: verified | E6: Azure OpenAI documents privacy controls, optional persistence features, modified abuse monitoring, and enterprise data-handling boundaries. Limitation: vendor-controlled source; use for route shape and test operating claims in diligence. | Open source |
| 9 | OpenAI - enterprise privacy and API data controls | Publisher: OpenAI; Kind: Privacy and data terms; Checked: 2026-06-19; Status: verified | E7: OpenAI states API and business data are not used for training by default and documents retention controls and zero data retention eligibility. | Open source |
| 10 | Google Cloud - Vertex AI data governance | Publisher: Google; Kind: Privacy and data terms; Checked: 2026-06-19; Status: verified | E8: Google Cloud states Vertex AI managed models are subject to a training restriction absent customer permission and supports data governance controls. | Open source |
| 11 | Anthropic help - commercial products as processor | Publisher: Anthropic; Kind: Privacy and data terms; Checked: 2026-06-19; Status: verified | E9: Anthropic states commercial customers control their data and Anthropic processes it as instructed without using it to train generative models. | Open source |
| 12 | Meta - Llama get started and Llama 4 model access | Publisher: Meta; Kind: Model documentation; Checked: 2026-06-19; Status: verified | E10: Meta distributes Llama 4 models and highlights Scout, Maverick, and Llama Guard 4 for open and partner deployment paths. | Open source |
| 13 | DeepSeek - V3.1 release notes | Publisher: DeepSeek; Kind: Model documentation; Checked: 2026-06-19; Status: verified | E11: DeepSeek V3.1 added hybrid think and non-think modes, 128K context, tool/function support, and open-source weights. | Open source |
| 14 | Qwen - Qwen3 embedding and reranker | Publisher: Qwen; Kind: Model documentation; Checked: 2026-06-19; Status: verified | E12: Qwen3 Embedding and Qwen3 Reranker are released in 0.6B, 4B, and 8B variants for multilingual embedding and reranking workloads. | Open source |
| 15 | Mistral - models overview | Publisher: Mistral; Kind: Official model documentation; Checked: 2026-06-19; Status: verified | E13: Mistral lists open frontier-class generalist, coding, reasoning, OCR, audio, and moderation models, including Mistral Large 3 and OCR 3. | Open source |
| 16 | Mistral - Document AI OCR processor | Publisher: Mistral; Kind: Document AI documentation; Checked: 2026-06-19; Status: verified | E14: Mistral Document AI OCR returns markdown, tables, images, hyperlinks, headers, footers, and metadata for PDFs, images, and Office formats. | Open source |
| 17 | LegalBench - RegLab and Stanford | Publisher: LegalBench; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E15: LegalBench is a 162-task legal reasoning benchmark built with legal professionals and covering multiple forms of legal reasoning. | Open source |
| 18 | Vals AI - LegalBench leaderboard | Publisher: Vals AI; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E16: Vals AI's LegalBench leaderboard updated 26 May 2026 lists 110 tested models, with source-backed rows across Google, OpenAI, Anthropic, Alibaba, Moonshot AI, xAI, Zhipu AI, Meta, MiniMax, DeepSeek, Mistral, and Llama/Together alternatives. | E218: Vals LegalBench's visible 26 May 2026 leaderboard rows used in the benchmark grid show Gemini 3.1 Pro Preview at 87.398, Gemini 3 Pro Preview at 87.025, Gemini 3 Flash Preview at 86.858, GPT-5.5 at 86.515, and GPT-5.4 at 86.044. | Open source |
| 19 | LegalBench-RAG | Publisher: LegalBench-RAG; Kind: Retrieval benchmark; Checked: 2026-06-19; Status: verified | E17: LegalBench-RAG evaluates retrieval in legal RAG with 6,858 query-answer pairs over more than 79M characters. | Open source |
| 20 | Legal RAG Bench | Publisher: Legal RAG Bench; Kind: Retrieval benchmark; Checked: 2026-06-19; Status: verified | E18: Legal RAG Bench reports retrieval as the primary driver of legal RAG performance and quantifies gains from Kanon 2 Embedder. | Open source |
| 21 | LawBench | Publisher: LawBench; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E19: LawBench evaluates Chinese legal tasks across 20 tasks and cognitive levels, exposing jurisdiction and language gaps in English-heavy evaluation. | E221: LawBench's zero-shot overall average rows used in the grid show Qwen1.5-72B-Chat at 56.26, GPT-4 at 52.35, Qwen1.5-14B-Chat at 46.85, InternLM2-7B-Chat at 45.80, and Baichuan2-13B-Chat at 44.57. | Open source |
| 22 | Isaacus - Kanon 2 Embedder and legal models | Publisher: Isaacus; Kind: Model documentation; Checked: 2026-06-19; Status: verified | E20: Isaacus describes Kanon 2 Embedder as a legal embedding model for legal information retrieval and RAG with configurable dimensions and a 16K context window. | Open source |
| 23 | Kanon 2 Reranker release | Publisher: Kanon 2 Reranker release; Kind: Model documentation; Checked: 2026-06-19; Status: verified | E21: Kanon 2 Reranker is positioned as a legal-domain reranker with reported gains over Qwen3 Reranker and Voyage Rerank on legal benchmarks. | Open source |
| 24 | Voyage AI - legal embeddings and Harvey partnership | Publisher: Voyage AI; Kind: Vendor documentation; Checked: 2026-06-19; Status: verified | E22: Harvey and Voyage describe domain-specific legal embeddings trained on more than 20B tokens of US case law for improved legal retrieval. Limitation: vendor-controlled source; use for route shape and test operating claims in diligence. | Open source |
| 25 | Cohere - Rerank model docs | Publisher: Cohere; Kind: Vendor documentation; Checked: 2026-06-19; Status: verified | E23: Cohere documents Rerank v3.5 for multilingual documents and semi-structured JSON with 4,096-token document context. Limitation: vendor-controlled source; use for route shape and test operating claims in diligence. | Open source |
| 26 | Azure AI Document Intelligence layout model | Publisher: Azure; Kind: Document AI documentation; Checked: 2026-06-19; Status: verified | E24: Azure Document Intelligence layout extracts text, tables, selection marks, and document structure from PDFs, images, and Office documents. | Open source |
| 27 | Google Document AI form parser and enterprise OCR | Publisher: Google; Kind: Document AI documentation; Checked: 2026-06-19; Status: verified | E25: Google Document AI form parser extracts key-value pairs, tables, selection marks, generic entities, text, and layout. | Open source |
| 28 | AWS Textract | Publisher: AWS; Kind: Document AI documentation; Checked: 2026-06-19; Status: verified | E26: AWS Textract extracts printed text, handwriting, tables, forms, and related document data from PDFs and images. | Open source |
| 29 | Stanford HAI - legal hallucinations | Publisher: Stanford HAI; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E27: Stanford HAI reports pervasive legal hallucinations in general-purpose LLMs on legal factual questions. | Open source |
| 30 | Stanford HAI - AI on trial for legal RAG tools | Publisher: Stanford HAI; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E28: Stanford HAI tested legal research AI systems and found hallucination risk persisted in RAG-based legal products. | Open source |
| 31 | ABA Formal Opinion 512 | Publisher: ABA Formal Opinion 512; Kind: Legal ethics guidance; Checked: 2026-06-19; Status: verified | E29: ABA Formal Opinion 512 ties generative AI use to duties of competence, confidentiality, communication, supervision, and fees. | Open source |
| 32 | EUR-Lex - Regulation (EU) 2024/1689 Artificial Intelligence Act | Publisher: EUR-Lex; Kind: Regulation; Checked: 2026-06-19; Status: monitor | E30: The EU AI Act high-risk framework creates operational requirements for risk management, data governance, technical documentation, logs, transparency, oversight, accuracy, robustness, and cybersecurity. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 33 | Legora product and aOS announcement | Publisher: Legora; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E31: Legora announced an agentic operating system for legal work with agents, monitors, knowledge, interfaces, and governance. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 34 | Harvey legal AI platform | Publisher: Harvey; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E32: Harvey positions itself as AI software for legal and professional services covering assistant, document analysis, drafting, and enterprise workflows. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 35 | Hebbia and OpenAI case study | Publisher: Hebbia; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E33: OpenAI's Hebbia case study describes a multi-agent research platform for finance and legal workflows over complex private documents. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 36 | Colabra for law firms | Publisher: Colabra; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E34: Colabra describes an AI workspace for M&A due diligence producing clause-linked risk registers, entity maps, and live gap analysis. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 37 | Microsoft - AI agents in Microsoft 365 and Copilot Studio | Publisher: Microsoft; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E35: Microsoft describes agents embedded in Microsoft 365, Copilot Studio, and Azure AI for documents, spreadsheets, email, workflows, and custom automation. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 38 | LegalBenchmarks.ai - legal AI evaluation framework | Publisher: LegalBenchmarks.ai; Kind: Evaluation framework; Checked: 2026-06-19; Status: monitor | E36: LegalBenchmarks.ai's public framework uses a staged procurement process and 8 criteria across strategic fit, functionality, robustness, security, data privacy, vendor risk, adoption support, and cost/resourcing. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 39 | LegalBenchmarks.ai - contract drafting benchmark | Publisher: LegalBenchmarks.ai; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E37: LegalBenchmarks.ai's contract drafting benchmark evaluated hundreds of legal drafting outputs and found that legal-specific tools do not automatically beat general-purpose assistants on reliability or usefulness. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 40 | LegalBenchmarks.ai - information extraction benchmark | Publisher: LegalBenchmarks.ai; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E38: LegalBenchmarks.ai's information extraction research identifies practical failure modes including multi-document misses, leading-question mirroring, file/OCR handling failures, and failure to flag contradictions. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 41 | LegalBenchmarks.ai - purpose-built does not mean better | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E39: LegalBenchmarks.ai argues that purpose-built legal AI earns its premium through better context, workflow design, and legal output conventions rather than through a legal label alone. | E100: LegalBenchmarks.ai argues that production legal AI systems cannot route every output through the most powerful model, longest context, deepest reasoning, and multiple verification loops because unit economics fall apart fast; real systems trade off smaller models, latency, output length, workflow scope, and verification passes. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 42 | LegalBenchmarks.ai - vendor legal benchmarks compared | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E40: LegalBenchmarks.ai compares Harvey LAB, Ivo, and GC AI and highlights why vendor benchmarks need public tasks, clear scoring, human baselines, judge-agreement reporting, and disclosed limitations. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 43 | LegalBenchmarks.ai - speculative bets in legal AI | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E41: LegalBenchmarks.ai describes legal AI market hedging across infrastructure, model, knowledge, application, and services layers, with switchability, integration debt, and evaluation paralysis as procurement concerns. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 44 | LegalBenchmarks.ai - Claude for Word demo analysis | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E42: LegalBenchmarks.ai's Claude for Word analysis highlights legal-domain gaps in demo realism, surgical redlining, proofreading, counterparty comments, and expert verification. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 45 | LegalBenchmarks.ai - identify legal artifacts first | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E43: LegalBenchmarks.ai recommends starting legal AI evaluation by identifying the legal artifact, lifecycle stage, human-attention level, and acceptable AI-assistance level before comparing tools. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 46 | LegalBenchmarks.ai - modern legal AI exploits | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E44: LegalBenchmarks.ai's modern legal AI exploits analysis surfaces adversarial patterns including hidden instructions in documents, agent goal hijacking, long-context manipulation, adversarial audio, knowledge poisoning, and misuse by authorised users. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 47 | LegalBenchmarks.ai - writing for the AI audience | Publisher: LegalBenchmarks.ai; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E45: LegalBenchmarks.ai argues that legal work is increasingly read first by AI, so lawyers need drafting practices that preserve qualifiers, context, defined terms, and risk signals through summarization. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 48 | Harvey - Legal Agent Benchmark | Publisher: Harvey; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E46: Harvey introduced Legal Agent Benchmark as an open-source legal-agent evaluation framework with more than 1,200 tasks across 24 practice areas and rubric-based evaluation. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 49 | PRBench - professional reasoning benchmark | Publisher: PRBench; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E47: PRBench is an expert-rubric benchmark for high-stakes finance and legal reasoning with open-ended tasks, professional contributors, and low top-model scores on hard legal subsets. | E219: PRBench's hard legal subset without web search reports the visible top rows used in the grid as GPT-5 (High) at 0.377, o3 (High) at 0.352, Grok 4 Fast Reasoning at 0.325, Gemini 2.5 Pro at 0.297, and Claude Sonnet 4.5 at 0.294. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 50 | MLEB - massive legal embedding benchmark | Publisher: MLEB; Kind: Retrieval benchmark; Checked: 2026-06-19; Status: monitor | E48: MLEB reports NDCG@10 scores for 21 embedding models across ten expert-annotated legal datasets, with Kanon 2 Embedder first, followed by Voyage, Qwen3 Embedding, Gemini Embedding, Voyage Law, OpenAI Text Embedding, Jina, Gemma, Snowflake, Granite, BGE, Mxbai, and E5 alternatives. | E163: MLEB ranks newer general Voyage embeddings ahead of the older Voyage Law 2 legal-specific model on task-average legal retrieval NDCG@10, making the procurement lesson evaluation cadence rather than domain branding. | E220: MLEB's task-average NDCG@10 rows used in the grid show Kanon 2 Embedder at 86.03, Voyage 3 Large at 85.71, Voyage 3.5 at 84.07, Qwen3 Embedding 8B at 82.96, and Voyage 3.5 Lite at 82.41. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 51 | COLIEE 2026 | Publisher: COLIEE 2026; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E49: COLIEE is an annual legal information extraction and entailment competition focused on legal retrieval, entailment, and related legal NLP tasks. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 52 | LexGLUE | Publisher: LexGLUE; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E50: LexGLUE is an English legal language understanding benchmark that remains useful for classification and legal NLU baselines. | E226: LexGLUE's main-paper arithmetic mean macro-F1 rows used in the grid show Legal-BERT at 72.0, CaseLaw-BERT at 70.9, Longformer at 70.5, DeBERTa at 69.7, and BigBird at 69.6. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 53 | LEXam | Publisher: LEXam; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E51: LEXam benchmarks legal reasoning over law exams in English and German, including open-ended and multiple-choice questions across multiple legal subjects. | E225: LEXam's open-question judge-score rows used in the grid show GPT-5 at 70.20, Gemini 2.5 Pro at 67.40, Claude 3.7 Sonnet at 62.86, Claude 4.5 Sonnet at 62.76, and GPT-5 mini at 60.32. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 54 | Legal Benchmarks LinkedIn public page | Publisher: Legal Benchmarks LinkedIn public page; Kind: Public expert commentary; Checked: 2026-06-19; Status: monitor | E52: Legal Benchmarks' public LinkedIn presence reinforces the practitioner discourse shift from feature comparison toward artefact definition, risk, structure, and acceptable legal performance. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 55 | LinkedIn - Legalweek 2026 AI adoption post | Publisher: LinkedIn; Kind: Public expert commentary; Checked: 2026-06-19; Status: monitor | E53: Public Legalweek commentary described agentic systems as practical infrastructure for legal operations, knowledge management, and litigation rather than only futureware. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 56 | Law What's Next - evals and benchmarking legal AI with Anna Guo | Publisher: Law What's Next; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E54: A public legal AI benchmarking interview with Anna Guo emphasized output reliability, output usefulness, platform workflow support, context management, verification, and prompt-injection risk. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 57 | Artificial Lawyer - Claude for Legal launches | Publisher: Artificial Lawyer; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E55: Artificial Lawyer's Claude for Legal coverage frames frontier-model labs as moving deeper into lawyer workflows through legal offerings, connectors, Word workflows, and legal tech partnerships. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 58 | LawBench GitHub - Chinese legal benchmark | Publisher: LawBench GitHub; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E56: The LawBench repository states that its tasks are based on the Chinese legal system, span 20 tasks across three cognitive levels, and report results for multilingual, Chinese, and legal-specific models. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 59 | LexEval - comprehensive Chinese legal benchmark | Publisher: LexEval; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E57: LexEval introduces a Chinese legal benchmark with 23 tasks and 14,150 questions, using existing datasets, exam material, and legal-expert annotations to evaluate legal capability and ethical issues. | E222: LexEval's zero-shot average rows used in the grid show GPT-4 at 52.4, Qwen-14B-Chat at 48.6, Qwen-7B-Chat at 44.8, ChatGPT at 39.6, and InternLM-7B-Chat at 36.9. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 60 | Japanese bar exam open-ended legal reasoning benchmark | Publisher: Japanese bar exam open-ended legal reasoning benchmark; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E58: The Japanese bar-exam writing benchmark evaluates open-ended Japanese legal reasoning with legal-expert review and manual hallucination analysis, rather than only multiple-choice answers. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 61 | JBE-QA - Japanese bar exam question-answering dataset | Publisher: JBE-QA; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E59: JBE-QA is a Japanese legal knowledge question-answering dataset built from short-answer Japanese bar examination questions from 2015 to 2024. | E224: JBE-QA's zero-shot F1 rows used in the grid show Opus 4.1 with reasoning at 0.814, GPT-5 at 0.794, Sonnet 4 with reasoning at 0.780, o3 at 0.769, and GPT-4.1 at 0.724. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 62 | KBL - Korean legal language understanding benchmark | Publisher: KBL; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E60: KBL proposes a Korean legal language benchmark covering legal knowledge, legal reasoning, and Korean bar-exam tasks for assessing Korean legal language understanding. | E223: KBL's closed-book overall rows used in the grid show GPT-4 at 72.0, Claude 3.5 Sonnet at 70.0, Claude 3 Opus at 67.0, GPT-4o mini at 64.6, and Claude 3 Sonnet at 62.0. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 63 | LawTimes - SuperLawyer Korean bar exam report | Publisher: LawTimes; Kind: Market signal; Checked: 2026-06-19; Status: monitor | E61: LawTimes reported that Ro&Company's SuperLawyer scored 74% on the multiple-choice section of the 14th Korean Bar Examination, above reported GPT-4.5 and Claude 3.7 Sonnet scores on the same questions. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 64 | KMMLU - Korean massive multitask language understanding | Publisher: KMMLU; Kind: Multilingual benchmark; Checked: 2026-06-19; Status: monitor | E62: KMMLU is a Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects, designed to test Korean knowledge and reasoning rather than translated English competence alone. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 65 | MMLU-ProX - multilingual benchmark across 29 languages | Publisher: MMLU-ProX; Kind: Multilingual benchmark; Checked: 2026-06-19; Status: monitor | E63: MMLU-ProX covers 29 languages with parallel questions and reports substantial cross-language performance disparities, including a gap of up to 24.3%. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 66 | Global MMLU - cultural and linguistic bias benchmark | Publisher: Global MMLU; Kind: Multilingual benchmark; Checked: 2026-06-19; Status: monitor | E64: Global MMLU argues that translation alone is insufficient for multilingual evaluation, documents cultural bias in MMLU, and releases a 42-language benchmark with culturally sensitive and agnostic subsets. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 67 | Zakon.ru - Russian legal AI benchmark commentary | Publisher: Zakon.ru; Kind: Expert analysis; Checked: 2026-06-19; Status: monitor | E65: Russian practitioner commentary describes LegalBench-style legal benchmarks as useful but treats benchmark selection as task-specific, underscoring the need for local legal validation. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 68 | Habr - Shlepa large Russian benchmark | Publisher: Habr; Kind: General benchmark; Checked: 2026-06-19; Status: monitor | E66: Shlepa is a Russian-language benchmark effort for general Russian LLM evaluation; the search pass did not identify a comparable public Russian-law legal benchmark with model scores. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 69 | OpenRouter public model catalogue API | Publisher: OpenRouter public model catalogue API; Kind: Model-router catalogue; Checked: 2026-06-19; Status: monitor | E67: OpenRouter's public model catalogue exposes model IDs, context lengths, prices, supported tool parameters, structured-output support, and provider availability signals that can be diffed over time. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 70 | Hugging Face Hub models API | Publisher: Hugging Face; Kind: Model hub; Checked: 2026-06-19; Status: monitor | E68: The Hugging Face Hub models API exposes public model metadata, downloads, tags, licences, and task categories; the legal-model search surface makes open-model momentum observable outside papers and vendor pages. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 71 | npm download counts API | Publisher: npm download counts API; Kind: Package telemetry; Checked: 2026-06-19; Status: monitor | E69: npm download-count telemetry exposes adoption signals for AI SDKs and agent infrastructure packages, including OpenAI, Anthropic, LangChain, Vercel AI SDK, and MCP packages. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 72 | PyPI JSON API | Publisher: PyPI JSON API; Kind: Package registry; Checked: 2026-06-19; Status: monitor | E70: The PyPI JSON API exposes package versions, release history, summaries, and project links for Python AI and agent packages such as OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, and AutoGen. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 73 | GitHub REST API - search issues and pull requests | Publisher: GitHub REST API; Kind: Issue tracker; Checked: 2026-06-19; Status: monitor | E71: GitHub issue search exposes production friction that benchmark leaderboards miss: tool-call errors, retrieval defects, validation failures, agent loops, and integration regressions. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 74 | OpenAI status API | Publisher: OpenAI status API; Kind: Status page; Checked: 2026-06-19; Status: monitor | E72: OpenAI's status API exposes current operational indicators and incident surfaces that should feed model-routing fallback and reliability analysis. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 75 | Claude status API | Publisher: Claude status API; Kind: Status page; Checked: 2026-06-19; Status: monitor | E73: Claude's status API exposes current operational indicators and incident surfaces that should feed model-routing fallback and reliability analysis. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 76 | US DOJ Data Security Program | Publisher: US DOJ Data Security Program; Kind: Regulatory control; Checked: 2026-06-19; Status: monitor | E74: The US DOJ Data Security Program treats foreign-adversary access to US government-related and bulk sensitive personal data as an export-control-like national-security problem and went into effect on April 8, 2025. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 77 | US DOJ Data Security Program compliance guide | Publisher: US DOJ Data Security Program compliance guide; Kind: Regulatory guidance; Checked: 2026-06-19; Status: monitor | E75: The DOJ compliance guide lists countries of concern including China, Hong Kong, Macau, Russia, Iran, North Korea, Cuba, and Venezuela, and covers data brokerage, vendor, employment, and investment transactions involving access to covered data. | E76: Restricted transactions under the DOJ programme require security requirements, data compliance programmes, audits, recordkeeping, risk-based verification of data flows, vendor identity, ownership, citizenship or residency, end use, and transfer method. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 78 | BIS Connected Vehicles rule overview | Publisher: BIS Connected Vehicles rule overview; Kind: Sector restriction; Checked: 2026-06-19; Status: monitor | E77: BIS's connected-vehicles rule prohibits import or sale of certain connected vehicles and related hardware/software in the US when products have a sufficient nexus to China or Russia, with phased model-year restrictions. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 79 | BIS AI Diffusion Rule rescission and chip-control guidance | Publisher: BIS AI Diffusion Rule rescission and chip-control guidance; Kind: Export control; Checked: 2026-06-19; Status: monitor | E78: BIS announced non-enforcement and planned rescission of the AI Diffusion Rule while separately warning about PRC advanced computing ICs and U.S. AI chips being used for training or inference of Chinese AI models. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 80 | Federal Register - Chinese AI and surveillance entities added to Entity List | Publisher: Federal Register; Kind: Sanctions export control; Checked: 2026-06-19; Status: monitor | E79: The US Entity List rule added Chinese AI and surveillance companies including IFLYTEK, Megvii, SenseTime, and Yitu, imposing licence requirements for exports, reexports, or in-country transfers involving listed entities. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 81 | European Commission AI Act Explorer | Publisher: European Commission; Kind: Regulatory control; Checked: 2026-06-19; Status: monitor | E80: The EU AI Act official explorer maps prohibited AI practices, high-risk AI systems, provider and deployer obligations, transparency rules, GPAI models, enforcement, penalties, and application dates. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 82 | European Commission GPAI obligations under the AI Act | Publisher: European Commission; Kind: Regulatory guidance; Checked: 2026-06-19; Status: monitor | E81: European Commission GPAI guidance states that all GPAI providers must prepare technical documentation, implement a copyright policy, and publish a training-content summary, while systemic-risk GPAI providers face notification, risk mitigation, incident reporting, and cybersecurity obligations. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 83 | EUR-Lex summary - EU dual-use export controls | Publisher: EUR-Lex summary; Kind: Export control; Checked: 2026-06-19; Status: monitor | E82: EU dual-use controls cover exports, brokering, technical assistance, transit, and transfer of dual-use items, including software and technology, with additional attention to cyber-surveillance and military or WMD end-use risk. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 84 | European Commission Russia sanctions FAQs | Publisher: European Commission; Kind: Sanctions guidance; Checked: 2026-06-19; Status: monitor | E83: European Commission Russia sanctions FAQs include Article 5n provision-of-services guidance; the updated service-prohibition matrix covers legal advisory services, IT consultancy, software, and AI services consisting of access to models or platforms for training, fine-tuning, and inference. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 85 | CAC Interim Measures for Generative AI Services | Publisher: CAC Interim Measures for Generative AI Services; Kind: Regulatory control; Checked: 2026-06-19; Status: monitor | E84: China's Interim Measures for Generative AI Services apply to generative AI services offered to the mainland public, exclude non-public internal R&D use, require lawful data and base models, impose content and personal-information duties, and require security assessment and algorithm filing for services with public-opinion or social-mobilisation attributes. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 86 | CAC Provisions on Promoting and Regulating Cross-Border Data Flows | Publisher: CAC Provisions on Promoting and Regulating Cross-Border Data Flows; Kind: Data transfer control; Checked: 2026-06-19; Status: monitor | E85: CAC's 2024 cross-border data provisions clarify exemptions and thresholds but require security assessment for critical information infrastructure operators and for handlers exporting important data or large volumes of personal or sensitive personal information. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 87 | MOFCOM/MOST Catalogue of Technologies Prohibited or Restricted from Export | Publisher: MOFCOM/MOST Catalogue of Technologies Prohibited or Restricted from Export; Kind: Export control; Checked: 2026-06-19; Status: monitor | E86: MOFCOM and MOST's 2023 technology export catalogue took effect on publication and states that dual-use technologies in the catalogue are incorporated into export-control management. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 88 | Australia PSPF Direction update - DeepSeek | Publisher: Australia PSPF Direction update; Kind: Government restriction; Checked: 2026-06-19; Status: monitor | E87: Australia's PSPF Direction requires Australian Government entities to prevent use or installation of DeepSeek products, applications, and web services and remove existing instances from government systems and devices. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 89 | Taiwan Ministry of Digital Affairs - DeepSeek government restriction | Publisher: Taiwan Ministry of Digital Affairs; Kind: Government restriction; Checked: 2026-06-19; Status: monitor | E88: Taiwan's Ministry of Digital Affairs warned government agencies and critical-infrastructure entities to restrict DeepSeek AI products because the Chinese product raised cross-border transfer and information-leakage concerns. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 90 | Italian Garante - DeepSeek processing limitation | Publisher: Italian Garante; Kind: Privacy enforcement; Checked: 2026-06-19; Status: monitor | E89: Italy's Garante ordered an urgent limitation on processing Italian users' data by the Chinese companies providing DeepSeek and opened an investigation after finding the companies' response insufficient. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 91 | Korea PIPC DeepSeek status examination | Publisher: Korea PIPC DeepSeek status examination; Kind: Privacy enforcement; Checked: 2026-06-19; Status: monitor | E90: Korea's PIPC found DeepSeek had transferred user personal data to servers in China and the US without separate consent or adequate disclosure, and noted that new downloads had been temporarily suspended pending compliance improvements. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 92 | NIST CAISI DeepSeek evaluation | Publisher: NIST; Kind: Government evaluation; Checked: 2026-06-19; Status: monitor | E91: NIST CAISI reported that evaluated DeepSeek models showed security and censorship risks, including higher susceptibility to agent hijacking and jailbreaking than evaluated US reference models. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 93 | UK strategic export controls guidance | Publisher: UK strategic export controls guidance; Kind: Export control; Checked: 2026-06-19; Status: monitor | E92: UK strategic export controls guidance states that exports or transfers of goods, software, technology, data, information, and technical assistance may require licences and that exporting controlled items without the correct licence is a criminal offence. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 94 | UK National Security and Investment Act AI guidance | Publisher: UK National Security and Investment Act AI guidance; Kind: Regulatory guidance; Checked: 2026-06-19; Status: monitor | E93: UK National Security and Investment guidance treats AI as a sensitive sector and requires mandatory notification for acquisitions involving AI used for identification or tracking, advanced robotics, or cybersecurity. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 95 | OFAC FAQ 1188 - Russia IT and Software Services Determination | Publisher: OFAC FAQ 1188; Kind: Sanctions guidance; Checked: 2026-06-19; Status: monitor | E94: OFAC FAQ 1188 says the US Russia IT and Software Services Determination can prohibit indirect provision of covered services when the benefit is ultimately received by a person located in the Russian Federation. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 96 | BIS Entity List overview | Publisher: BIS Entity List overview; Kind: Sanctions export control; Checked: 2026-06-19; Status: monitor | E95: BIS states that Entity List parties create licence requirements for exports, reexports, and in-country transfers of covered items, so AI providers, affiliates, resellers, and end users need entity-list screening. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 97 | Google AI - Gemini API pricing | Publisher: Google; Kind: Pricing page; Checked: 2026-06-19; Status: monitor | E98: Google's Gemini API pricing lists Gemini 3.1 Pro Preview at $2 input / $12 output per 1M tokens for prompts up to 200K, $4 input / $18 output above 200K, and Gemini 3.1 Flash-Lite Preview at $0.25 input / $1.50 output. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 98 | Internet Archive CDX server | Publisher: Internet Archive CDX server; Kind: Archived web; Checked: 2026-06-19; Status: monitor | E101: Internet Archive CDX can monitor model, legal, privacy, pricing, and trust pages for changed claims, removed limits, and policy drift across model labs and legal AI vendors. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 99 | Common Crawl index API | Publisher: Common Crawl index API; Kind: Archived web; Checked: 2026-06-19; Status: monitor | E102: Common Crawl public index records provide dated evidence for public model-lab and legal-vendor surfaces when a page disappears or a current sitemap no longer exposes the old claim. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 100 | NVD CVE API | Publisher: NVD CVE API; Kind: Security advisory; Checked: 2026-06-19; Status: monitor | E103: NVD keyword searches expose security-advisory count signals for model-serving and agent-stack dependencies such as vLLM, Ray, Ollama, ChromaDB, LangChain, and LlamaIndex. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 101 | CISA known exploited vulnerabilities catalogue | Publisher: CISA; Kind: Security advisory; Checked: 2026-06-19; Status: monitor | E104: CISA's known exploited vulnerabilities catalogue gives model-routing teams a separate exploited-vulnerability screen for automation, notebook, browser, and Python dependencies that may sit around legal AI systems; broad keyword hits need human review for false positives. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 102 | RDAP domain lookup | Publisher: RDAP domain lookup; Kind: Domain registry; Checked: 2026-06-19; Status: monitor | E105: RDAP domain metadata exposes public domain-registration and nameserver signals for model labs and legal AI vendors without probing private infrastructure. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 103 | Cert Spotter certificate-transparency API | Publisher: Cert Spotter certificate-transparency API; Kind: Certificate transparency; Checked: 2026-06-19; Status: monitor | E106: Certificate-transparency issuance data exposes subdomain and certificate changes that can indicate new regional surfaces, trust portals, developer endpoints, or deployment environments. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 104 | NSF awards API | Publisher: NSF awards API; Kind: Grant database; Checked: 2026-06-19; Status: monitor | E107: NSF award search exposes pre-company research signals for AI agents, robot foundation models, autonomous laboratories, and data-centre AI that may become future model and workflow categories. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 105 | NIH RePORTER API | Publisher: NIH RePORTER API; Kind: Grant database; Checked: 2026-06-19; Status: monitor | E108: NIH RePORTER exposes health, clinical-workflow, surgical-robotics, and autonomous-lab AI project signals relevant to regulated legal work in healthcare and life sciences. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 106 | CORDIS project search | Publisher: CORDIS project search; Kind: Grant database; Checked: 2026-06-19; Status: monitor | E109: CORDIS project search adds a non-US frontier-research lens, but public project records can miss emerging AI-agent, robotics-AI, and autonomous-laboratory work when EU taxonomy and project language do not match buyer categories. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 107 | CourtListener REST API | Publisher: CourtListener REST API; Kind: Court docket; Checked: 2026-06-19; Status: monitor | E110: CourtListener can monitor AI hallucination, contract, and product-liability disputes that should feed model-risk evaluation and legal workflow approval gates. | E136: CourtListener searches surface recent AI copyright, legal research, and hallucination-sanctions disputes, including AI-hallucination sanctions results since January 1, 2025. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 108 | US Senate LDA API | Publisher: US Senate LDA API; Kind: Lobbying disclosure; Checked: 2026-06-19; Status: monitor | E111: US Senate LDA filings expose lobbying deltas that can signal regulatory, procurement, export-control, copyright, and liability pressure around model routes before formal rules settle. | E139: LDA lobbying disclosures expose AI-company and investor pressure around copyright, liability, procurement, export controls, and preemption before final rules settle. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 109 | Legal Benchmarks - about and core team | Publisher: Legal Benchmarks; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E112: Legal Benchmarks identifies Anna Guo as founder and describes her legal AI benchmarking work with 500+ legal professionals and procurement decisions. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 110 | Legal Benchmarks - legal AI evaluation framework | Publisher: Legal Benchmarks; Kind: Evaluation framework; Checked: 2026-06-19; Status: monitor | E113: Legal Benchmarks' evaluation framework documents a practitioner-built process for pre-demo, demo, and pilot scoring of legal AI tools. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 111 | LegalQuants - Jamie Tso profile | Publisher: LegalQuants; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E114: LegalQuants profiles Jamie Tso as a funds lawyer and founder building legal AI tools including redline, signature-packet, tabular-review, charting, redline-agent, and contract-simulation projects. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 112 | LegalQuants - Raymond Sun profile | Publisher: LegalQuants; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E115: LegalQuants profiles Raymond Sun as a tech lawyer and builder of a global AI regulation tracker with active legal-technology and AI-regulation tooling. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 113 | LegalQuants - Hou Fu Ang profile | Publisher: LegalQuants; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E116: LegalQuants profiles Hou Fu Ang as an in-house counsel, open-source legal tooling builder, and Alt + Counsel writer focused on practical legal AI and legal data infrastructure. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 114 | LegalQuants - Troy Zhaoyi Song profile | Publisher: LegalQuants; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E117: LegalQuants profiles Troy Zhaoyi Song as a disputes and technology lawyer building CaseThread and writing on legal AI data bottlenecks in English judgments. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 115 | Neel Guha - AI and law research profile | Publisher: Neel Guha; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E118: Neel Guha's profile lists LegalBench, a reasoning-focused legal retrieval benchmark, LegalBench-RAG-adjacent legal retrieval work, and AI-and-law research across ML and legal venues. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 116 | Stanford Report - legal judgment and AI | Publisher: Stanford Report; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E119: Stanford's legal AI coverage identifies Julian Nyarko, Daniel Ho, and Neel Guha as contributors to legal AI benchmarking, evaluation, legal-persona, and contract-risk research. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 117 | Stanford CodeX FutureLaw - Megan Ma profile | Publisher: Stanford CodeX FutureLaw; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E120: Stanford CodeX describes Megan Ma's work on generative AI in legal applications, legal knowledge to code, and human-machine collaboration in law. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 118 | Daniel Martin Katz - computational legal profile | Publisher: Daniel Martin Katz; Kind: Expert profile; Checked: 2026-06-19; Status: monitor | E121: Daniel Martin Katz's public profile and institutional pages identify work across legal informatics, legal NLP, machine learning, and applied legal technology. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 119 | Artificial Analysis - model comparison and Intelligence Index cost | Publisher: Artificial Analysis; Kind: Model benchmark; Checked: 2026-06-19; Status: monitor | E122: Artificial Analysis publishes model intelligence, token pricing, speed, latency, context, and the cost to run the Artificial Analysis Intelligence Index, making token price and benchmark bill separable cost measures. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 120 | FinOps Foundation - token economics | Publisher: FinOps Foundation; Kind: Cost benchmark; Checked: 2026-06-19; Status: monitor | E123: The FinOps Foundation frames token economics around total workload behaviour, including input and output tokens, latency, throughput, context reuse, provider route, reserved capacity, and the risk that self-hosted infrastructure is underused. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 121 | arXiv - price reversal in reasoning models | Publisher: arXiv; Kind: Cost benchmark; Checked: 2026-06-19; Status: monitor | E124: The price-reversal paper argues that reasoning models can look cheap by visible input/output token price while becoming more expensive on actual cost because hidden reasoning tokens and inference strategies change the accounting. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 122 | vLLM documentation | Publisher: vLLM; Kind: Inference infrastructure; Checked: 2026-06-19; Status: monitor | E125: vLLM documents high-throughput model serving features such as PagedAttention, continuous batching, prefix caching, chunked prefill, speculative decoding, and quantization, underscoring that self-hosted cost depends on serving efficiency as well as model licence. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 123 | Thomson Reuters CoCounsel Legal customer stories | Publisher: Thomson Reuters; Kind: Case study; Checked: 2026-06-19; Status: monitor | E126: Thomson Reuters' CoCounsel customer-story surface shows legal AI value being framed through productised legal workflows and named customer deployments, not through raw model identity alone. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 124 | Zarwin Baum responsible legal AI case study | Publisher: Zarwin Baum responsible legal AI case study; Kind: Case study; Checked: 2026-06-19; Status: monitor | E127: Zarwin Baum's CoCounsel adoption material describes document review, research, drafting, responsible adoption, confidentiality, client service, and lawyer oversight as the implementation frame. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 125 | Rupp Pfalzgraf Lexis+ AI case workload announcement | Publisher: Rupp Pfalzgraf Lexis+ AI case workload announcement; Kind: Case study; Checked: 2026-06-19; Status: monitor | E128: Rupp Pfalzgraf reports that its Lexis+ AI adoption coincided with a 10% increase in case workload, making throughput a measurable implementation signal rather than only a model-capability claim. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 126 | LexisNexis AI case studies | Publisher: LexisNexis; Kind: Case study; Checked: 2026-06-19; Status: monitor | E129: LexisNexis AI case-study pages package legal AI around adoption, workflow speed, usage, and matter output, which are the measures a model comparison must normalize before comparing providers. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 127 | Harvey and Reed Smith spotlight | Publisher: Harvey; Kind: Case study; Checked: 2026-06-19; Status: monitor | E130: Harvey's Reed Smith spotlight presents implementation as platform rollout, training, adoption, and matter-work enablement, showing that model capability becomes valuable only after it is wrapped in firm workflow. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 128 | Harvey senior law firm leaders on AI adoption | Publisher: Harvey; Kind: Case study; Checked: 2026-06-19; Status: monitor | E131: Harvey's senior law firm leader interviews emphasize leadership, workflow choice, training, and adoption moments; those factors explain legal AI outcomes that model benchmark rows cannot see. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 129 | A&O Shearman artificial intelligence practice and tools | Publisher: A&O Shearman artificial intelligence practice and tools; Kind: Case study; Checked: 2026-06-19; Status: monitor | E132: A&O Shearman's public AI page points to legal AI as firm-developed tooling and legal expertise embedded in regulated workflows, reinforcing that the model route has to fit practice-area governance. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 130 | CIPD LegalCo AI legal assistant case study | Publisher: CIPD LegalCo AI legal assistant case study; Kind: Case study; Checked: 2026-06-19; Status: monitor | E133: CIPD's LegalCo case study frames AI legal assistant implementation around internal experts, culture, adoption, and operating model, which are practical dependencies outside the generator model. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 131 | Harvard Center on the Legal Profession AI law firm interviews | Publisher: Harvard Center on the Legal Profession AI law firm interviews; Kind: Expert interview; Checked: 2026-06-19; Status: monitor | E134: Harvard CLP's law firm AI interviews surface productivity, business-model, staffing, governance, and funding questions that should be part of model and provider comparisons. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 132 | Reed Smith and Harvey public adoption video | Publisher: Reed Smith and Harvey public adoption video; Kind: Product demo; Checked: 2026-06-19; Status: monitor | E135: The Reed Smith and Harvey public discussion adds an adoption-mechanics signal: the platform is described as a primary firm AI route with training, user rollout, leadership, and workflow emphasis. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 133 | CourtListener RECAP - Thomson Reuters v ROSS summary judgment | Publisher: CourtListener RECAP; Kind: Court opinion; Checked: 2026-06-19; Status: monitor | E137: The February 11, 2025 Thomson Reuters v ROSS memorandum opinion is a direct legal-AI model-routing signal because source rights, fair-use assumptions, and competing-product use can change which legal data or retrieval route is acceptable. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 134 | Federal Register documents API | Publisher: Federal Register documents API; Kind: Regulatory registry; Checked: 2026-06-19; Status: monitor | E138: Federal Register document searches expose regulatory surfaces for artificial intelligence, AI models, algorithmic discrimination, and state artificial intelligence law. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 135 | GDELT 2.0 Doc API | Publisher: GDELT 2.0 Doc API; Kind: News index; Checked: 2026-06-19; Status: monitor | E140: GDELT can provide directional news-volume context for AI politics and legal salience, but model-route claims still need a primary legal, regulatory, or official source behind them. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 136 | FEC dates and deadlines | Publisher: FEC dates and deadlines; Kind: Election calendar; Checked: 2026-06-19; Status: monitor | E141: The FEC dates-and-deadlines page is the official federal campaign-reporting and election-calendar entrypoint, making model-route risk analysis sensitive to election timing. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 137 | FEC 2026 congressional primary dates | Publisher: FEC 2026 congressional primary dates; Kind: Election calendar; Checked: 2026-06-19; Status: monitor | E142: The FEC 2026 congressional primary calendar states the general election date as November 3, 2026, lists state primary and filing dates, and notes that dates are subject to change. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 138 | White House - national AI legislative framework | Publisher: White House; Kind: Policy framework; Checked: 2026-06-19; Status: monitor | E143: The White House March 20, 2026 national AI legislative framework frames AI policy around children, communities, intellectual property, speech, innovation, workforce, and uniform federal policy over a patchwork of state laws. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 139 | House Democratic Commission on AI and the Innovation Economy | Publisher: House Democratic Commission on AI and the Innovation Economy; Kind: Party policy; Checked: 2026-06-19; Status: monitor | E144: The House Democratic Commission on AI and the Innovation Economy will convene throughout 2026 and frames party posture around innovation, privacy, safety, guardrails, competitiveness, and worker or public-welfare concerns. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 140 | RealClearPolling - 2026 generic congressional vote | Publisher: RealClearPolling; Kind: Polling data; Checked: 2026-06-19; Status: monitor | E145: RealClearPolling's generic congressional vote page is a public political-environment surface for tracking election sentiment that may affect AI hearings, enforcement, procurement, and preemption windows. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 141 | FiveThirtyEight/DataHub generic congressional ballot historical CSV | Publisher: FiveThirtyEight/DataHub generic congressional ballot historical CSV; Kind: Polling data; Checked: 2026-06-19; Status: monitor | E146: Historical FiveThirtyEight/DataHub generic-ballot data is useful for methodology context, but current 2026 polling claims need a live polling source or aggregator-methodology check. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 142 | AI Act Service Desk - implementation timeline | Publisher: AI Act Service Desk; Kind: Regulatory timeline; Checked: 2026-06-19; Status: verified | E212: The AI Act Service Desk implementation timeline lists 2 August 2026 as the start of Commission enforcement powers for GPAI providers, making GPAI documentation, copyright-policy and incident-reporting evidence a dated model-route trigger. | Open source |
| 143 | European Commission - high-risk AI classification consultation | Publisher: European Commission; Kind: Regulatory consultation; Checked: 2026-06-19; Status: verified | E213: The European Commission targeted consultation on draft high-risk AI classification guidelines closes on 23 June 2026, making high-risk use-case classification a near-term legal-model route trigger. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 144 | IMDA - Legal Responsibility for AI Agents discussion paper (May 2026) | Publisher: IMDA; Kind: Regulator; Checked: 2026-06-19; Status: monitor | E147: Model developers are usually best placed to shape an agent's base behaviour and safety properties, while deployers have more visibility into the specific use case but less ability to intervene in base model behaviour. The asymmetry maps directly onto model-route decisions across direct API, cloud, legal SaaS, and self-hosted routes. | E148: Proving fault for non-deterministic models is hard because review must cover training data, fine-tuning provenance, and architecture, not just code. Chain-of-thought outputs are statistical natural-language explanations, not faithful traces of internal decision-making. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 145 | Singapore MinLaw - Guide for Using Generative AI in the Legal Sector (6 March 2026) | Publisher: Singapore MinLaw; Kind: Regulator; Checked: 2026-06-19; Status: monitor | E149: A lifecycle map of legal work identifies augmentable tasks across intake, research, drafting, review, contract analysis, due diligence, and end-of-matter; the map gives a concrete frame for which model routes can serve which lifecycle stage. | E150: A tiered tool ladder selected by confidentiality need should exclude consumer or personal AI accounts from legal-work routes; approved enterprise routes start with contracted commercial tools and move to in-house or controlled deployments when privilege, regulation, sovereignty, or audit needs require it. | E151: Retrieval-augmented generation is the practical grounding mechanism for reducing legal hallucination; direct citations and confidence scores are the trust-building features that make verification feasible. | E152: Two human-oversight modes apply across legal work: human-in-the-loop for tasks requiring lawyer judgement, and human-on-the-loop for reversible low-risk tasks. Mode selection is driven by internal versus external use and by legal, reputational, or financial impact. | E153: Bias in legal AI compounds across training data, algorithmic design, and feedback loops; mitigation needs cross-group testing, requested model reasoning, and vendor evidence of bias measurement, not a single disclosure line. | E154: Auditability (traceable input and output logs) and explainability (confidence scores, source citations, reasoning) are the two trust-building features procurement should test for at the tool layer. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 146 | ISO/IEC 42001:2023 - Artificial intelligence management system | Publisher: ISO/IEC; Kind: International standard; Checked: 2026-06-19; Status: monitor | E155: ISO/IEC 42001:2023 is the first international management-system standard for AI within organisations; certification gives a vendor-checkable signal of documented governance, risk treatment, lifecycle controls, monitoring, and continuous improvement for an AI provider's systems. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 147 | OpenAI - API pricing | Publisher: OpenAI; Kind: Pricing page; Checked: 2026-06-19; Status: monitor | E156: OpenAI publishes list prices per million input and output tokens for every API model, including GPT-5, GPT-5 mini, GPT-5 nano, GPT-4.1 family, o3, o4-mini, and the legacy GPT-4o, GPT-4 Turbo, and GPT-3.5 Turbo SKUs; the page is the canonical reference for blended-token computations. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 148 | Anthropic - API pricing | Publisher: Anthropic; Kind: Pricing page; Checked: 2026-06-19; Status: monitor | E157: Anthropic publishes list prices per million input and output tokens for Claude Opus, Sonnet, and Haiku families across release generations; the page is the canonical reference for any Claude-route blended-token cost comparison. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 149 | a16z - Welcome to LLMflation: LLM inference cost is going down fast | Publisher: a16z; Kind: Industry analysis; Checked: 2026-06-19; Status: monitor | E158: a16z's LLMflation analysis introduces the cumulative intelligence-tier visualisation method: plot the cheapest available model that falls within each intelligence band (<10, <20, <30, <40, <50, ≥50) over time. Artificial Analysis publishes per-model intelligence scores but does not bucket them into tiers; the tier-band framing is a16z's analytical contribution. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 150 | Harvey support - model comparison and BigLaw Bench scores | Publisher: Harvey; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E159: Harvey's model comparison page, last updated 21 May 2026, publishes vendor-reported BigLaw Bench scores for the model routes Harvey exposes, including GPT-5.5, GPT-5.4, GPT-5.2, GPT-5.1, GPT-5 reasoning, GPT-4.1, Gemini 2.5 Pro, Claude Sonnet 4.x, and Claude Opus 4.x routes. | E228: Harvey's BigLaw Bench support page rows used with the Opus 4.8 update show GPT-5.1 at 91.8, GPT-5.5 at 91.7, GPT-5.4 at 91.0, Claude Opus 4.6 at 90.2, and GPT-5.2 at 89.8 on the vendor-reported route benchmark. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 151 | LegalBenchmarks.ai - legal task leaderboard | Publisher: LegalBenchmarks.ai; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E160: LegalBenchmarks.ai's May 2026 leaderboard tests six leading models on 65 legal tasks built from lawyer/community contributions and expert-validated synthetic cases, split into 34 contract-drafting tasks and 31 information-extraction tasks, and reports reliability separately from usefulness. | E206: On LegalBenchmarks.ai's May 2026 contract-drafting leaderboard, Claude Sonnet 4.6 leads at 55.9% reliability (19/34), Claude Opus 4.7 and Gemini 3.1 Pro tie at 52.9% (18/34), and no raw model clears 56% under the all-pass task standard. | E207: LegalBenchmarks.ai's drafting usefulness scores split from reliability: GPT-5.5 has the highest usefulness score at 2.82 but only 35.3% drafting reliability, while the more reliable Sonnet 4.6 and Opus 4.7 rows have the lowest length sub-scores. | E208: LegalBenchmarks.ai's app-versus-API check on the nine hardest drafting tasks found Claude app at 55.6% versus 44.4% for Opus 4.7 API, ChatGPT app at 22.2% versus 0.0% for GPT-5.5 API, and Gemini app at 22.2% versus 33.3% for Gemini 3.1 Pro API. | E209: On LegalBenchmarks.ai's 31-task information-extraction leaderboard, Claude Opus 4.7 and GPT-5.5 tie at 80.6% reliability (25/31), followed by Claude Sonnet 4.6 at 74.2% (23/31), Gemini 3.1 Pro at 64.5% (20/31), Gemini 3 Flash at 41.9% (13/31), and GPT-4o-mini at 16.1% (5/31). | Open source |
| 152 | Voyage AI - voyage-law-2 legal embeddings | Publisher: Voyage AI; Kind: Vendor documentation; Checked: 2026-06-19; Status: monitor | E161: Voyage AI's 2024 voyage-law-2 release framed legal retrieval as a domain-specific embedding problem, reporting that voyage-law-2 led its legal retrieval comparison with a 65.39 average NDCG@10 across eight legal retrieval datasets. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 153 | Voyage AI - voyage-3.5 embeddings | Publisher: Voyage AI; Kind: Model documentation; Checked: 2026-06-19; Status: monitor | E162: Voyage AI's 2025 voyage-3.5 release framed the newer voyage-3.5 and voyage-3.5-lite models as general embedding models evaluated across 100 datasets spanning technical documentation, code, law, finance, web reviews, multilingual, long-document, and conversation domains. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 154 | COLIEE 2026 - task results | Publisher: COLIEE 2026; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E164: COLIEE 2026 publishes task-level team-run results for legal case retrieval and Japanese statute-law tasks; those rows are competition-system results, not model-route leaderboards. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 155 | NeurIPS - Retrieval-Augmented Generation | Publisher: NeurIPS; Kind: Research paper; Checked: 2026-06-19; Status: monitor | E165: The original RAG paper defines retrieval-augmented generation as combining a pretrained generator with non-parametric external memory accessed through a retriever, improving knowledge-intensive generation and source updateability. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 156 | TACL - Lost in the Middle | Publisher: TACL; Kind: Research paper; Checked: 2026-06-19; Status: monitor | E166: Lost in the Middle finds that long-context models often use information best at the beginning or end of context, degrade when relevant information sits in the middle, and can saturate before retriever recall is exhausted. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 157 | RULER long-context benchmark | Publisher: RULER long-context benchmark; Kind: Research paper; Checked: 2026-06-19; Status: monitor | E167: RULER extends needle-in-a-haystack evaluation into configurable long-context tasks covering retrieval, multi-hop tracing, aggregation, and question answering, because simple single-needle tests overstate usable context capacity. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 163 | OpenAI status page | Publisher: OpenAI status page; Kind: Status page; Checked: 2026-06-19; Status: monitor | E173: OpenAI's status page showed APIs at 99.98% uptime for its Feb 2026-May 2026 published window on 27 May 2026; its incident feed showed the busiest API/model-relevant seven-day cluster at about 36.1 hours during 13-20 May 2026. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 164 | Claude status page | Publisher: Claude status page; Kind: Status page; Checked: 2026-06-19; Status: monitor | E174: Claude's status page showed Claude API at 99.07% uptime over the past 90 days on 27 May 2026; its incident feed showed the busiest model/API-relevant seven-day cluster at about 13.7 hours during 16-23 May 2026. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 165 | Microsoft Azure OpenAI Service | Publisher: Microsoft; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E175: Microsoft's Azure OpenAI Service page states that Azure OpenAI is guaranteed to be available at least 99.9% of the time. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 166 | Google Cloud Service Health JSON history | Publisher: Google; Kind: Status page; Checked: 2026-06-19; Status: monitor | E176: Google Cloud Service Health records a Vertex AI Gemini API incident on 27 February 2026 in which global-endpoint customers experienced increased error rates for 1 hour and 58 minutes. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 167 | LegalEvalHub - LegalBench full leaderboard | Publisher: LegalEvalHub; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E177: LegalEvalHub's LegalBench Full leaderboard aggregates performance across all 161 LegalBench tasks. Its selected task rows for meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo round to 100%, 99%, 98%, 98%, 94%, 93%, 90%, 88%, 87%, and 82%. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 169 | Azure status history - Azure OpenAI Service | Publisher: Azure; Kind: Status page; Checked: 2026-06-19; Status: monitor | E179: Azure status history records an Azure OpenAI GPT-5.2 availability degradation across seven regions from 23:20 UTC on 9 March to 19:32 UTC on 10 March 2026, a 20.2-hour incident window. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 170 | Vertex AI Platform Service Level Agreement | Publisher: Vertex AI Platform Service Level Agreement; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E180: The Vertex AI Platform SLA lists monthly uptime percentage commitments including at least 99.9% for covered training, deployment, and batch-prediction services. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 171 | OpenAI GPT-5.5 model rate limits | Publisher: OpenAI GPT-5.5 model rate limits; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E181: OpenAI's GPT-5.5 model page publishes hard usage-tier limits, including Tier 1 at 500 RPM and 500K TPM and Tier 5 at 15K RPM and 40M TPM; the live usage tier, long-context cap and shared-family cap still need account verification. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 172 | OpenAI Batch API | Publisher: OpenAI Batch API; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E182: OpenAI's Batch API gives asynchronous jobs a separate pool with higher rate-limit headroom, lower cost and a 24-hour completion target, making it a different lane from interactive legal work. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 173 | Claude API rate limits | Publisher: Claude API rate limits; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E183: Claude API limits are measured as RPM, ITPM and OTPM by model class; Claude Opus 4.x publishes Tier 1 at 50 RPM, 500K ITPM and 80K OTPM and Tier 4 at 4K RPM, 10M ITPM and 800K OTPM, with custom and Priority Tier increases handled separately. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 174 | Groq API rate limits | Publisher: Groq; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E184: Groq's API rate-limit page defines organization-level RPM, RPD, TPM, TPD, ITPM and OTPM controls, publishes plan and model limit tables, and says higher limits are available for select workloads and enterprise use cases; it does not expose one public production TPM ceiling suitable for the tier-wait chart. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 175 | Azure OpenAI quotas and limits | Publisher: Azure; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E185: Azure OpenAI quotas are defined by model, deployment type, subscription and region; the public gpt-5.5 GlobalStandard rows list 0 RPM and 0 TPM through lower tiers, then 10K RPM and 10M TPM at Tier 5 and 15K RPM and 15M TPM at Tier 6. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 176 | Azure OpenAI provisioned throughput | Publisher: Azure; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E186: Azure provisioned throughput separates PTU quota from actual regional capacity, supports Global, Data Zone and Regional provisioned deployment types, and adds spillover plus hourly or reservation billing options. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 177 | Vertex AI Standard PayGo throughput | Publisher: Vertex AI Standard PayGo throughput; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E187: Vertex AI Standard PayGo uses spend-based throughput tiers: GA Gemini Pro publishes 500K TPM at Tier 1 and 2M TPM at Tier 3, Gemini Flash and Flash-Lite publish 2M TPM at Tier 1 and 10M TPM at Tier 3, and a 30K RPM per-model per-region system limit applies outside the tier-specific TPM baseline. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 178 | Vertex AI Provisioned Throughput | Publisher: Vertex AI Provisioned Throughput; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E188: Vertex AI Provisioned Throughput is a dedicated-capacity route with request-type controls and monitoring, distinct from shared PayGo throughput. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 179 | Amazon Bedrock quota increase mechanism | Publisher: Amazon; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E189: Amazon Bedrock does not publish one comparable default public TPM row for every model route; AWS documents model-specific on-demand and cross-Region InvokeModel token-per-minute and request-per-minute quota families that must be checked in Service Quotas and increased by model. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 180 | Amazon Bedrock cross-Region inference | Publisher: Amazon; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E190: Amazon Bedrock cross-Region inference can route on-demand requests across regions to increase throughput during quota or peak-usage constraints, while Provisioned Throughput is the separate fixed-capacity route for higher model invocation capacity. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 181 | Amazon Bedrock Provisioned Throughput | Publisher: Amazon; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E191: Amazon Bedrock Provisioned Throughput is purchased for higher fixed throughput, with Model Units specifying input and output tokens per minute and commitment terms ranging from no commitment to one or six months. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 182 | OpenAI Flex processing | Publisher: OpenAI Flex processing; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E192: OpenAI Flex processing is a lower-priority processing lane for supported models and is charged at Batch API rates, making it a price and latency trade-off rather than an interactive default. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 183 | OpenAI prompt caching | Publisher: OpenAI prompt caching; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E193: OpenAI prompt caching automatically discounts repeated prompt prefixes above the documented threshold, exposes cached-token usage and can use prompt-cache keys to improve cache routing for related requests. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 184 | Anthropic Message Batches | Publisher: Anthropic; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E194: Anthropic Message Batches support high-volume asynchronous Claude requests with discounted pricing and can be combined with prompt caching for repeated context. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 185 | Anthropic prompt caching | Publisher: Anthropic; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E195: Anthropic prompt caching uses explicit cache-control breakpoints across tools, system and message content, with separate cache write and read economics and documented cache durations. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 186 | Vertex AI generative AI pricing | Publisher: Vertex AI generative AI pricing; Kind: Pricing page; Checked: 2026-06-19; Status: monitor | E196: Vertex AI generative AI pricing separates standard input and output tokens from batch-mode pricing and cached-input pricing, so Gemini route cost depends on lane as well as model. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 187 | Vertex AI context caching | Publisher: Vertex AI context caching; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E197: Vertex AI context caching supports explicit cached content and implicit caching, with cache duration, minimum-token and exact-prefix behaviours that affect whether repeated source packets earn the cached-input price. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 188 | Vertex AI Gemini batch prediction | Publisher: Vertex AI Gemini batch prediction; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E198: Vertex AI Gemini batch prediction is a non-real-time route for bulk jobs; Google documents batch mode separately from provisioned throughput and online inference. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 189 | Amazon Bedrock pricing | Publisher: Amazon; Kind: Pricing page; Checked: 2026-06-19; Status: monitor | E199: Amazon Bedrock pricing separates on-demand, batch, prompt-caching and provisioned-throughput economics across supported models, so the approved AWS lane changes the bill. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 190 | Amazon Bedrock prompt caching | Publisher: Amazon; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E200: Amazon Bedrock prompt caching uses cache points with TTL controls and model support limits, reducing repeated-context processing where the route is eligible. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 191 | Amazon Bedrock batch inference | Publisher: Amazon; Kind: Cloud documentation; Checked: 2026-06-19; Status: monitor | E201: Amazon Bedrock batch inference runs asynchronous jobs from S3 input to S3 output for supported models, making it a separate bulk lane from synchronous inference. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 192 | Groq prompt caching | Publisher: Groq; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E202: Groq prompt caching automatically detects repeated exact prefixes above the documented threshold, gives discounted cached-token treatment and documents rate-limit effects for cache hits. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 193 | Groq Batch API | Publisher: Groq; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E203: Groq Batch API is an asynchronous discounted lane that does not consume standard synchronous rate limits, but completion windows and batch limits remain part of the production route. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 194 | Groq Flex processing | Publisher: Groq; Kind: Official model documentation; Checked: 2026-06-19; Status: monitor | E204: Groq Flex processing lets requests wait for available capacity to raise effective throughput without changing regular API pricing, so it solves a different problem from batch discounting. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 195 | Harvey - LAB initial results | Publisher: Harvey; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E205: Before the Opus 4.8 update, Harvey's initial LAB holdout results reported strict all-pass scores below 10% for the evaluated frontier legal-agent routes, showed different model families leading different practice-area groupings, and put the highest all-pass route at about $50.90 per task with roughly 22 minutes of wall-clock latency. | E229: Harvey's initial LAB holdout rows used with the Opus 4.8 update show Claude Opus 4.7 at 7.1% all-pass, Claude Sonnet 4.6 at 5.4%, Claude Opus 4.6 at 4.2%, GPT-5.5 at 2.1%, and Gemini 3.5 Flash at 0.8%. | Open source |
| 196 | Artificial Analysis - Claude Opus 4.8 analysis and benchmarks | Publisher: Artificial Analysis; Kind: Model benchmark; Checked: 2026-06-19; Status: verified | E210: Artificial Analysis' 28 May 2026 analysis reports Claude Opus 4.8 at 61.4 on the Artificial Analysis Intelligence Index, up 4.1 points from Opus 4.7, with GDPval-AA at 1,890 Elo, a 1M-token context window, and unchanged $5/$25 per 1M input/output token pricing. | Open source |
| 197 | Harvey - Claude Opus 4.8 now live in Harvey | Publisher: Harvey; Kind: Legal benchmark; Checked: 2026-06-19; Status: verified | E211: Harvey's 28 May 2026 Opus 4.8 note reports 10.4% on the Legal Agent Benchmark all-pass standard, up from 7.1% for Opus 4.7, and 91.1% on BigLaw Bench, a new Claude-family high inside Harvey's vendor-reported benchmark. | Open source |
| 198 | Surge AI - GDP.pdf leaderboard | Publisher: Surge AI; Kind: Professional benchmark; Checked: 2026-06-19; Status: verified | E214: Surge's GDP.pdf leaderboard, last updated 6 June 2026, reports macro accuracy on professional PDF question answering; current visible rows include Claude Fable 5 / Mythos 5 at 30%, GPT-5.5 at 25%, Claude Opus 4.8 at 23%, Claude Opus 4.7 at 21%, and Gemini 3.1 Pro at 17%. | Open source |
| 199 | Surge AI - GDP.pdf benchmark blog | Publisher: Surge AI; Kind: Professional benchmark; Checked: 2026-06-19; Status: verified | E215: Surge describes GDP.pdf as 100 real-world prompts and PDFs across finance, healthcare, legal, STEM/research, engineering, construction, manufacturing/supply chain, insurance, real estate and HR, with tasks requiring parsing, understanding and synthesizing complex PDFs. | Open source |
| 200 | Surge AI - GDP.pdf dataset | Publisher: Surge AI; Kind: Dataset; Checked: 2026-06-19; Status: verified | E216: The GDP.pdf dataset is public on Hugging Face with 100 test rows, document question-answering metadata, PDF paths, prompts, domains and rubric fields. | Open source |
| 201 | Reducto - GDP.pdf parser rerun | Publisher: Reducto; Kind: Vendor research; Checked: 2026-06-19; Status: monitor | E217: Reducto's vendor-reported GDP.pdf rerun compared PDF-only inputs with PDF plus Reducto parse output over the same 100 documents and reported aggregate macro accuracy rising from 21.7% to 30.7%, micro accuracy rising from 75.6% to 77.9%, answer latency falling from 120s to 111s, reasoning tokens falling from 8.8K to 7.7K per task, and legal-domain macro accuracy rising by 9.5 percentage points. Limitation: verify dated release or consultation status before relying on timing. | Open source |
| 202 | COLIEE 2026 - task 3 results | Publisher: COLIEE 2026; Kind: Legal benchmark; Checked: 2026-06-19; Status: monitor | E227: COLIEE 2026 Task 3 statute-law QA results used in the grid show NOWJ run1 at 0.9512, JNLP JNLP1 at 0.9024, NOWJ run2 and NOWJ run3 at 0.8902, and cody codyrag at 0.8537. Limitation: verify dated release or consultation status before relying on timing. | Open source |
Glossary
Terms used in this report
| Term | Plain meaning |
|---|---|
| Route | The approved path from work to accepted output: source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger. |
| API | Application programming interface: a governed software connection used by an app or workflow to call a model or service. |
| SDK | Software development kit: packaged code and tools developers use to connect a product to an API or service. |
| LLM | Large language model: a model trained to read and generate text-like tokens. |
| Token | A chunk of text used for billing, context windows and rate limits. It is usually smaller than a word, not a page count. |
| RPM | Requests per minute: how many separate API calls the route can accept each minute. |
| RPD | Requests per day: the daily cap on separate API calls. |
| TPM | Tokens per minute: how much token traffic a route can process each minute, unless the provider splits input and output separately. |
| TPD | Tokens per day: the daily cap on token traffic. |
| ITPM | Input tokens per minute: how many prompt and source-document tokens may enter the model each minute. |
| OTPM | Output tokens per minute: how many generated tokens may leave the model each minute. |
| PTU | Provisioned throughput unit: reserved Azure OpenAI capacity bought for steadier throughput. |
| GSU | Google scale unit: provisioned Vertex AI capacity used to reserve Gemini throughput. |
| MU | Model Unit: Amazon Bedrock provisioned-throughput capacity reserved for a specific model route. |
| PayGo | Pay as you go: shared cloud capacity billed by usage rather than pre-reserved throughput. |
| RAG | Retrieval-augmented generation: search approved sources first, then pass selected evidence into the model. |
| OCR | Optical character recognition: converting scanned or image text into machine-readable text. |
| VDR | Virtual data room: a permissioned deal, diligence or transaction document repository. |
| SLA | Service-level agreement: a contractual availability or support commitment, not observed uptime history. |
| JSON | JavaScript Object Notation: a structured text format commonly used for machine-readable API output. |
| 429 | HTTP 429 means too many requests: the route hit a rate limit or shared-capacity throttle. |
Definitions are reader aids, not separate evidence claims.
