Verified intelligence · Expert-reviewed from public records, platform data, primary sources, and private research beyond AI search.

INFRACross-industryGLOBAL

Deployment options for open-weight LLM models

Open-weight LLM deployment is an infrastructure decision across licence, data boundary, serving stack, utilisation, latency, retrieval, controls, recovery and accepted-output cost.

The colophon

Landscape reviewed: 2026-06-08
Latest revision: 2026-06-08
Kinetics: Open-weight deployment has shifted from a model-download question to an approved deployment-path question spanning cloud catalogues, serverless providers, private GPU fleets, RAG systems and gateway controls. Serving engines, provider catalogues, licence terms, GPU prices and governance standards are changing quickly enough that deployment approval needs versioned evidence and reapproval triggers.
To watch: 2 August 2026: European Commission GPAI enforcement powers enter into application; European open-model deployments may need stronger provider-documentation and systemic-risk evidence ³¹.
Meta Llama licence terms add, remove or tighten acceptable-use or scale restrictions; approved open-weight deployment paths may need legal reroute ².
Artificial Analysis provider leaderboard, model-price and model-release updates; managed API speed, latency, intelligence or price shifts can change first-deployment economics ³⁷⁷³.
AWS EC2 Capacity Blocks pricing changes p5.48xlarge commitment price, duration or region availability; H100 break-even may move ⁴¹⁶¹.
vLLM release notes; cache, LoRA, speculative-decoding, quantization or security changes can force deployment reapproval.
Known unknowns: Comparable public evidence is still thin for end-to-end accepted-output cost across the same open model, workload, serving engine, cloud deployment path, concurrency profile, RAG stack, human-verification burden and recovery target.

Contents

Executive summary
Decision frame
Owner decision lanes
Deployment economics
Scenario economics
Deployment options
Deployment atlas
Speed-first inference
Serving stack
Workload deployment paths
Benchmarks and private evals
Risks and controls
Licence and regulatory gates
Procurement diligence
Implementation plan
Deployment case studies
Known unknowns
Vendor and stack directory
Source ledger
Glossary

▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒Executive summary

The buyer is not choosing between open-source and cloud. It is choosing how a workload may cross a boundary, spend a GPU hour, find context, fail over and prove what happened. Start with the work. A public chatbot, internal analyst copilot, regulated-document assistant, batch extraction system, tenant-specific SaaS feature and field-device workflow should not share one approval. Each has a different data boundary, latency profile, context length, recovery target and audit burden.

The phrase open-source is often too loose for procurement. The Open Source AI Definition requires enough information to use, study, modify and share the system; many popular models are better described as open-weight because the licence, training data, acceptable-use terms or derivative rights are not the same as a normal open-source software dependency ¹².

The default sequence is pragmatic. Use managed APIs for prototypes and bursty workloads. Use cloud model catalogues when the enterprise already trusts the cloud control plane. Use dedicated managed or self-hosted GPUs when volume, latency, custom weights or data boundaries justify the operating burden. Use private cloud, colocation or edge only when the boundary is worth the slower procurement ³⁹.

The winning number is cost per accepted output: the model bill plus serving, retries, vector search, review minutes, security, patching, observability and fallback capacity. Cheap token pricing can still be expensive work.

Board-level deployment decision

Question	Answer	Buyer move
Which deployment option wins?	The deployment path that fits the workload's data boundary, latency, utilisation and proof burden.	Approve deployment paths by workload, not by model family.
Where should a first deployment start?	Managed APIs, serverless open-model providers or the existing cloud model catalogue.	Buy speed until traffic, privacy or customisation proves the need for dedicated infrastructure.
When does self-hosting win?	High steady utilisation, custom serving control, restricted data or a clear latency requirement.	Build only after a GPU utilisation and accepted-output cost model beats managed options.
What breaks production first?	Licence ambiguity, weak retrieval, low GPU utilisation, poor p95 latency, hidden fallback, missing logs and unrecovered vector state.	Test the whole deployment path: model, serving engine, RAG, gateway, observability, security and recovery.

Deployment decision tree

Classify the data and workload before comparing model providers.

Can prompts, files, outputs and logs leave the buyer's approved boundary?
No
Start with private cloud, on-prem, colocation or edge.
Yes
Continue to control-plane fit.
Next: Q2 control plane
Does an existing cloud control plane already cover IAM, network, audit, procurement and region?
Yes
Start with managed cloud endpoints or model catalogue deployments.
No
Use serverless APIs for pilots, then prove whether a platform team is needed.
Is traffic steady enough to keep dedicated GPUs busy?
Yes
Compare dedicated managed deployments with self-hosted GPUs.
No
Variable APIs usually beat idle infrastructure.
Does the deployment need custom weights, LoRA adapters, RAG isolation or hard latency guarantees?
Yes
Test serving engine, adapter lineage, vector tenancy and p95 latency before approval.
No
Use the least operationally ambitious deployment path that clears data terms and evals.

The deployment object is a controlled production path, not a model binary. Model deployment has four moving parts: rights, runtime, context and proof. Rights decide whether the model can be used and modified. Runtime decides where and how it serves. Context decides which sources it can see. Proof decides whether security, compliance and the business can trust the deployment after the next model, adapter or provider change.

The buyer's first register should therefore name the deployment path, not merely the model: licence, data class, endpoint, serving engine, vector store, gateway policy, logging standard, reviewer, recovery target and reapproval trigger.

Cloud approval does not approve every use case. A deployment inside Azure, AWS, Google Cloud or Databricks can still be wrong for customer data, regulated outputs, cross-border records or external communications.

The practical test is whether the deployment path can be explained to the person who owns the failure.

Minimum deployment register

Field	Bad entry	Approved entry	Why it matters
Model status	Open-source Llama.	Llama-family weight under reviewed licence, approved for internal summarization.	Licence and acceptable-use terms can block deployment before runtime does.
Data class	Internal data.	Public docs, confidential employee data, regulated customer data or export-controlled content.	Retention, region, support access and fallback policy change by class.
Serving path	GPU endpoint.	Azure managed endpoint, Together serverless API, vLLM on Kubernetes or on-device runtime.	The serving path determines control plane, logs, region, latency and recovery.
Context layer	RAG.	Embedding model, vector store, tenant namespace, filter policy and rebuild trigger.	Retrieval mistakes are deployment failures, not prompt failures.
Fallback	Use another provider.	Allow-listed fallback path by data class and benchmark threshold.	Fallback can silently change data terms and answer quality.
Reapproval trigger	Annual review.	Model version, licence, serving engine, adapter, region, vector schema, threshold or provider term change.	Open-model deployments move too quickly for calendar-only governance.

Price the accepted output, not the token or the GPU hour. The GPU line item is not deployment cost. Production cost is what the system spends to produce accepted work under the required data boundary, latency, audit and recovery constraints. Token price matters, but it is only one line. Open-model deployment also pays for idle GPUs, serving engineers, cache misses, long prompts, vector search, retries, rejected answers, review minutes, monitoring, patching, support and failover.

GPU-hour price matters, but it is also only a line. A cheap H100 that sits idle, lacks private networking, cannot meet p95 latency or needs manual recovery can lose to a pricier managed endpoint that clears the whole acceptance test.

Public infrastructure price rows are only inputs: GPU hour, endpoint commitment, provisioned throughput, platform fee, region, support and capacity window. The denominator that matters is accepted output: a correct answer, classified document, extracted field, completed support response, reviewed agent action or refreshed knowledge-base packet ³⁹⁴⁰.

Self-hosting wins when utilisation is high and operations are funded. Managed APIs win when use is bursty, uncertain or early. Dedicated managed deployments sit in the middle: stronger isolation and performance control than serverless, less operating burden than a self-run cluster.

The procurement mistake is to compare a cloud list price with a self-hosted GPU line and call the cheaper row the answer. Compare the full deployment, not the cheapest visible line item.

A cheap GPU is not a production system

Public price pages checked 28 May 2026. Outlined rows are visible GPU rental or on-demand input prices; the filled row is a committed enterprise cloud deployment path. Rows exclude storage, networking, reserved terms, idle replicas, support, serving engineering, security, monitoring and review burden. Use the chart as an input to the accepted-output model, not as a procurement ranking. ³⁹⁴⁰⁴¹

The GPU chart supplies input prices. The table converts those inputs into accepted-output economics by adding utilisation, hidden operating work, review burden, rejected runs and the deployment boundary each path must satisfy.

How to price each deployment path

Deployment path	Visible unit	Hidden cost	Accepted-output test
Serverless open-model API	Tokens, requests or provider plan	Rate limits, retries, provider queues, gateway fees and review minutes.	Cost per correct response at p95 latency and approved data terms.
Managed cloud endpoint	Endpoint, tokens, compute or provisioned throughput	Cloud service overhead, region constraints, endpoint idle time, logging and security controls.	Cost per accepted answer inside the existing cloud control plane.
Dedicated managed deployment	Reserved GPU, replica or throughput commitment	Idle capacity, autoscaling limits, quota and failover capacity.	Cost per accepted output at planned concurrency and fallback policy.
Self-hosted cloud GPU	GPU hour	Serving engineers, patching, observability, SRE, orchestration, storage, networking and incident response.	Cost per accepted output at realistic utilisation, including rejected runs.
Private cloud, colocation or on-prem	Capex, managed infrastructure or subscription	Power, cooling, supply chain, upgrades, support, DR and slower elasticity.	Cost per accepted output where data boundary or sovereignty justifies the burden.
Edge or on-device	Device capacity and update cost	Quantization loss, telemetry gaps, fleet updates, support and weaker fallback.	Accepted local answer under offline constraints and recoverable update path.

Terms used in this report

Term	Plain meaning
Route	The approved path from work to accepted output: source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger.
API	Application programming interface: a governed software connection used by an app or workflow to call a model or service.
LLM	Large language model: a model trained to read and generate text-like tokens.
Token	A chunk of text used for billing, context windows and rate limits. It is usually smaller than a word, not a page count.
RAG	Retrieval-augmented generation: search approved sources first, then pass selected evidence into the model.
SLA	Service-level agreement: a contractual availability or support commitment, not observed uptime history.
SLO	Service-level objective: a service target that may be published or internal, and may not be contractual.
JSON	JavaScript Object Notation: a structured text format commonly used for machine-readable API output.

Definitions are reader aids, not separate evidence claims.

MCP PREVIEW INVENTORY

What the locked report contains

Open-weight LLM deployment is an infrastructure decision across licence, data boundary, serving stack, utilisation, latency, retrieval, controls, recovery and accepted-output cost.

Schema endpoints

Sample object: /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/sample
Inventory object: /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/inventory
Schema contract: /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/schema

Coverage fields

Category: INFRA
Sector: Cross-industry
Region: GLOBAL
Topics: open-source LLMs, open-weight models, LLM inference, model serving, vLLM, SGLang, TensorRT-LLM, TGI, Kubernetes, managed endpoints, self-hosting, private cloud, edge AI, RAG, model gateways, AI governance, GPU economics, low-latency inference, ASIC inference, Cerebras, Groq, SambaNova, NVIDIA Nemotron

Schema contract

Contract: colophon.report_document v2026-05-28
Payload schemas: document, summary, sample, inventory, full_report
Required top-level fields: title, subtitle, abstract, category, sector, region, topics, sources, evidence, claims, colophon, body
Integrity rules: evidence.sourceId must reference sources[].id claim.evidenceIds[] must reference evidence[].id evidence.claimIds[] must reference claims[].id section and block evidence references must resolve colophon.kinetics.speed must match the momentum band for the business impact x momentum model

Data-point inventory

vendor_or_actor_ledger: entity names; product categories; source references; confidence notes
buyer_decision_fields: pricing signals; deployment constraints; risk flags; monitoring fields
evidence_index: source class; access date; claim linkage; verification status

Section inventory

1. Executive summaryPreview
decision-ready conclusions; major findings; confidence caveats
2. Decision framePreview
selection criteria; tradeoff map; recommended buyer actions
3. Owner decision lanesLocked
selection criteria; tradeoff map; recommended buyer actions
4. Deployment economicsPreview
structured findings; supporting evidence; buyer-use fields
5. Scenario economicsLocked
structured findings; supporting evidence; buyer-use fields
6. Deployment optionsLocked
structured findings; supporting evidence; buyer-use fields
7. Deployment atlasLocked
structured findings; supporting evidence; buyer-use fields
8. Speed-first inferenceLocked
structured findings; supporting evidence; buyer-use fields
9. Serving stackLocked
structured findings; supporting evidence; buyer-use fields
10. Workload deployment pathsLocked
structured findings; supporting evidence; buyer-use fields
11. Benchmarks and private evalsLocked
structured findings; supporting evidence; buyer-use fields
12. Risks and controlsLocked
structured findings; supporting evidence; buyer-use fields
13. Licence and regulatory gatesLocked
structured findings; supporting evidence; buyer-use fields
14. Procurement diligenceLocked
structured findings; supporting evidence; buyer-use fields
15. Implementation planLocked
structured findings; supporting evidence; buyer-use fields
16. Deployment case studiesLocked
structured findings; supporting evidence; buyer-use fields
17. Known unknownsLocked
structured findings; supporting evidence; buyer-use fields
18. Vendor and stack directoryLocked
normalized entity rows; source ledger; evidence-to-claim mapping
19. Source ledgerLocked
normalized entity rows; source ledger; evidence-to-claim mapping
20. GlossaryPreview
structured findings; supporting evidence; buyer-use fields

Relevance signals

Confidence score: 0.86
Source classes: standard, licence_terms, platform_docs, serving_docs, cloud_docs, open_source_repo, edge_docs, retrieval_docs, gateway_docs, observability_standard, governance_standard, security_standard, official_guidance, model_adaptation_docs, serving_benchmark, benchmark, model_benchmark, pricing_page, case_study, cloud_security_docs, orchestration_docs, private_cloud_docs, finops_guidance, vendor_reported_benchmark, model_release
Known unknowns: Comparable public evidence is still thin for end-to-end accepted-output cost across the same open model, workload, serving engine, cloud deployment path, concurrency profile, RAG stack, human-verification burden and recovery target.
Watch signals: 2 August 2026: European Commission GPAI enforcement powers enter into application; European open-model deployments may need stronger provider-documentation and systemic-risk evidence [[E31]]. Meta Llama licence terms add, remove or tighten acceptable-use or scale restrictions; approved open-weight deployment paths may need legal reroute [[E2]]. Artificial Analysis provider leaderboard, model-price and model-release updates; managed API speed, latency, intelligence or price shifts can change first-deployment economics [[E37, E73]]. AWS EC2 Capacity Blocks pricing changes p5.48xlarge commitment price, duration or region availability; H100 break-even may move [[E41, E61]]. vLLM release notes; cache, LoRA, speculative-decoding, quantization or security changes can force deployment reapproval.

Locked continuation

Purchase or subscribe to unlock the full findings, evidence ledger, and buyer-ready recommendations.