report.object
Deployment options for open-weight LLM models
Expert-reviewed from public records, platform data, primary sources, and private research beyond AI search. Use /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/sample before purchase, then download PDF, JSON, or Markdown after checkout.
page.markdown
Deployment options for open-weight LLM models.md
# Deployment options for open-weight LLM models
Open-weight LLM deployment is an infrastructure decision across licence, data boundary, serving stack, utilisation, latency, retrieval, controls, recovery and accepted-output cost.
## Object
| field | value |
| ---------------- | ------------------------------ |
| report_id | RPT_01KSHN11BXA9HDRNBPX41MYJ61 |
| schema_version | 2026-05-28 |
| last_updated_at | 2026-06-08 |
| category | INFRA |
| sector | Cross-industry |
| region | GLOBAL |
| confidence_score | 0.86 |
| source_count | 73 |
| evidence_count | 73 |
## Endpoints
| action | endpoint |
| ------------ | --------------------------------------------------------- |
| summary | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61 |
| sample | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/sample |
| schema | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/schema |
| quote | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/quote |
| purchase | /checkout/RPT_01KSHN11BXA9HDRNBPX41MYJ61?view=agent |
| versions | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/versions |
| changelog | /api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/changelog |
| monitor | /api/v1/monitors?report_id=RPT_01KSHN11BXA9HDRNBPX41MYJ61 |
| mcp_resource | colophon://reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/sample |
## Coverage
| field | value |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| category | INFRA |
| sector | Cross-industry |
| region | GLOBAL |
| topics | open-source LLMs, open-weight models, LLM inference, model serving, vLLM, SGLang, TensorRT-LLM, TGI, Kubernetes, managed endpoints, self-hosting, private cloud, edge AI, RAG, model gateways, AI governance, GPU economics, low-latency inference, ASIC inference, Cerebras, Groq, SambaNova, NVIDIA Nemotron |
| source_classes | standard, licence_terms, platform_docs, serving_docs, cloud_docs, open_source_repo, edge_docs, retrieval_docs, gateway_docs, observability_standard, governance_standard, security_standard, official_guidance, model_adaptation_docs, serving_benchmark, benchmark, model_benchmark, pricing_page, case_study, cloud_security_docs, orchestration_docs, private_cloud_docs, finops_guidance, vendor_reported_benchmark, model_release |
| update_cadence | Weekly |
## Sample JSON
```json
{
"report_id": "RPT_01KSHN11BXA9HDRNBPX41MYJ61",
"slug": "open-source-llm-deployment-options",
"schema_version": "2026-05-28",
"title": "Deployment options for open-weight LLM models",
"abstract": "Open-weight LLM deployment is an infrastructure decision across licence, data boundary, serving stack, utilisation, latency, retrieval, controls, recovery and accepted-output cost.",
"category": "INFRA",
"sector": "Cross-industry",
"region": "GLOBAL",
"topics": [
"open-source LLMs",
"open-weight models",
"LLM inference",
"model serving",
"vLLM",
"SGLang",
"TensorRT-LLM",
"TGI",
"Kubernetes",
"managed endpoints",
"self-hosting",
"private cloud",
"edge AI",
"RAG",
"model gateways",
"AI governance",
"GPU economics",
"low-latency inference",
"ASIC inference",
"Cerebras",
"Groq",
"SambaNova",
"NVIDIA Nemotron"
],
"last_updated_at": "2026-06-08T00:00:00Z",
"source_count": 73,
"evidence_count": 73,
"confidence_score": 0.86,
"web_url": "/r/open-source-llm-deployment-options/RPT_01KSHN11BXA9HDRNBPX41MYJ61",
"sample_endpoint": "/api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/sample",
"inventory_endpoint": "/api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/inventory",
"full_endpoint": "/api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/full",
"schema_endpoint": "/api/v1/reports/RPT_01KSHN11BXA9HDRNBPX41MYJ61/schema",
"redaction": {
"status": "sample",
"paid_sections_redacted": true,
"preview_limits": [
"summary",
"table_of_contents",
"evidence_profile"
]
},
"coverage": {
"category": "INFRA",
"sector": "Cross-industry",
"region": "GLOBAL",
"topics": [
"open-source LLMs",
"open-weight models",
"LLM inference",
"model serving",
"vLLM",
"SGLang",
"TensorRT-LLM",
"TGI",
"Kubernetes",
"managed endpoints",
"self-hosting",
"private cloud",
"edge AI",
"RAG",
"model gateways",
"AI governance",
"GPU economics",
"low-latency inference",
"ASIC inference",
"Cerebras",
"Groq",
"SambaNova",
"NVIDIA Nemotron"
],
"source_classes": [
"standard",
"licence_terms",
"platform_docs",
"serving_docs",
"cloud_docs",
"open_source_repo",
"edge_docs",
"retrieval_docs",
"gateway_docs",
"observability_standard",
"governance_standard",
"security_standard",
"official_guidance",
"model_adaptation_docs",
"serving_benchmark",
"benchmark",
"model_benchmark",
"pricing_page",
"case_study",
"cloud_security_docs",
"orchestration_docs",
"private_cloud_docs",
"finops_guidance",
"vendor_reported_benchmark",
"model_release"
]
},
"confidence": {
"overall_confidence": 0.86,
"source_quality": 0.88,
"source_recency": 0.9,
"source_density": 0.86,
"claim_verifiability": 0.85,
"geographic_coverage": 0.84,
"known_gaps": [
"No public dataset fully compares the same open-weight model across API, managed endpoint, dedicated managed GPU, self-hosted cloud GPU, private cloud and edge deployment paths.",
"Vendor case studies rarely publish baseline, denominator, workload mix, evaluation method and independent corroboration.",
"Published GPU prices do not include full serving, support, security, observability, idle capacity, review and recovery costs.",
"Serving-engine benchmarks remain hard to compare across hardware, context length, concurrency, model family and request mix.",
"Live provider-speed pages and model-release benchmark articles expose useful performance signals, but they do not publish the buyer's prompt mix, durable serving route, p95 concurrency, rate-limit headroom or accepted-output rate.",
"Open-weight licence and downstream regulatory obligations can change faster than enterprise deployment approvals."
]
},
"document": {
"object": "report_document",
"schemaVersion": "2026-05-28",
"title": "Deployment options for open-weight LLM models",
"subtitle": "The right deployment is the one that can be approved, operated and recovered.",
"abstract": "Open-weight LLM deployment is an infrastructure decision across licence, data boundary, serving stack, utilisation, latency, retrieval, controls, recovery and accepted-output cost.",
"category": "INFRA",
"sector": "Cross-industry",
"region": "GLOBAL",
"topics": [
"open-source LLMs",
"open-weight models",
"LLM inference",
"model serving",
"vLLM",
"SGLang",
"TensorRT-LLM",
"TGI",
"Kubernetes",
"managed endpoints",
"self-hosting",
"private cloud",
"edge AI",
"RAG",
"model gateways",
"AI governance",
"GPU economics",
"low-latency inference",
"ASIC inference",
"Cerebras",
"Groq",
"SambaNova",
"NVIDIA Nemotron"
],
"sources": [
{
"id": "S1",
"label": "Open Source Initiative - Open Source AI Definition 1.0",
"sourceClass": "standard",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://opensource.org/ai/open-source-ai-definition"
},
{
"id": "S2",
"label": "Meta - Llama licence terms",
"sourceClass": "licence_terms",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://www.llama.com/llama3_1/license/"
},
{
"id": "S3",
"label": "Hugging Face - Inference Providers documentation",
"sourceClass": "platform_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://huggingface.co/docs/hub/models-inference"
},
{
"id": "S4",
"label": "Hugging Face - Inference Endpoints vLLM engine",
"sourceClass": "platform_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://huggingface.co/docs/inference-endpoints/main/engines/vllm"
},
{
"id": "S5",
"label": "Hugging Face - Text Generation Inference documentation",
"sourceClass": "serving_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://huggingface.co/docs/text-generation-inference/en/index"
},
{
"id": "S6",
"label": "Together AI - inference platform",
"sourceClass": "platform_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://www.together.ai/inference"
},
{
"id": "S7",
"label": "Fireworks AI - serverless inference overview",
"sourceClass": "platform_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.fireworks.ai/serverless/overview"
},
{
"id": "S8",
"label": "GroqCloud - developer documentation",
"sourceClass": "platform_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://console.groq.com/docs"
},
{
"id": "S9",
"label": "Microsoft - Azure AI Foundry deployments overview",
"sourceClass": "cloud_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/deployments-overview"
},
{
"id": "S10",
"label": "Google Cloud - deploy open models from Model Garden",
"sourceClass": "cloud_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.cloud.google.com/vertex-ai/generative-ai/docs/open-models/deploy-model-garden"
},
{
"id": "S11",
"label": "AWS SageMaker JumpStart foundation models",
"sourceClass": "cloud_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models.html"
},
{
"id": "S12",
"label": "Databricks - Mosaic AI Model Serving",
"sourceClass": "cloud_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.databricks.com/en/machine-learning/model-serving/index.html"
},
{
"id": "S13",
"label": "NVIDIA TensorRT-LLM key features",
"sourceClass": "serving_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://nvidia.github.io/TensorRT-LLM/key-features.html"
},
{
"id": "S14",
"label": "NVIDIA NIM documentation",
"sourceClass": "serving_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.nvidia.com/nim/"
},
{
"id": "S15",
"label": "vLLM documentation - features",
"sourceClass": "serving_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.vllm.ai/en/latest/features.html"
},
{
"id": "S16",
"label": "SGLang documentation",
"sourceClass": "serving_docs",
"accessedAt": "2026-06-08",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.sglang.ai/"
}
],
"evidence": [
{
"id": "E1",
"sourceId": "S1",
"claimIds": [
"C1"
],
"summary": "The Open Source AI Definition treats an AI system as open source only when the information needed to use, study, modify, and share it is available, making many weight-release models open-weight rather than fully open-source.",
"status": "verified",
"confidence": 0.86,
"access": "sample"
},
{
"id": "E2",
"sourceId": "S2",
"claimIds": [
"C1"
],
"summary": "Meta's Llama licence uses custom terms and acceptable-use restrictions, so a buyer cannot treat every Llama deployment as equivalent to an Apache-licensed software dependency.",
"status": "verified",
"confidence": 0.86,
"access": "sample"
},
{
"id": "E3",
"sourceId": "S3",
"claimIds": [
"C2"
],
"summary": "Hugging Face Inference Providers expose hosted access to many open models through a managed API path, lowering time-to-pilot while shifting gateway, retention, quota, and provider-selection questions into the platform contract.",
"status": "verified",
"confidence": 0.86,
"access": "sample"
},
{
"id": "E7",
"sourceId": "S9",
"claimIds": [
"C3"
],
"summary": "Azure AI Foundry deployment documentation distinguishes deployment types and control-plane choices, making cloud model use an enterprise-governed deployment rather than a generic API call.",
"status": "verified",
"confidence": 0.86,
"access": "sample"
},
{
"id": "E15",
"sourceId": "S15",
"claimIds": [],
"summary": "vLLM documents serving features including paged attention, continuous batching, tensor parallelism, speculative decoding, prefix caching, and LoRA support.",
"status": "verified",
"confidence": 0.86,
"access": "sample"
}
],
"claims": [
{
"id": "C1",
"statement": "Open-weight is not the same thing as open-source; deployment due diligence starts with licence, training-data transparency, derivative rights, and acceptable-use terms.",
"confidence": 0.91,
"evidenceIds": [
"E1",
"E2"
],
"access": "sample"
},
{
"id": "C2",
"statement": "The fastest open-model deployment path is usually a managed API or serverless provider, but that speed trades away some gateway, retention, region, and model-change control.",
"confidence": 0.88,
"evidenceIds": [
"E3"
],
"access": "sample"
},
{
"id": "C3",
"statement": "Cloud model catalogues are the default deployment path for enterprises that already govern identity, network, logging, procurement, and regions through AWS, Azure, Google Cloud, or Databricks.",
"confidence": 0.9,
"evidenceIds": [
"E7"
],
"access": "sample"
}
],
"colophon": {
"checkedAt": "2026-06-08",
"latestRevisionAt": "2026-06-08T00:00:00Z",
"reviewStandard": "expert_human_reviewed",
"kinetics": {
"speed": "moderate",
"impact": 0.82,
"momentum": 0.83,
"businessImpact": "Open-weight deployment has shifted from a model-download question to an approved deployment-path question spanning cloud catalogues, serverless providers, private GPU fleets, RAG systems and gateway controls.",
"rateOfChange": "Serving engines, provider catalogues, licence terms, GPU prices and governance standards are changing quickly enough that deployment approval needs versioned evidence and reapproval triggers.",
"from": "single open-weight experiments and ad hoc GPU notebooks",
"toward": "workload-specific deployment paths with licence review, serving telemetry, retrieval controls, accepted-output economics and recovery evidence"
},
"watch": [
"2 August 2026: European Commission GPAI enforcement powers enter into application; European open-model deployments may need stronger provider-documentation and systemic-risk evidence [E31].",
"Meta Llama licence terms add, remove or tighten acceptable-use or scale restrictions; approved open-weight deployment paths may need legal reroute [E2].",
"Artificial Analysis provider leaderboard, model-price and model-release updates; managed API speed, latency, intelligence or price shifts can change first-deployment economics [E37, E73].",
"AWS EC2 Capacity Blocks pricing changes p5.48xlarge commitment price, duration or region availability; H100 break-even may move [E41, E61].",
"vLLM release notes; cache, LoRA, speculative-decoding, quantization or security changes can force deployment reapproval."
],
"gapAnalysis": "Comparable public evidence is still thin for end-to-end accepted-output cost across the same open model, workload, serving engine, cloud deployment path, concurrency profile, RAG stack, human-verification burden and recovery target."
},
"body": {
"sections": [
{
"id": "verdict",
"title": "Executive summary",
"access": "sample",
"claimIds": [
"C1",
"C2",
"C3"
],
"evidenceIds": [
"E1",
"E2",
"E3",
"E7",
"E15"
],
"blocks": [
{
"id": "verdict-narrative",
"evidenceIds": [
"E1",
"E2",
"E3",
"E7",
"E15"
],
"type": "section_narrative",
"lead": "The buyer is not choosing between open-source and cloud. It is choosing how a workload may cross a boundary, spend a GPU hour, find context, fail over and prove what happened.",
"body": [
"Start with the work. A public chatbot, internal analyst copilot, regulated-document assistant, batch extraction system, tenant-specific SaaS feature and field-device workflow should not share one approval. Each has a different data boundary, latency profile, context length, recovery target and audit burden.",
"The phrase open-source is often too loose for procurement. The Open Source AI Definition requires enough information to use, study, modify and share the system; many popular models are better described as open-weight because the licence, training data, acceptable-use terms or derivative rights are not the same as a normal open-source software dependency [E1, E2]."
],
"detail": [
"The default sequence is pragmatic. Use managed APIs for prototypes and bursty workloads. Use cloud model catalogues when the enterprise already trusts the cloud control plane. Use dedicated managed or self-hosted GPUs when volume, latency, custom weights or data boundaries justify the operating burden. Use private cloud, colocation or edge only when the boundary is worth the slower procurement [E3, E7].",
"The winning number is cost per accepted output: the model bill plus serving, retries, vector search, review minutes, security, patching, observability and fallback capacity. Cheap token pricing can still be expensive work."
]
},
{
"id": "route-first-table",
"title": "Board-level deployment decision",
"evidenceIds": [
"E1",
"E2",
"E3",
"E7",
"E15"
],
"type": "table",
"columns": [
{
"key": "question",
"label": "Question",
"align": "left"
},
{
"key": "answer",
"label": "Answer",
"align": "left"
},
{
"key": "buyer_move",
"label": "Buyer move",
"align": "left"
}
],
"rows": [
{
"question": "Which deployment option wins?",
"answer": "The deployment path that fits the workload's data boundary, latency, utilisation and proof burden.",
"buyer_move": "Approve deployment paths by workload, not by model family."
},
{
"question": "Where should a first deployment start?",
"answer": "Managed APIs, serverless open-model providers or the existing cloud model catalogue.",
"buyer_move": "Buy speed until traffic, privacy or customisation proves the need for dedicated infrastructure."
},
{
"question": "When does self-hosting win?",
"answer": "High steady utilisation, custom serving control, restricted data or a clear latency requirement.",
"buyer_move": "Build only after a GPU utilisation and accepted-output cost model beats managed options."
},
{
"question": "What breaks production first?",
"answer": "Licence ambiguity, weak retrieval, low GPU utilisation, poor p95 latency, hidden fallback, missing logs and unrecovered vector state.",
"buyer_move": "Test the whole deployment path: model, serving engine, RAG, gateway, observability, security and recovery."
}
]
},
{
"id": "deployment-route-tree",
"title": "Deployment decision tree",
"evidenceIds": [
"E1",
"E2",
"E3",
"E7",
"E15"
],
"type": "decision_tree",
"lead": "Classify the data and workload before comparing model providers.",
"steps": [
{
"prompt": "Can prompts, files, outputs and logs leave the buyer's approved boundary?",
"tone": "blocked",
"branches": [
{
"label": "No",
"outcome": "Start with private cloud, on-prem, colocation or edge.",
"tone": "blocked"
},
{
"label": "Yes",
"outcome": "Continue to control-plane fit.",
"next": "Q2 control plane",
"tone": "neutral"
}
]
},
{
"prompt": "Does an existing cloud control plane already cover IAM, network, audit, procurement and region?",
"tone": "warning",
"branches": [
{
"label": "Yes",
"outcome": "Start with managed cloud endpoints or model catalogue deployments.",
"tone": "success"
},
{
"label": "No",
"outcome": "Use serverless APIs for pilots, then prove whether a platform team is needed.",
"tone": "info"
}
]
},
{
"prompt": "Is traffic steady enough to keep dedicated GPUs busy?",
"tone": "warning",
"branches": [
{
"label": "Yes",
"outcome": "Compare dedicated managed deployments with self-hosted GPUs.",
"tone": "success"
},
{
"label": "No",
"outcome": "Variable APIs usually beat idle infrastructure.",
"tone": "info"
}
]
},
{
"prompt": "Does the deployment need custom weights, LoRA adapters, RAG isolation or hard latency guarantees?",
"tone": "warning",
"branches": [
{
"label": "Yes",
"outcome": "Test serving engine, adapter lineage, vector tenancy and p95 latency before approval.",
"tone": "warning"
},
{
"label": "No",
"outcome": "Use the least operationally ambitious deployment path that clears data terms and evals.",
"tone": "success"
}
]
}
]
},
{
"id": "verdict-section-callout",
"title": "Approve the deployment, not the weight",
"evidenceIds": [
"E1",
"E2",
"E3",
"E7"
],
"type": "callout",
"tone": "warning",
"body": "A model licence tells you whether you can use the weight. It does not tell you whether a deployment path is approved for this data class, latency target, recovery target, customer promise or audit obligation."
}
]
},
{
"id": "decision-frame",
"title": "Decision frame",
"access": "sample",
"dek": "The deployment object is a controlled production path, not a model binary.",
"claimIds": [
"C1",
"C2",
"C3"
],
"evidenceIds": [
"E3",
"E7"
],
"blocks": [
{
"id": "decision-frame-narrative",
"evidenceIds": [
"E3",
"E7"
],
"type": "section_narrative",
"lead": "Model deployment has four moving parts: rights, runtime, context and proof.",
"body": [
"Rights decide whether the model can be used and modified. Runtime decides where and how it serves. Context decides which sources it can see. Proof decides whether security, compliance and the business can trust the deployment after the next model, adapter or provider change.",
"The buyer's first register should therefore name the deployment path, not merely the model: licence, data class, endpoint, serving engine, vector store, gateway policy, logging standard, reviewer, recovery target and reapproval trigger."
],
"detail": [
"Cloud approval does not approve every use case. A deployment inside Azure, AWS, Google Cloud or Databricks can still be wrong for customer data, regulated outputs, cross-border records or external communications.",
"The practical test is whether the deployment path can be explained to the person who owns the failure."
]
},
{
"id": "route-register-table",
"title": "Minimum deployment register",
"evidenceIds": [
"E3",
"E7"
],
"type": "table",
"columns": [
{
"key": "field",
"label": "Field",
"align": "left"
},
{
"key": "bad_entry",
"label": "Bad entry",
"align": "left"
},
{
"key": "approved_entry",
"label": "Approved entry",
"align": "left"
},
{
"key": "why_it_matters",
"label": "Why it matters",
"align": "left"
}
],
"rows": [
{
"field": "Model status",
"bad_entry": "Open-source Llama.",
"approved_entry": "Llama-family weight under reviewed licence, approved for internal summarization.",
"why_it_matters": "Licence and acceptable-use terms can block deployment before runtime does."
},
{
"field": "Data class",
"bad_entry": "Internal data.",
"approved_entry": "Public docs, confidential employee data, regulated customer data or export-controlled content.",
"why_it_matters": "Retention, region, support access and fallback policy change by class."
},
{
"field": "Serving path",
"bad_entry": "GPU endpoint.",
"approved_entry": "Azure managed endpoint, Together serverless API, vLLM on Kubernetes or on-device runtime.",
"why_it_matters": "The serving path determines control plane, logs, region, latency and recovery."
},
{
"field": "Context layer",
"bad_entry": "RAG.",
"approved_entry": "Embedding model, vector store, tenant namespace, filter policy and rebuild trigger.",
"why_it_matters": "Retrieval mistakes are deployment failures, not prompt failures."
},
{
"field": "Fallback",
"bad_entry": "Use another provider.",
"approved_entry": "Allow-listed fallback path by data class and benchmark threshold.",
"why_it_matters": "Fallback can silently change data terms and answer quality."
},
{
"field": "Reapproval trigger",
"bad_entry": "Annual review.",
"approved_entry": "Model version, licence, serving engine, adapter, region, vector schema, threshold or provider term change.",
"why_it_matters": "Open-model deployments move too quickly for calendar-only governance."
}
]
}
]
},
{
"id": "c-suite-decision-lanes",
"title": "Owner decision lanes",
"access": "locked",
"dek": "The same deployment path has different owners because it fails in different budgets.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "c-suite-decision-lanes-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "deployment-economics",
"title": "Deployment economics",
"access": "sample",
"dek": "Price the accepted output, not the token or the GPU hour.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "deployment-economics-narrative",
"evidenceIds": [],
"type": "section_narrative",
"lead": "The GPU line item is not deployment cost. Production cost is what the system spends to produce accepted work under the required data boundary, latency, audit and recovery constraints.",
"body": [
"Token price matters, but it is only one line. Open-model deployment also pays for idle GPUs, serving engineers, cache misses, long prompts, vector search, retries, rejected answers, review minutes, monitoring, patching, support and failover.",
"GPU-hour price matters, but it is also only a line. A cheap H100 that sits idle, lacks private networking, cannot meet p95 latency or needs manual recovery can lose to a pricier managed endpoint that clears the whole acceptance test.",
"Public infrastructure price rows are only inputs: GPU hour, endpoint commitment, provisioned throughput, platform fee, region, support and capacity window. The denominator that matters is accepted output: a correct answer, classified document, extracted field, completed support response, reviewed agent action or refreshed knowledge-base packet [E39, E40]."
],
"detail": [
"Self-hosting wins when utilisation is high and operations are funded. Managed APIs win when use is bursty, uncertain or early. Dedicated managed deployments sit in the middle: stronger isolation and performance control than serverless, less operating burden than a self-run cluster.",
"The procurement mistake is to compare a cloud list price with a self-hosted GPU line and call the cheaper row the answer. Compare the full deployment, not the cheapest visible line item."
]
},
{
"id": "gpu-hour-price-chart",
"title": "A cheap GPU is not a production system",
"evidenceIds": [],
"type": "chart",
"chartType": "bar",
"data": [
{
"provider": "RunPod secure cloud",
"route": "1x H100 PCIe",
"hourly_price": 1.99,
"unit": "US dollars per GPU hour",
"caveat": "Marketplace-style GPU rental; enterprise controls and region availability vary.",
"secondary": true
},
{
"provider": "Lambda cloud",
"route": "1x H100 SXM",
"hourly_price": 2.99,
"unit": "US dollars per GPU hour",
"caveat": "On-demand cloud GPU price; storage, network and reserved terms are separate.",
"secondary": true
},
{
"provider": "Lambda cloud",
"route": "1x H200 SXM",
"hourly_price": 3.99,
"unit": "US dollars per GPU hour",
"caveat": "On-demand cloud GPU price; useful for memory-bound long-context serving comparisons.",
"secondary": true
},
{
"provider": "AWS EC2 P5 capacity block",
"route": "8x H100 effective per GPU",
"hourly_price": 12.29,
"unit": "US dollars per GPU hour",
"caveat": "Effective per-GPU rate from a p5.48xlarge capacity block; hourly commitment and regional availability matter.",
"highlight": true
}
],
"spec": {
"$schema": "https://vega.github.io/schema/vega-lite/v6.json",
"width": 820,
"height": 172,
"padding": {
"top": 8,
"right": 74,
"bottom": 8,
"left": 0
},
"data": {
"values": [
{
"provider": "RunPod secure cloud",
"route": "1x H100 PCIe",
"hourly_price": 1.99,
"unit": "US dollars per GPU hour",
"caveat": "Marketplace-style GPU rental; enterprise controls and region availability vary.",
"secondary": true
},
{
"provider": "Lambda cloud",
"route": "1x H100 SXM",
"hourly_price": 2.99,
"unit": "US dollars per GPU hour",
"caveat": "On-demand cloud GPU price; storage, network and reserved terms are separate.",
"secondary": true
},
{
"provider": "Lambda cloud",
"route": "1x H200 SXM",
"hourly_price": 3.99,
"unit": "US dollars per GPU hour",
"caveat": "On-demand cloud GPU price; useful for memory-bound long-context serving comparisons.",
"secondary": true
},
{
"provider": "AWS EC2 P5 capacity block",
"route": "8x H100 effective per GPU",
"hourly_price": 12.29,
"unit": "US dollars per GPU hour",
"caveat": "Effective per-GPU rate from a p5.48xlarge capacity block; hourly commitment and regional availability matter.",
"highlight": true
}
]
},
"encoding": {
"y": {
"field": "route",
"type": "nominal",
"sort": "-x",
"axis": {
"title": null,
"labelFont": "IBM Plex Mono",
"labelFontSize": 12,
"labelLimit": 0,
"labelColor": "#111111",
"domain": false,
"ticks": false
}
},
"x": {
"field": "hourly_price",
"type": "quantitative",
"scale": {
"zero": true
},
"axis": {
"title": "US dollars per accelerator hour",
"titleFont": "IBM Plex Mono",
"titleFontSize": 12,
"labelFont": "IBM Plex Mono",
"labelFontSize": 11,
"labelExpr": "'$' + format(datum.value, '.2f')",
"gridColor": "#ece9df",
"domain": false
}
},
"tooltip": [
{
"field": "provider",
"type": "nominal",
"title": "Provider"
},
{
"field": "route",
"type": "nominal",
"title": "Deployment path"
},
{
"field": "hourly_price",
"type": "quantitative",
"title": "Hourly price",
"format": "$,.2f"
},
{
"field": "caveat",
"type": "nominal",
"title": "Caveat"
}
]
},
"layer": [
{
"mark": {
"type": "bar",
"size": 20
}
},
{
"mark": {
"type": "text",
"align": "left",
"baseline": "middle",
"dx": 6,
"font": "IBM Plex Mono",
"fontSize": 11,
"fontWeight": 700,
"color": "#111111"
},
"encoding": {
"text": {
"field": "hourly_price",
"type": "quantitative",
"format": "$,.2f"
}
}
}
]
},
"notes": "Public price pages checked 28 May 2026. Outlined rows are visible GPU rental or on-demand input prices; the filled row is a committed enterprise cloud deployment path. Rows exclude storage, networking, reserved terms, idle replicas, support, serving engineering, security, monitoring and review burden. Use the chart as an input to the accepted-output model, not as a procurement ranking. [E39, E40, E41]"
},
{
"id": "accepted-output-cost-bridge",
"evidenceIds": [],
"type": "paragraph",
"text": "The GPU chart supplies input prices. The table converts those inputs into accepted-output economics by adding utilisation, hidden operating work, review burden, rejected runs and the deployment boundary each path must satisfy.",
"variant": "small"
},
{
"id": "accepted-output-cost-table",
"title": "How to price each deployment path",
"evidenceIds": [],
"type": "table",
"columns": [
{
"key": "route",
"label": "Deployment path",
"align": "left"
},
{
"key": "visible_unit",
"label": "Visible unit",
"align": "left"
},
{
"key": "hidden_cost",
"label": "Hidden cost",
"align": "left"
},
{
"key": "accepted_output_test",
"label": "Accepted-output test",
"align": "left"
}
],
"rows": [
{
"route": "Serverless open-model API",
"visible_unit": "Tokens, requests or provider plan",
"hidden_cost": "Rate limits, retries, provider queues, gateway fees and review minutes.",
"accepted_output_test": "Cost per correct response at p95 latency and approved data terms."
},
{
"route": "Managed cloud endpoint",
"visible_unit": "Endpoint, tokens, compute or provisioned throughput",
"hidden_cost": "Cloud service overhead, region constraints, endpoint idle time, logging and security controls.",
"accepted_output_test": "Cost per accepted answer inside the existing cloud control plane."
},
{
"route": "Dedicated managed deployment",
"visible_unit": "Reserved GPU, replica or throughput commitment",
"hidden_cost": "Idle capacity, autoscaling limits, quota and failover capacity.",
"accepted_output_test": "Cost per accepted output at planned concurrency and fallback policy."
},
{
"route": "Self-hosted cloud GPU",
"visible_unit": "GPU hour",
"hidden_cost": "Serving engineers, patching, observability, SRE, orchestration, storage, networking and incident response.",
"accepted_output_test": "Cost per accepted output at realistic utilisation, including rejected runs."
},
{
"route": "Private cloud, colocation or on-prem",
"visible_unit": "Capex, managed infrastructure or subscription",
"hidden_cost": "Power, cooling, supply chain, upgrades, support, DR and slower elasticity.",
"accepted_output_test": "Cost per accepted output where data boundary or sovereignty justifies the burden."
},
{
"route": "Edge or on-device",
"visible_unit": "Device capacity and update cost",
"hidden_cost": "Quantization loss, telemetry gaps, fleet updates, support and weaker fallback.",
"accepted_output_test": "Accepted local answer under offline constraints and recoverable update path."
}
]
},
{
"id": "route-definition-callout",
"title": "Route definition",
"evidenceIds": [],
"type": "callout",
"tone": "neutral",
"body": "Route means the approved path from work to accepted output: the source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger that make an AI result usable."
},
{
"id": "deployment-economics-section-callout",
"title": "A free weight is not a free system",
"evidenceIds": [],
"type": "callout",
"tone": "info",
"body": "Count GPUs, utilisation, serving work, vector stores, adapters, retries, review minutes, security, patching, observability and fallback capacity before calling self-hosting cheaper."
}
]
},
{
"id": "scenario-economics",
"title": "Scenario economics",
"access": "locked",
"dek": "Break-even moves with traffic shape, not with ideology.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "scenario-economics-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "deployment-options",
"title": "Deployment options",
"access": "locked",
"dek": "Every deployment path buys one kind of control and sells another.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "deployment-options-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "route-atlas",
"title": "Deployment atlas",
"access": "locked",
"dek": "A premium shortlist compares deployment families, not vendor logos.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "route-atlas-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "speed-first-inference",
"title": "Speed-first inference",
"access": "locked",
"dek": "Tokens per second buy time only when the deployment path can be approved.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "speed-first-inference-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "serving-stack",
"title": "Serving stack",
"access": "locked",
"dek": "The runtime is where model quality turns into latency, cost and failure.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "serving-stack-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "workload-routing",
"title": "Workload deployment paths",
"access": "locked",
"dek": "Workload shape decides the deployment path before model preference does.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "workload-routing-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "benchmarks-and-private-evals",
"title": "Benchmarks and private evals",
"access": "locked",
"dek": "Public benchmarks show where to look. Private evals decide what can run.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "benchmarks-and-private-evals-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "route-risk-controls",
"title": "Risks and controls",
"access": "locked",
"dek": "The deployment fails when the hidden dependency changes.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "route-risk-controls-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "licence-and-regulatory-gates",
"title": "Licence and regulatory gates",
"access": "locked",
"dek": "A technically good deployment can still be legally unusable.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "licence-and-regulatory-gates-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "procurement-diligence",
"title": "Procurement diligence",
"access": "locked",
"dek": "The contract should describe the deployment the architecture diagram promises.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "procurement-diligence-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "implementation-plan",
"title": "Implementation plan",
"access": "locked",
"dek": "Pilot the cheapest path that can become the production deployment.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "implementation-plan-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "case-studies",
"title": "Deployment case studies",
"access": "locked",
"dek": "Public examples show deployment shape better than they prove ROI.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "case-studies-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "known-unknowns",
"title": "Known unknowns",
"access": "locked",
"dek": "The public record is deepest on deployment mechanics and thinnest on comparable accepted-output cost.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "known-unknowns-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "vendor-directory",
"title": "Vendor and stack directory",
"access": "locked",
"dek": "Representative deployment components for shortlist building.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "vendor-directory-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "source-ledger",
"title": "Source ledger",
"access": "locked",
"dek": "Primary sources and diligence anchors behind the analysis.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "source-ledger-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "glossary",
"title": "Glossary",
"access": "sample",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "glossary-table",
"title": "Terms used in this report",
"caption": "Definitions are reader aids, not separate evidence claims.",
"evidenceIds": [],
"type": "table",
"columns": [
{
"key": "term",
"label": "Term",
"align": "left"
},
{
"key": "meaning",
"label": "Plain meaning",
"align": "left"
}
],
"rows": [
{
"term": "Route",
"meaning": "The approved path from work to accepted output: source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger."
},
{
"term": "API",
"meaning": "Application programming interface: a governed software connection used by an app or workflow to call a model or service."
},
{
"term": "LLM",
"meaning": "Large language model: a model trained to read and generate text-like tokens."
},
{
"term": "Token",
"meaning": "A chunk of text used for billing, context windows and rate limits. It is usually smaller than a word, not a page count."
},
{
"term": "RAG",
"meaning": "Retrieval-augmented generation: search approved sources first, then pass selected evidence into the model."
},
{
"term": "SLA",
"meaning": "Service-level agreement: a contractual availability or support commitment, not observed uptime history."
},
{
"term": "SLO",
"meaning": "Service-level objective: a service target that may be published or internal, and may not be contractual."
},
{
"term": "JSON",
"meaning": "JavaScript Object Notation: a structured text format commonly used for machine-readable API output."
}
]
}
]
}
]
}
},
"gap_analysis": "Comparable public evidence is still thin for end-to-end accepted-output cost across the same open model, workload, serving engine, cloud deployment path, concurrency profile, RAG stack, human-verification burden and recovery target.",
"kinetics": {
"speed": "moderate",
"impact": 0.82,
"momentum": 0.83,
"businessImpact": "Open-weight deployment has shifted from a model-download question to an approved deployment-path question spanning cloud catalogues, serverless providers, private GPU fleets, RAG systems and gateway controls.",
"rateOfChange": "Serving engines, provider catalogues, licence terms, GPU prices and governance standards are changing quickly enough that deployment approval needs versioned evidence and reapproval triggers.",
"from": "single open-weight experiments and ad hoc GPU notebooks",
"toward": "workload-specific deployment paths with licence review, serving telemetry, retrieval controls, accepted-output economics and recovery evidence"
},
"watch_signals": [
"2 August 2026: European Commission GPAI enforcement powers enter into application; European open-model deployments may need stronger provider-documentation and systemic-risk evidence [E31].",
"Meta Llama licence terms add, remove or tighten acceptable-use or scale restrictions; approved open-weight deployment paths may need legal reroute [E2].",
"Artificial Analysis provider leaderboard, model-price and model-release updates; managed API speed, latency, intelligence or price shifts can change first-deployment economics [E37, E73].",
"AWS EC2 Capacity Blocks pricing changes p5.48xlarge commitment price, duration or region availability; H100 break-even may move [E41, E61].",
"vLLM release notes; cache, LoRA, speculative-decoding, quantization or security changes can force deployment reapproval."
],
"sample_findings": [
"Open-weight deployment is infrastructure approval across licence, data boundary, serving engine, traffic shape, retrieval, observability and recovery."
]
}
```