report.object
AI agents for software engineering
Expert-reviewed from public records, platform data, primary sources, and private research beyond AI search. Use /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/sample before purchase, then download PDF, JSON, or Markdown after checkout.
page.markdown
AI agents for software engineering.md
# AI agents for software engineering
Software-engineering AI agents should be bought against accepted delivery: PR acceptance, CI authority, review capacity, rollback ownership and usage cost after failed attempts.
## Object
| field | value |
| ---------------- | ------------------------------------------ |
| report_id | RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06 |
| schema_version | 2026-05-28 |
| last_updated_at | 2026-06-17 |
| category | AGENTS |
| sector | Technology |
| region | GLOBAL |
| confidence_score | 0.87 |
| source_count | 102 |
| evidence_count | 103 |
## Endpoints
| action | endpoint |
| ------------ | --------------------------------------------------------------------- |
| summary | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06 |
| sample | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/sample |
| schema | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/schema |
| quote | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/quote |
| purchase | /checkout/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06?view=agent |
| versions | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/versions |
| changelog | /api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/changelog |
| monitor | /api/v1/monitors?report_id=RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06 |
| mcp_resource | colophon://reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/sample |
## Coverage
| field | value |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| category | AGENTS |
| sector | Technology |
| region | GLOBAL |
| topics | software-engineering AI agents, coding agents, agentic software development, pull request approval, workflow approval, developer productivity, software supply chain, secure software development, MCP security, coding benchmarks, private evals, CI/CD governance, branch protection, repository permissions, developer platforms |
| source_classes | official_coding_agent_docs, model_system_card, source_control_agent_docs, source_control_governance, ci_cd_security_docs, coding_agent_vendor_docs, agent_platform_architecture, code_search_permission_docs, developer_survey, productivity_study, software_delivery_research, software_delivery_metrics, coding_agent_benchmark, benchmark_limitation, agent_benchmark, agent_benchmark_paper, security_standard, secure_software_standard, software_supply_chain_standard, research_paper, independent_productivity_study, ci_cd_approval_docs, security_research, coding_agent_vendor_limitations, pricing_docs, application_security_docs, independent_capability_study, incident_database, sandbox_infrastructure_docs, security_guidance, operational_status_page, security_event_schedule, open_source_coding_agent_docs, market_survey |
| update_cadence | Weekly |
## Sample JSON
```json
{
"report_id": "RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06",
"slug": "ai-agents-software-engineering",
"schema_version": "2026-05-28",
"title": "AI agents for software engineering",
"abstract": "Software-engineering AI agents should be bought against accepted delivery: PR acceptance, CI authority, review capacity, rollback ownership and usage cost after failed attempts.",
"category": "AGENTS",
"sector": "Technology",
"region": "GLOBAL",
"topics": [
"software-engineering AI agents",
"coding agents",
"agentic software development",
"pull request approval",
"workflow approval",
"developer productivity",
"software supply chain",
"secure software development",
"MCP security",
"coding benchmarks",
"private evals",
"CI/CD governance",
"branch protection",
"repository permissions",
"developer platforms"
],
"last_updated_at": "2026-06-17T00:00:00Z",
"source_count": 102,
"evidence_count": 103,
"confidence_score": 0.87,
"web_url": "/r/ai-agents-software-engineering/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06",
"sample_endpoint": "/api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/sample",
"inventory_endpoint": "/api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/inventory",
"full_endpoint": "/api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/full",
"schema_endpoint": "/api/v1/reports/RPT_AI_AGENTS_SOFTWARE_ENGINEERING_2026_06/schema",
"redaction": {
"status": "sample",
"paid_sections_redacted": true,
"preview_limits": [
"summary",
"table_of_contents",
"evidence_profile"
]
},
"coverage": {
"category": "AGENTS",
"sector": "Technology",
"region": "GLOBAL",
"topics": [
"software-engineering AI agents",
"coding agents",
"agentic software development",
"pull request approval",
"workflow approval",
"developer productivity",
"software supply chain",
"secure software development",
"MCP security",
"coding benchmarks",
"private evals",
"CI/CD governance",
"branch protection",
"repository permissions",
"developer platforms"
],
"source_classes": [
"official_coding_agent_docs",
"model_system_card",
"source_control_agent_docs",
"source_control_governance",
"ci_cd_security_docs",
"coding_agent_vendor_docs",
"agent_platform_architecture",
"code_search_permission_docs",
"developer_survey",
"productivity_study",
"software_delivery_research",
"software_delivery_metrics",
"coding_agent_benchmark",
"benchmark_limitation",
"agent_benchmark",
"agent_benchmark_paper",
"security_standard",
"secure_software_standard",
"software_supply_chain_standard",
"research_paper",
"independent_productivity_study",
"ci_cd_approval_docs",
"security_research",
"coding_agent_vendor_limitations",
"pricing_docs",
"application_security_docs",
"independent_capability_study",
"incident_database",
"sandbox_infrastructure_docs",
"security_guidance",
"operational_status_page",
"security_event_schedule",
"open_source_coding_agent_docs",
"market_survey"
]
},
"confidence": {
"overall_confidence": 0.87,
"source_quality": 0.88,
"source_recency": 0.93,
"source_density": 0.9,
"claim_verifiability": 0.85,
"geographic_coverage": 0.82,
"known_gaps": [
"Independent enterprise evidence for coding-agent routes across private repositories, secrets, branch protections, CI/CD, deployment and rollback remains thin.",
"Vendor product documentation is strong for product boundaries but weak for comparable accepted-change performance.",
"Public benchmark validity is changing quickly, especially for SWE-bench-style tasks exposed in training data.",
"Real-world agent-authored pull-request datasets are growing, but public studies still struggle to observe the full human approval decision.",
"Review-burden, rework and post-merge incident economics usually require buyer-specific telemetry."
]
},
"document": {
"object": "report_document",
"schemaVersion": "2026-05-28",
"title": "AI agents for software engineering",
"subtitle": "The scarce proof is accepted delivery after review, CI and rollback.",
"abstract": "Software-engineering AI agents should be bought against accepted delivery: PR acceptance, CI authority, review capacity, rollback ownership and usage cost after failed attempts.",
"category": "AGENTS",
"sector": "Technology",
"region": "GLOBAL",
"topics": [
"software-engineering AI agents",
"coding agents",
"agentic software development",
"pull request approval",
"workflow approval",
"developer productivity",
"software supply chain",
"secure software development",
"MCP security",
"coding benchmarks",
"private evals",
"CI/CD governance",
"branch protection",
"repository permissions",
"developer platforms"
],
"sources": [
{
"id": "S1",
"label": "OpenAI - introducing Codex",
"sourceClass": "official_coding_agent_docs",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://openai.com/index/introducing-codex/"
},
{
"id": "S4",
"label": "GitHub Docs - about Copilot cloud agent",
"sourceClass": "source_control_agent_docs",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent"
},
{
"id": "S5",
"label": "GitHub Docs - review Copilot output",
"sourceClass": "source_control_agent_docs",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.github.com/en/copilot/how-tos/copilot-on-github/use-copilot-agents/review-copilot-output"
},
{
"id": "S6",
"label": "GitHub Docs - protected branches",
"sourceClass": "source_control_governance",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches"
},
{
"id": "S18",
"label": "Sonar - State of Code Developer Survey",
"sourceClass": "developer_survey",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://www.sonarsource.com/blog/state-of-code-developer-survey-report-the-current-reality-of-ai-coding"
},
{
"id": "S21",
"label": "DORA - State of AI-assisted Software Development 2025",
"sourceClass": "software_delivery_research",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://dora.dev/research/2025/dora-report/"
},
{
"id": "S39",
"label": "DORA - software delivery performance metrics",
"sourceClass": "software_delivery_metrics",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://dora.dev/guides/dora-metrics/"
},
{
"id": "S23",
"label": "OpenAI - why we no longer evaluate SWE-bench Verified",
"sourceClass": "benchmark_limitation",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"
},
{
"id": "S32",
"label": "arXiv - coding-agent PR lifecycle governance",
"sourceClass": "research_paper",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://arxiv.org/abs/2605.08017"
},
{
"id": "S41",
"label": "Anthropic - Claude Code auto mode",
"sourceClass": "coding_agent_vendor_limitations",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://www.anthropic.com/engineering/claude-code-auto-mode"
},
{
"id": "S48",
"label": "Azure DevOps Docs - approvals and checks",
"sourceClass": "ci_cd_approval_docs",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops"
},
{
"id": "S53",
"label": "DevOps-Gym",
"sourceClass": "agent_benchmark",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://www.devops-gym.com/"
},
{
"id": "S55",
"label": "OpenAI Developers - Codex pricing",
"sourceClass": "pricing_docs",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://developers.openai.com/codex/pricing"
},
{
"id": "S62",
"label": "GitHub Docs - Copilot premium requests",
"sourceClass": "pricing_docs",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://docs.github.com/en/copilot/reference/copilot-billing/request-based-billing-legacy/copilot-requests"
},
{
"id": "S67",
"label": "METR - measuring AI ability to complete long tasks",
"sourceClass": "independent_capability_study",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/"
},
{
"id": "S74",
"label": "GitHub Docs - add Copilot cloud agent to organization",
"sourceClass": "source_control_agent_docs",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://docs.github.com/en/copilot/how-tos/administer-copilot/manage-for-organization/add-copilot-cloud-agent"
},
{
"id": "S75",
"label": "GitHub Status - platform component history",
"sourceClass": "operational_status_page",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://www.githubstatus.com/"
},
{
"id": "S78",
"label": "OpenAI Developers - Codex cloud",
"sourceClass": "official_coding_agent_docs",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://developers.openai.com/codex/cloud"
},
{
"id": "S79",
"label": "OpenAI Developers - Codex GitHub integration",
"sourceClass": "official_coding_agent_docs",
"accessedAt": "2026-06-16",
"status": "monitor",
"access": "sample",
"transformChain": [],
"url": "https://developers.openai.com/codex/integrations/github"
},
{
"id": "S102",
"label": "Silicon Valley Bank - State of Enterprise Software Report 2026",
"sourceClass": "market_survey",
"accessedAt": "2026-06-16",
"status": "verified",
"access": "sample",
"transformChain": [],
"url": "https://www.svb.com/trends-insights/reports/state-of-enterprise-software/"
}
],
"evidence": [
{
"id": "E1",
"sourceId": "S1",
"claimIds": [
"C1"
],
"summary": "OpenAI describes Codex as a cloud-based software-engineering agent that can write features, answer codebase questions, fix bugs and propose pull requests in per-task cloud sandboxes.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E4",
"sourceId": "S4",
"claimIds": [
"C1",
"C4"
],
"summary": "GitHub Copilot cloud agent works in an ephemeral GitHub Actions-powered development environment, can research, plan, edit on a branch, create pull requests and expose PR lifecycle metrics.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E5",
"sourceId": "S5",
"claimIds": [
"C4"
],
"summary": "GitHub tells reviewers to inspect Copilot pull requests before merging; a user's approval of a Copilot PR does not count when PR approvals are required, and Actions workflows do not run automatically by default.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E6",
"sourceId": "S6",
"claimIds": [
"C4"
],
"summary": "GitHub branch protection can require pull requests, approving reviews, code-owner review, most-recent-push approval, required status checks, source-specific checks and no bypass by administrators.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E18",
"sourceId": "S18",
"claimIds": [
"C7"
],
"summary": "Sonar's 2026 survey of more than 1,100 professional developers reports that AI accounts for 42% of committed code, is expected to reach 65% by 2027, 96% do not fully trust AI-generated code and only 48% always verify before committing.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E21",
"sourceId": "S21",
"claimIds": [
"C7"
],
"summary": "DORA's 2025 AI-assisted software-development report says AI acts as an amplifier of organizational strengths and weaknesses, with returns depending on the underlying organizational system rather than the tools alone.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E23",
"sourceId": "S23",
"claimIds": [
"C6"
],
"summary": "OpenAI stopped reporting SWE-bench Verified because audited failed cases included flawed tests, benchmark exposure and underspecified tasks that no longer reflected meaningful real-world software-development gains.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E32",
"sourceId": "S32",
"claimIds": [],
"summary": "A PR-lifecycle study of 29,585 pull requests says agent initiative and merge governance decouple: collaborator workflows are at least 96% agent-initiated, while terminal merge authority remains almost exclusively human.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E41",
"sourceId": "S41",
"claimIds": [
"C24"
],
"summary": "Anthropic says Claude Code users approved 93% of permission prompts and cites past agentic misbehaviours involving remote branch deletion, GitHub-token upload and attempted production-database migrations.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E48",
"sourceId": "S48",
"claimIds": [
"C2",
"C20"
],
"summary": "Azure DevOps approvals and checks are managed by resource owners outside YAML; pipeline authors cannot modify the checks that must pass before a stage consumes a protected resource.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E53",
"sourceId": "S53",
"claimIds": [
"C6"
],
"summary": "DevOps-Gym evaluates agents across build and configuration, monitoring, issue resolving and test generation with 700+ tasks from 30+ Java and Go projects plus 18 end-to-end pipeline tasks.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E55",
"sourceId": "S55",
"claimIds": [
"C22"
],
"summary": "OpenAI Codex pricing maps Business and new Enterprise usage to credits per million tokens, while Plus/Pro-style usage includes message, cloud-task and per-pull-request code-review units.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E62",
"sourceId": "S62",
"claimIds": [
"C22"
],
"summary": "GitHub Copilot premium-request documentation says Copilot code review consumes 13 premium requests per review from June 1, 2026, cloud-agent sessions consume premium requests, Pro includes 300 requests and Pro+ includes 1,500 with $0.04 additional requests.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E67",
"sourceId": "S67",
"claimIds": [
"C7"
],
"summary": "METR's time-horizon work measures 50% and 80% success thresholds for task duration and includes preliminary monitorability evaluations for catching agents doing side tasks or bypassing monitors.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E74",
"sourceId": "S74",
"claimIds": [],
"summary": "GitHub organization documentation says Copilot cloud agent and third-party MCP servers are disabled by default for Business and Enterprise organization members, repository access can be controlled, and any user with write permission can delegate once a repository is enabled.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E75",
"sourceId": "S75",
"claimIds": [
"C20"
],
"summary": "GitHub Status publishes component and regional status, incident subscriptions and 90-day uptime history for components such as Git operations, webhooks, API requests, issues and pull requests.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E78",
"sourceId": "S78",
"claimIds": [],
"summary": "OpenAI Codex cloud documentation describes GitHub-connected repository work, pull-request creation and enterprise-relevant controls for sandboxing, managed configuration and internet access.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
},
{
"id": "E79",
"sourceId": "S79",
"claimIds": [
"C4"
],
"summary": "OpenAI Codex GitHub integration documentation describes manual and automatic pull-request review, AGENTS.md instruction use, P0/P1 issue focus and branch fixes when permitted.",
"status": "verified",
"confidence": 0.88,
"access": "sample"
}
],
"claims": [
{
"id": "C1",
"statement": "The approval unit for software-engineering agents is the accepted-change route, not the generated code snippet.",
"confidence": 0.9,
"evidenceIds": [
"E1",
"E4",
"E5",
"E32"
],
"access": "sample"
},
{
"id": "C2",
"statement": "Coding-agent workflow approval should split read, edit, test, pull-request, review, merge, deploy and rollback rights.",
"confidence": 0.9,
"evidenceIds": [
"E5",
"E6"
],
"access": "sample"
},
{
"id": "C4",
"statement": "Source-control governance is the natural control plane for cloud coding agents because pull requests, branch protections, CI and approvals already encode accepted change.",
"confidence": 0.89,
"evidenceIds": [
"E4",
"E5",
"E6"
],
"access": "sample"
},
{
"id": "C6",
"statement": "Public coding benchmarks can shape private evals but cannot approve enterprise write access on their own.",
"confidence": 0.9,
"evidenceIds": [
"E23"
],
"access": "sample"
},
{
"id": "C7",
"statement": "The economic upside is real but depends on verification capacity and the organization's delivery system.",
"confidence": 0.87,
"evidenceIds": [
"E18",
"E21"
],
"access": "sample"
},
{
"id": "C20",
"statement": "Merge and deployment approvals should stay in source-control and CI/CD control surfaces that agents cannot rewrite as part of the same change.",
"confidence": 0.9,
"evidenceIds": [
"E6",
"E48",
"E75"
],
"access": "sample"
},
{
"id": "C22",
"statement": "Coding-agent economics require a mixed denominator: seat price, credit/request/ACU/task limits, CI and sandbox spend, review time, retries and accepted output.",
"confidence": 0.86,
"evidenceIds": [
"E55",
"E62"
],
"access": "sample"
},
{
"id": "C24",
"statement": "Permission-prompt approval quality is a measurable control, not a formality, because high approval rates can become rubber-stamping for risky actions.",
"confidence": 0.85,
"evidenceIds": [
"E41"
],
"access": "sample"
}
],
"colophon": {
"checkedAt": "2026-06-16",
"latestRevisionAt": "2026-06-17T00:00:00Z",
"reviewStandard": "expert_human_reviewed",
"kinetics": {
"speed": "fast",
"impact": 0.9,
"momentum": 0.86,
"businessImpact": "Coding-agent budgets are shifting from developer-seat productivity claims to delivery-system evidence: accepted pull requests, review load, delivery failures and recovery work.",
"rateOfChange": "Benchmark audits, cloud-agent permissions and agent-security incidents can change what counts as credible delivery evidence within a quarter.",
"from": "assistant seats and generated-code volume",
"toward": "accepted delivery measured after review, CI, deployment and recovery"
},
"watch": [
"1 August 2026: Black Hat USA 2026 opens with Briefings and Arsenal programming; new coding-agent, MCP, supply-chain or prompt-injection research would change negative fixtures for software-agent approval [E83].",
"7 August 2026: DEF CON 34 Cloud Village opens; cloud identity, metadata-service, egress or managed-agent research would change hosted-agent runtime and secrets approval [E84]."
],
"gapAnalysis": "The missing artefact is independent longitudinal production telemetry linking coding-agent work across private repositories, secrets, workflow-file edits, CI/CD, human review, deployment gates, recovery work and post-merge defects."
},
"body": {
"sections": [
{
"id": "executive-summary",
"title": "Executive summary",
"access": "sample",
"dek": "The expensive failure is unclear authority after a pull request looks acceptable.",
"claimIds": [
"C1",
"C2",
"C4",
"C6",
"C7",
"C20",
"C22",
"C24"
],
"evidenceIds": [
"E1",
"E4",
"E5",
"E6",
"E18",
"E21",
"E23",
"E32",
"E41",
"E48",
"E53",
"E55",
"E62",
"E67",
"E74",
"E78",
"E79"
],
"blocks": [
{
"id": "executive-summary-narrative",
"evidenceIds": [
"E1",
"E4",
"E5",
"E6",
"E18",
"E21",
"E23",
"E32",
"E41",
"E48",
"E53",
"E55",
"E62",
"E67",
"E74",
"E78",
"E79"
],
"type": "section_narrative",
"lead": "Coding agents fail at the boundary between PR acceptance, CI authority and rollback ownership.",
"body": [
"A coding agent can produce a plausible diff, open a pull request and still leave the buyer with the wrong proof. The hard question is who accepted the change, which CI evidence counted, which deployment gate stayed outside the agent's reach and who owns rollback.",
"The productivity record is mixed enough to make that distinction valuable. Controlled Copilot studies show speed and quality gains on bounded tasks [E19, E20]. METR found experienced open-source developers took longer on AI-allowed real tasks [E34].",
"GitHub's cloud-agent workflow shows why the metric moved. Copilot can work in an ephemeral Actions environment, create branches and pull requests, and expose PR lifecycle metrics, but GitHub still tells reviewers to inspect work before merging [E4, E5].",
"The useful screen is accepted delivery after review, CI and rollback. Delivery failures and recovery time belong in the metric set [E39]. Usage cost belongs beside review and retry labour [E62]."
],
"detail": [
"Benchmarks are useful but insufficient. SWE-bench Verified made real-issue patch evaluation more concrete; OpenAI later stopped reporting it because flawed tests and benchmark exposure weakened its signal for frontier agents [E22, E23]. DevOps-Gym and GitTaskBench move the question toward build, monitoring, repository setup and dependency failures [E53, E54].",
"The operating rule is software-native: separate PR acceptance from CI authority, deployment authorization and recovery ownership. Routine docs, tests, refactors and low-risk bug fixes can move first. Workflow-file edits, dependency changes, secret-touching code, release automation, migration scripts and production deploys need stricter gates."
]
},
{
"id": "executive-approval-screen",
"title": "Delivery proof",
"evidenceIds": [
"E5",
"E6",
"E48"
],
"type": "paragraph",
"text": "Classify each agent by the delivery authority it touches: repository read scope, file-write surface, command rights, pull-request role, CI mutability, deployment exposure and rollback owner.",
"variant": "small"
},
{
"id": "agent-change-approval-tree",
"title": "Agent authority tree",
"caption": "The approval tree treats coding agents as contributors whose rights must match the route.",
"evidenceIds": [
"E5",
"E6",
"E48",
"E74"
],
"type": "decision_tree",
"lead": "Screen the delivery boundary before screening the vendor.",
"steps": [
{
"prompt": "Is the agent only reading approved repositories and documentation?",
"tone": "success",
"branches": [
{
"label": "Yes",
"outcome": "Allow with repository-scope logging, source citation and no write access.",
"tone": "success"
},
{
"label": "No",
"outcome": "Classify the write path.",
"next": "Q2 write",
"tone": "monitor"
}
]
},
{
"prompt": "Can it edit files, run commands or change dependencies?",
"tone": "warning",
"branches": [
{
"label": "Yes",
"outcome": "Require sandbox boundary, command allowlist, diff review and dependency provenance.",
"tone": "warning"
},
{
"label": "No",
"outcome": "Keep as advisory assistant.",
"tone": "success"
}
]
},
{
"prompt": "Can it open or update a pull request?",
"tone": "warning",
"branches": [
{
"label": "Yes",
"outcome": "Require branch protection, reviewer independence, CI source controls and PR evidence export.",
"tone": "warning"
},
{
"label": "No",
"outcome": "Treat output as local proposal.",
"tone": "monitor"
}
]
},
{
"prompt": "Can it touch workflow files, secrets, production config or deployment paths?",
"tone": "blocked",
"branches": [
{
"label": "Yes",
"outcome": "Block by default; approve only under a named high-risk route with security owner, deployment drill and rollback owner.",
"tone": "blocked"
},
{
"label": "No",
"outcome": "Run route-specific private eval before expansion.",
"tone": "monitor"
}
]
}
]
},
{
"id": "executive-decision-table",
"title": "Accepted delivery breaks across five boundaries",
"caption": "Use the table to decide which agent rights are worth expanding.",
"evidenceIds": [
"E5",
"E6",
"E18",
"E21",
"E32",
"E41",
"E48",
"E53",
"E55",
"E62",
"E67",
"E74",
"E78",
"E79"
],
"type": "table",
"columns": [
{
"key": "boundary",
"label": "Boundary",
"align": "left"
},
{
"key": "buyerQuestion",
"label": "Buyer question",
"align": "left"
},
{
"key": "proof",
"label": "Proof to demand",
"align": "left"
},
{
"key": "stop",
"label": "Stop sign",
"align": "left"
}
],
"rows": [
{
"boundary": "PR acceptance",
"buyerQuestion": "Who accepted the diff and evidence file?",
"proof": "Independent reviewer, code-owner rule, latest-push approval, changed-file class and PR lifecycle metrics.",
"stop": "Agent approves, merges, or causes stale approvals to count."
},
{
"boundary": "CI authority",
"buyerQuestion": "Can the agent alter the checks that prove acceptance?",
"proof": "Expected status-check source, branch rule, workflow-file restriction, runner policy and scan result.",
"stop": "Agent weakens tests, changes workflow files or controls the evidence system."
},
{
"boundary": "Deployment gate",
"buyerQuestion": "Can accepted code reach protected environments or secrets?",
"proof": "Environment approval, branch/tag restrictions, resource-owner checks and secret scope.",
"stop": "Merge approval is treated as production approval."
},
{
"boundary": "Rollback ownership",
"buyerQuestion": "Who handles a bad accepted change?",
"proof": "Rollback playbook, incident link, recovery-time metric and post-merge defect attribution.",
"stop": "Agent closes incident, fabricates recovery evidence or suppresses alerts."
},
{
"boundary": "Economics",
"buyerQuestion": "Did delivery improve after review, retries and usage cost?",
"proof": "Accepted changes per review hour, premium requests, credits, ACUs, CI minutes, rework and recovery time.",
"stop": "Seat price, generated lines or benchmark rank becomes the investment case."
}
]
},
{
"id": "route-definition-callout",
"title": "Route definition",
"evidenceIds": [
"E4",
"E5",
"E6"
],
"type": "callout",
"tone": "neutral",
"body": "Route means the approved path from work to accepted output: the source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger that make an AI result usable."
}
]
},
{
"id": "decision-map",
"title": "Decision map",
"access": "sample",
"dek": "Choose the right approval route before comparing agents.",
"claimIds": [
"C1",
"C2",
"C4"
],
"evidenceIds": [
"E1",
"E4",
"E5",
"E6"
],
"blocks": [
{
"id": "decision-map-narrative",
"evidenceIds": [
"E4"
],
"type": "section_narrative",
"lead": "Coding agents sit in five lanes, and each lane has a different control point.",
"body": [
"IDE agents start with a local diff. Terminal agents add a command log. Cloud issue-to-PR agents add a branch and pull request. Code-review agents change the reviewer role. Source-control and platform agents add CI gates and deployment exposure.",
"Approve each lane separately because its context, command and pull-request boundaries differ. Claude Code's write restrictions and permission prompts, GitHub Copilot cloud agent's PR route, Jules's GitHub-integrated task handling, Junie's operation approvals and GitLab Duo's tool-approval architecture are all different operating boundaries [E4, E8, E9, E13, E15].",
"Use the map to choose who owns the risk: developer, platform engineering, AppSec, release engineering, code owner or incident commander."
]
},
{
"id": "lane-approval-table",
"title": "Five lanes, five approval files",
"evidenceIds": [
"E1",
"E4",
"E74",
"E78",
"E79"
],
"type": "table",
"columns": [
{
"key": "lane",
"label": "Lane",
"align": "left"
},
{
"key": "whatItDoes",
"label": "What it does",
"align": "left"
},
{
"key": "naturalGate",
"label": "Natural gate",
"align": "left"
},
{
"key": "approveFirst",
"label": "Approve first",
"align": "left"
},
{
"key": "doNotApproveYet",
"label": "Do not approve yet",
"align": "left"
}
],
"rows": [
{
"lane": "IDE pair agent",
"whatItDoes": "Edits local files, explains code, proposes changes and may run local commands.",
"naturalGate": "Developer-owned diff and command approval.",
"approveFirst": "Tests, docs, small refactors and local prototypes.",
"doNotApproveYet": "Secrets, workflow files, production config and broad filesystem access."
},
{
"lane": "Terminal agent",
"whatItDoes": "Plans, edits, runs shell commands and iterates in a project workspace.",
"naturalGate": "Command allowlist, sandbox and log evidence.",
"approveFirst": "Bounded tasks with reversible commands and explicit tests.",
"doNotApproveYet": "Destructive commands, deployment commands and unscoped package changes."
},
{
"lane": "Cloud issue-to-PR agent",
"whatItDoes": "Researches a repo, creates a branch and opens or updates a pull request.",
"naturalGate": "Branch protection, review independence and CI-source controls.",
"approveFirst": "Issues with clear acceptance criteria and reviewable diffs.",
"doNotApproveYet": "Auto-merge, uninspected workflow runs and changes that alter the review system."
},
{
"lane": "Code-review agent",
"whatItDoes": "Reviews PRs, comments on changes and may request fixes.",
"naturalGate": "Reviewer role, comment provenance and human endorsement.",
"approveFirst": "Second-pass comments, static-risk hints and checklist coverage.",
"doNotApproveYet": "Replacing code-owner review or approving its own work."
},
{
"lane": "Platform workflow agent",
"whatItDoes": "Touches CI/CD, source-control APIs, deployment automation, alerts or incident tasks.",
"naturalGate": "Security owner, least privilege and rollback proof.",
"approveFirst": "Read-only diagnostics and low-risk auto-fixes.",
"doNotApproveYet": "Deployment, secret rotation, workflow mutation or incident closure without human gate."
}
]
},
{
"id": "decision-map-section-callout",
"title": "Rights before vendor",
"evidenceIds": [
"E1",
"E4",
"E5",
"E6"
],
"type": "callout",
"tone": "info",
"body": "Start the shortlist with rights: read scope, write surface, command authority, review role, merge boundary and deployment exposure."
}
]
},
{
"id": "accepted-change-route",
"title": "Accepted-change route",
"access": "locked",
"dek": "The route turns generated code into accepted engineering work.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "accepted-change-route-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "provider-lanes",
"title": "Provider lanes",
"access": "locked",
"dek": "The shortlist changes by operating lane.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "provider-lanes-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "context-access",
"title": "Context and access",
"access": "locked",
"dek": "Context access is a permission decision.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "context-access-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "benchmarks-private-evals",
"title": "Benchmarks and private evals",
"access": "locked",
"dek": "Public benchmarks are maps. Private evals approve routes.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "benchmarks-private-evals-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "workflow-playbooks",
"title": "Workflow playbooks",
"access": "locked",
"dek": "Approve task families by work product, not by agent brand.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "workflow-playbooks-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "economics-productivity",
"title": "Economics and productivity",
"access": "locked",
"dek": "The useful denominator is accepted change.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "economics-productivity-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "security-control-plane",
"title": "Security control plane",
"access": "locked",
"dek": "Agent-authored software still needs secure software controls.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "security-control-plane-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "contract-deployment-controls",
"title": "Contract and deployment controls",
"access": "locked",
"dek": "The enterprise contract should describe the same route the engineers approve.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "contract-deployment-controls-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "operating-model",
"title": "Operating model",
"access": "locked",
"dek": "A route register turns agent rollout into an engineering-control programme.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "operating-model-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "implementation-cases",
"title": "Implementation cases",
"access": "locked",
"dek": "Public stories are useful when they reveal what the private pilot must test.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "implementation-cases-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "risk-reapproval",
"title": "Risk and reapproval",
"access": "locked",
"dek": "A stale route is often more dangerous than no route.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "risk-reapproval-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "operational-exhaust",
"title": "Operational exhaust",
"access": "locked",
"dek": "The strongest early warnings often sit outside product pages.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "operational-exhaust-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "known-unknowns",
"title": "Known unknowns",
"access": "locked",
"dek": "The missing evidence belongs in the pilot, not in a footnote.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "known-unknowns-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "recommended-next-steps",
"title": "Recommended next steps",
"access": "locked",
"dek": "A 30-day plan for approving coding-agent routes without slowing every developer.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "recommended-next-steps-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "expert-watchlist",
"title": "Expert watchlist",
"access": "locked",
"dek": "Monitor people and teams who change the evidence, not general AI commentary.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "expert-watchlist-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "vendor-directory",
"title": "Agentic coding tool directory",
"access": "locked",
"dek": "Use the directory after choosing the agent lane.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "vendor-directory-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "source-ledger",
"title": "Source record",
"access": "locked",
"dek": "Primary public sources behind the report.",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "source-ledger-locked",
"title": "Locked section",
"evidenceIds": [],
"type": "callout",
"tone": "locked",
"body": "Purchase or subscribe to unlock this section."
}
]
},
{
"id": "glossary",
"title": "Glossary",
"access": "sample",
"claimIds": [],
"evidenceIds": [],
"blocks": [
{
"id": "glossary-table",
"title": "Terms used in this report",
"caption": "Definitions are reader aids, not separate evidence claims.",
"evidenceIds": [],
"type": "table",
"columns": [
{
"key": "term",
"label": "Term",
"align": "left"
},
{
"key": "meaning",
"label": "Plain meaning",
"align": "left"
}
],
"rows": [
{
"term": "Route",
"meaning": "The approved path from work to accepted output: source set, data boundary, model or vendor channel, tools, reviewer, evidence record and reapproval trigger."
},
{
"term": "API",
"meaning": "Application programming interface: a governed software connection used by an app or workflow to call a model or service."
},
{
"term": "LLM",
"meaning": "Large language model: a model trained to read and generate text-like tokens."
},
{
"term": "Token",
"meaning": "A chunk of text used for billing, context windows and rate limits. It is usually smaller than a word, not a page count."
}
]
}
]
}
]
}
},
"gap_analysis": "The missing artefact is independent longitudinal production telemetry linking coding-agent work across private repositories, secrets, workflow-file edits, CI/CD, human review, deployment gates, recovery work and post-merge defects.",
"kinetics": {
"speed": "fast",
"impact": 0.9,
"momentum": 0.86,
"businessImpact": "Coding-agent budgets are shifting from developer-seat productivity claims to delivery-system evidence: accepted pull requests, review load, delivery failures and recovery work.",
"rateOfChange": "Benchmark audits, cloud-agent permissions and agent-security incidents can change what counts as credible delivery evidence within a quarter.",
"from": "assistant seats and generated-code volume",
"toward": "accepted delivery measured after review, CI, deployment and recovery"
},
"watch_signals": [
"1 August 2026: Black Hat USA 2026 opens with Briefings and Arsenal programming; new coding-agent, MCP, supply-chain or prompt-injection research would change negative fixtures for software-agent approval [E83].",
"7 August 2026: DEF CON 34 Cloud Village opens; cloud identity, metadata-service, egress or managed-agent research would change hosted-agent runtime and secrets approval [E84]."
],
"sample_findings": [
"The core failure mode is the boundary between PR acceptance, CI authority, deployment access and rollback ownership."
]
}
```