The autonomy fantasy
The AI industry sells a vision of fully autonomous agents. “Set it and forget it.” “End-to-end automation.” “Zero-touch operations.” The pitch is seductive: deploy an agent, walk away, let it handle everything. The reality is less glamorous.
Every enterprise deployment that went fully autonomous has a horror story. The Slack bot that messaged 500 people at 3am because a threshold was miscalibrated. The agent that updated the wrong Jira tickets because it mismatched entity IDs across systems. The automation that closed “resolved” tickets that weren't actually resolved.
These aren't edge cases. They're the inevitable result of removing humans from critical decision points. Full autonomy without oversight is a demo, not a product. Demos don't have consequences. Products do.
Autonomous agent does something wrong at scale. Team scrambles to undo the damage. Leadership adds a manual approval step. The “fully autonomous” agent becomes a human-in-the-loop agent, but only after the incident. Build it that way from the start.
Three types of human involvement
Not all HITL is the same. There are three distinct patterns, each serving a different purpose.
1. Confirmation
Agent proposes an action. Human approves or rejects. This is the minimum viable HITL for any write operation. Example flow: Agent: “Mark CASE-4521 as blocked?” → Human: Yes → Action executed.
2. Review
Agent produces output. Human reviews before it reaches the end user or becomes permanent. Knowledge base proposals work this way: the agent discovers a pattern and proposes a KB update. Admin reviews, approves, or edits before it becomes part of the agent's knowledge.
3. Escalation
Agent recognizes it's out of its depth and hands off to a human. This requires the agent to know its own limitations, which is harder than it sounds. Example: Agent encounters 3 conflicting signals → Pages senior analyst → Human makes judgment call.
Most agent frameworks only think about confirmation. Review and escalation are equally important, and harder to get right.
Confirmation by default
In the Amodal model, every write operation requires confirmation. Not because you opted in. Because that's the default. You have to explicitly remove confirmation for a write endpoint to be auto-approved.
// Default: any POST/PATCH/DELETE endpoint requires confirmation
{
"endpoints": {
"POST /api/cases/*/status": {
"confirm": true // ← this is the default, you don't even need to set it
}
}
}// Opt-out: requires explicit admin decision, logged in audit trail
{
"endpoints": {
"POST /api/cases/*/comments": {
"confirm": false // ← admin deliberately chose this, and it's logged
}
}
}Bulk operations get extra scrutiny. More than 5 writes in a single action require itemized confirmation. Each one is listed. The human sees exactly what will happen.
The safe default is confirmed. The unsafe behavior (auto-approve) requires a deliberate, logged configuration change.
Knowledge proposals
The learning flywheel depends on HITL. An agent discovers a new pattern during a session. Maybe it finds that a specific vendor always submits invoices late in Q4. Maybe it notices that a particular alert fires every Tuesday at 2am and is always a false positive. The agent doesn't silently update the knowledge base. It proposes an update.
Admin reviews the proposal. Is this a real pattern or a one-time anomaly? Is the wording accurate? Should it be scoped to a specific tenant or shared across all tenants? If approved, the KB gets smarter for the next session. If rejected, no harm done.
Flow: Agent explores → Discovers pattern → propose_knowledge()→ Admin reviews → Approved → Next session starts smarter.
The agent never auto-approves KB updates. Every proposal goes through admin review.
Automations and HITL
Automations (scheduled agent runs) are the highest-risk surface for HITL failures. They run without a human in the session. There's no one watching when the 2am anomaly scanner kicks off.
The guardrail: automations cannot write by default. An automation can read, analyze, and report. It can send a summary to Slack. It can create a draft for review. But it cannot update a Jira ticket, close a case, or modify a record unless the admin explicitly enables write access.
{
"name": "daily-anomaly-scan",
"schedule": "0 6 * * *",
"writes_enabled": false, // ← default: read-only
"output_channel": "slack:#security-alerts"
}The trust ladder
HITL isn't permanent friction. It's a trust-building mechanism. New agent, new skill, new connection: everything confirmed. As the team gains confidence, they relax confirmation on low-risk operations. High-risk operations stay confirmed forever.
- Week 1: Everything confirmed. Every read, every write, every action. Trust is zero.
- Month 1: Read operations trusted, writes still confirmed. The team has seen hundreds of reads.
- Month 3: Low-risk writes auto-approved, high-risk still confirmed. Adding a comment to a Jira ticket? Auto-approved. Changing a ticket's status? Still confirmed.
- Forever: Bulk writes, cross-system writes, and data deletions always confirmed. Some operations never graduate.
The ladder isn't a feature. It emerges from sensible defaults and granular configuration. The admin relaxes constraints as trust grows, and the audit trail shows exactly when and why each change was made.
The goal isn't to remove humans from the loop. It's to put them at the right point in the loop. Confirming actions they should see. Reviewing knowledge that shapes future behavior. Escalating decisions that need judgment.
The human isn't the bottleneck. The human is the quality control.