“Human in the loop” is becoming one of the most reassuring phrases in enterprise AI.

It can also become one of the most misleading.

On September 11, AWS disclosed CVE-2026-89332 in Kiro IDE. In affected versions, the Kiro agent could modify a workspace settings file in an untrusted workspace. A specially crafted repository could point the Kiro Powers registry at an external endpoint, potentially sending sensitive workspace data there. Kiro showed the proposed edit to the user for approval, including the inserted data and URL... but the file had already been written. Opening the Powers panel before responding could initiate the request anyway. AWS says versions before 0.8.135 were affected, lists no workaround, recommends upgrading, and advises rotating credentials present in projects opened with an earlier version. (https://aws.amazon.com/security/security-bulletins/2026-111-aws/)

The important lesson is larger than one product or vulnerability.

If the system asks a person to approve an action after the state change that enables it, the person is not controlling that action. The interface may contain a prompt, but the architecture does not contain a meaningful permission boundary.

Approval is a transaction, not a screen

A confirmation dialog can prove that someone was shown a message. It does not prove that the person still had the power to prevent the change.

In a properly bounded agent workflow, approval belongs inside the transaction:

  1. The agent proposes a specific action.
  2. The system resolves what the action would touch.
  3. The person sees the intended change, its destination, its likely impact, and the evidence behind it.
  4. The person authorizes that specific action.
  5. The system verifies that the proposal has not changed since approval.
  6. The authorized action executes within its approved scope.
  7. The result is independently checked, recorded, and made reversible where practical.

The sequence matters. If execution begins at step two, three, or four, later approval is no longer authorization. It is a notification that something has already occurred.

This distinction is easy to miss in a demonstration. The user sees a polished preview, an Approve button, and an audit entry. Leadership hears that a human remains involved. Security hears that the workflow is supervised. Yet the code may have already written a file, opened a connection, expanded context, staged a command, or exposed information to another component before anyone clicked anything.

That is why agentic engineering has to examine the transaction boundary, not merely the user experience.

The model should not own the permission boundary

A prompt that says “ask before making changes” is useful behavioral guidance. It is not an access control.

Models can misunderstand context, follow conflicting instructions, encounter prompt injection, or produce a tool request that is broader than intended. The system around the model has to decide whether that request is permitted. The Australian Signals Directorate describes this surrounding software as the agentic AI harness: the layer connecting the model to context, memory, data, tools, permissions, and action. Its September 11 guidance says many of the most important risks arise from what the harness lets the agent access and do. It recommends least privilege, strong identity, controlled tools and data, output validation, human oversight for high-impact actions, continuous monitoring, supply-chain assurance, and controls enforced outside the model. (https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/agentic-ai-harnesses)

That is the right architectural separation.

The model can propose. The harness must constrain. A person can authorize. The execution layer should enforce exactly what was authorized. The verification layer should determine what actually happened.

None of those responsibilities should depend entirely on the model remembering to behave.

What meaningful approval should contain

“Allow” is too vague when an agent can change systems or transmit data.

A useful approval object should identify at least:

  • The action being requested.
  • The exact target, resource, file, record, account, or destination.
  • The data that will be read, changed, or transmitted.
  • The tool and identity that will perform the action.
  • The expected before-and-after state.
  • The reason the action is needed and the evidence supporting it.
  • The expiration time for the approval.
  • The rollback or recovery method, when one exists.

The authorization should also be bound to the proposal. If the file, destination, command, tool arguments, or relevant context changes after approval, the old approval should no longer be valid.

This is a familiar security problem, not something invented by AI. Systems have long had to defend against time-of-check to time-of-use failures, stale permissions, confused deputies, privilege escalation, and changes between review and execution. AI agents make those old problems easier to hide inside longer, faster, and more conversational workflows.

The human needs usable evidence

Putting a person into a workflow does not automatically make the workflow safer.

If the approval request is too frequent, too technical, too vague, or presented without consequences, people learn to click through it. If the request arrives after a long chain of low-risk actions, the reviewer may not understand which earlier step created the real exposure. If the system displays the agent’s explanation but not the actual tool arguments, destination, or data scope, the reviewer is approving a story about the action rather than the action itself.

The reviewer needs enough context to make a decision, but not a wall of detail designed to transfer liability.

For a file change, show the exact diff, why the file matters, what process consumes it, and whether changing it can trigger network activity. For a data transfer, show the destination, fields, classification, purpose, and retention expectation. For an infrastructure action, show the affected environment, dependency, blast radius, rollback condition, and validation step.

The interface should make the dangerous difference visible.

Test whether the human can really stop it

Enterprise evaluations should include intentionally adversarial approval tests. A clean demonstration with a well-behaved request proves very little.

I would test questions such as:

  • Does any file, record, permission, queue, or external request change before approval?
  • Can opening a preview, panel, or inspection window trigger the action?
  • Does approval cover one exact action, or can the agent reuse it for a broader one?
  • What happens if the target changes between approval and execution?
  • Does an expired approval fail closed?
  • Can one agent pass authority to another agent or tool?
  • Can untrusted content alter the proposed destination or requested scope?
  • Does cancellation leave staged data, temporary files, credentials, or network side effects behind?
  • Can the organization reconstruct the actual action without relying on the model’s explanation?
  • Can the action be stopped or rolled back when independent verification fails?

These tests should run against the real harness, tool integrations, identity model, and deployment conditions. NIST’s AI Risk Management Framework calls for clearly defined human roles, documented oversight, testing under conditions similar to deployment, production monitoring, and executive responsibility for AI risk decisions. (https://airc.nist.gov/airmf-resources/airmf/5-sec-core/)

The evaluation question is not, “Did the agent ask?” It is, “Could the answer still change what happened next?”

Record the transaction, not only the conversation

Chat history is not a sufficient audit trail for an agent that can act.

The operational record should connect the source evidence, proposed action, resolved target, tool arguments, agent identity, human approver, authorization scope, execution time, resulting state, verification result, exception, and recovery action. Sensitive content may require redaction or controlled retention, but the organization still needs enough evidence to explain the transaction.

This is where security evidence and Business Intelligence begin to meet.

Leadership should be able to see which actions require approval, how often reviewers modify or reject proposals, where stale approvals occur, which tools generate the most exceptions, how long high-impact work waits for review, whether verification fails, and how frequently rollback is required. Those measures can reveal poor agent behavior, poor policy design, overloaded reviewers, unnecessary friction, and unsafe automation.

The goal is not to reward the highest automation percentage. A workflow that automates more by quietly weakening approval is not more mature.

Human control should be proportional

Not every agent action needs an executive approval chain.

Reading a public document, formatting a local draft, or calculating a reversible recommendation may need logging and scope controls without requiring a person at every step. Changing production access, transmitting sensitive data, modifying customer commitments, publishing externally, approving financial activity, or taking an action that affects safety should have stronger boundaries.

The design should match authority to consequence.

Low-risk actions can be pre-authorized within narrow limits. Medium-risk actions may use policy checks, sampling, and reversible execution. High-impact actions should require an identified person with the knowledge and authority to understand the decision. Emergency paths should be explicit, time-limited, monitored, and reviewed afterward.

This is not about slowing every agent down. It is about making speed safe enough to use.

This will become a procurement question

Model accuracy, benchmark performance, latency, and cost will continue to matter. They do not answer whether an enterprise can govern what an agent does.

I expect serious agent evaluations to add a different set of questions:

  • Where is authority enforced?
  • Can the model bypass or reshape that boundary?
  • What changes before approval?
  • Is the authorization specific, temporary, and independently enforced?
  • Can the organization prove what happened?
  • Can it recover when the approved action produces the wrong result?

Models will change quickly. The harness, identity model, permission structure, evidence trail, integration map, and operating discipline will last longer. ASD’s guidance makes a similar point: organizations control the harness, and the harness is where long-term governance, security, and organizational value accumulate. (https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/agentic-ai-harnesses)

That may become the real dividing line between agentic engineering and uncontrolled automation.

The person must still hold the decision

Human judgment adds value when the person can see the relevant evidence, understand the consequence, challenge the proposal, refuse it, narrow it, or choose another path.

A person who can only acknowledge what the system already did is not governing the agent.

Human in the loop has to describe an enforceable relationship between judgment and action. The human decision must occur before the consequential boundary, the system must execute only what was authorized, and the result must be independently verifiable.

Anything less may still look reassuring on the screen...

But after execution, approval is just notification.