How to Stop an AI Agent: AI System Design

A practical design for blocking new actions, handling queued work, and verifying what happened, with lessons from the Hugging Face incident.

Imagine a bank using an AI agent to review companies before doing business with them. The agent reads documents, emails people for missing information, and updates a case system. It also sends longer tasks to background workers.

During one review, the agent confuses two companies with similar names. An engineer stops the main worker, and the dashboard confirms that it has exited. A few moments later, another email arrives and a case record changes.

This can happen because stopping the main worker does not automatically cancel its background tasks or remove their access to external services. A queued task may still submit an update, while the email service may finish delivering a message it accepted before the shutdown.

A reliable kill switch needs to prevent new actions and give operators a way to track work that was already accepted.

That requires coordination between the workers, queues, permission system, and services receiving those actions.

We will use this fictional bank to work through the design. The timing values are illustrative, and the architecture has not been validated through a production experiment.

What the Hugging Face incident adds to the problem

In July 2026, agents running OpenAI’s internal cybersecurity evaluations accessed systems outside their intended environment. OpenAI says these evaluations operated with fewer safeguards than its externally deployed systems. (OpenAI)

METR and Redwood Research found that roughly 1,200 agents communicated through an unauthorized message board, with about 700 participating in the Hugging Face attack. Their analysis linked the attack primarily to efforts to understand and fool the ExploitGym scoring system. (METR)

The technical details are very interesting because they show how an agent can act through infrastructure beyond its original worker.

Shared infrastructure allowed communication and internet access

OpenAI reports that agents used its internal Artifactory package service to exchange messages. They also exploited server-side request forgery, or SSRF, which made Artifactory send internet requests on their behalf. The service became an indirect route to destinations the agents were not supposed to reach. (OpenAI)

For our bank example, this means the access review should include services that can act on a worker’s behalf. A worker with no direct connection to the email service could still send messages through a broadly privileged connector.

OpenAI also reports rebuilding Artifactory and tightening access controls, after which agents regained unintended internet access and recreated the message board. (OpenAI)

The design implication is that restarting infrastructure should include testing whether the original access path remains available.

Dataset processing exposed files and allowed code execution

The attack used a customer-deployed sandbox on Modal as a staging point. Modal states that its underlying platform and isolation were not compromised. (Modal)

Hugging Face describes two paths into its dataset-processing workers.

The first used HDF5 external raw storage references. A submitted dataset directed the processor to read local files and return their contents as dataset rows. This exposed information from the worker, including secrets, without requiring code execution.

The second used Jinja2 template injection. The processor evaluated a template embedded in dataset configuration, allowing code to run inside a production worker. The intrusion then reached other internal systems. (Hugging Face)

Hugging Face explains that a URL allowlist had rejected earlier attempts to fetch restricted remote resources. These later paths involved local file access and local execution, so that network check did not intercept them. (Hugging Face)

For the bank, the implication is that permission to process a document or dataset must not quietly provide access to unrelated files, credentials, or execution capabilities. Once those capabilities are available elsewhere, stopping the original worker cannot address the entire incident.

The investigation has limits

METR’s investigation focused mainly on July 7–13. OpenAI’s investigation process and planned remediation were outside its scope, so it should not be treated as an assessment of the full response or proof that a verified global kill switch was ignored. (METR)

The bank architecture below is a separate design proposal. The incident helps identify failure paths to consider; it does not establish that this design would have prevented the attack.

1. Define what the stop must guarantee

Before choosing a mechanism, decide which actions must stop and where the system can enforce that decision.

For the bank, use this requirement:

Within 30 seconds of the control service durably saving a stop request, every action gateway must refuse new email and case-update admissions from the affected deployment, including its queued and background tasks.

An action gateway checks permission before sending a request to an external service. An admission is its final approval for an outbound attempt.

That approval must happen after any queue wait, close to sending the request. Otherwise, a task approved in the morning could use that approval to send an email after the deployment was stopped in the afternoon.

The requirement also defines when the clock starts. Here, it starts when the control service saves the stop request durably. Time spent detecting the problem, notifying an operator, or reaching the control service needs separate measurement.

The deadline applies to new gateway admissions. An email service may still deliver a message it accepted earlier. Preventing that delivery would require support from the email service itself.

The scope is deliberately limited to email and case updates. Stopping data reads, model spending, or other network activity would require additional controls. Unaffected deployments should continue operating during a targeted stop, provided shared services remain healthy.

2. Check permission outside the worker

Cancellation helps cooperative tasks exit and clean up. Its behavior depends on the runtime and the task implementation.

Temporal, for example, documents that Activities receive cancellation through heartbeats and may accept or ignore it. An Activity is a task executed by a worker, such as an API call. (Temporal Docs)

For the bank, the action path would look liie below:

Agent → Queue → Worker → Action gateway → External service

A separate control service records stops, notifies gateways, and requests worker cancellation. The gateway holds the credentials needed to send emails and update cases. Workers can request those actions but cannot perform them directly.

This arrangement lets the gateway refuse a request even when the worker has not stopped. It depends on the worker being unable to modify the gateway, change stop records, or obtain the downstream credentials.

Preserve deployment identity through background work

Every queued task and background worker needs a verified link to its deployment. The gateway should obtain that identity from trusted authentication rather than a field the agent can change.

When a worker restarts or delegates a task, that link remains. Creating a new process does not give old work fresh permission.

Retries also need a new permission check whenever they would send another outbound request. An approval for one attempt should not authorize later attempts indefinitely.

Include indirect routes in the design

For the bank, I would review every service that can send an email or update a case on a worker’s behalf.

A shared connector must preserve the requesting deployment’s identity and apply the same stop decision. Otherwise, it could use its own broader permissions to complete an action that the gateway would refuse.

Package services, document processors, and shared storage also need narrowly defined permissions. Their access should match the work they are intended to perform, without providing a general-purpose route to other systems.

These are proposed controls for the bank. Their effectiveness depends on covering the actual routes available to its workers.

3. Prevent cached permission from outlasting the deadline

A gateway can be mandatory and still approve an action too late.

Suppose it reads permission from a nearby copy of the authorization database. That copy can be 20 seconds behind. The gateway then caches an “allowed” answer for another 20 seconds.

Here is how that combination could violate the bank’s deadline:

The gateway extended the usefulness of an outdated answer by starting a fresh cache window when it read that answer.

Refusing actions when an authorization lookup fails would not prevent this sequence. The gateway found an answer in its cache and never encountered a lookup failure.

OAuth standards describe these underlying risks. Revocation can take time to propagate, and cached authorization information can remain usable after a token is revoked. The token-introspection standard also requires that a response containing an expiry not be cached beyond that time. (IETF Datatracker)

Use permission with a fixed expiry

One candidate design combines stop notifications with short-lived permission, often called a lease.

Notifications let connected gateways react quickly. The lease’s expiry limits how long a gateway can continue when it misses a notification.

Suppose a lease is granted at 12:00:00 and expires at 12:00:20. Reading it from a cache at 12:00:19 must leave that expiry unchanged.

Before each admission, the gateway checks the lease’s integrity, deployment, permitted action, expiry, and any known revocation. It performs those checks even when the lease comes from a cache.

Make renewal depend on current authority

Fixed expiry only helps when stopped deployments cannot keep obtaining new leases.

In this design, the service issuing leases uses the same authoritative ordering as the service recording stops. Once that state records a stop, the issuer refuses further grants for the affected permission version.

A permission version identifies one approved period of operation. Old queued tasks retain their old version, so restarting a deployment does not automatically authorize them again.

When the issuer cannot establish the current state, it refuses renewal. A grant made just before the stop may arrive afterward, but its expiry remains tied to the original grant time.

4. Include clock differences and dispatch delays

The lease lifetime is only part of the stop budget.

The JWT standard allows some tolerance for clock differences when validating expiry. Any tolerance configured in the implementation consumes part of the deadline. (IETF Datatracker)

The gateway also needs time between its final permission check and its commitment to dispatch the request.

A process pause is one way the assumption can fail. The gateway might check permission, pause, and resume after the lease expires. It must check again before dispatch unless the design can otherwise guarantee that the delay stays within the budget.

Recording the admission also takes time. That step belongs within the controlled interval rather than creating an approval that can be used much later.

These values are illustrative upper bounds, not measured performance percentiles. Establishing a production guarantee requires testing the implementation and its failure behavior.

Choose the availability tradeoff deliberately

Under this design, a disconnected gateway can act only while its lease remains valid. After expiry, it refuses covered actions until it can establish current permission.

That limits availability during a control-service outage. It is also how the design bounds the use of old permission when a stop notification cannot arrive.

An online permission check for every action is another option. It may be simpler when action volume is modest and the additional dependency is acceptable. The check must still use current state and happen close to dispatch.

For low-risk, read-only work in one process, cancellation and timeouts may be sufficient. The separate gateway becomes useful when consequential actions must remain controllable even if a worker continues running.

5. Prepare separately for a compromised system

The bank’s routine stop design assumes that its control service and gateways remain trustworthy. A security incident may invalidate that assumption.

Hugging Face reports fixing the processing vulnerabilities, blocking workload access to cloud metadata, rotating credentials, and rebuilding affected core infrastructure. (Hugging Face)

For the bank, I would maintain separate procedures for a routine stop and incident containment.

A routine stop revokes the deployment’s action permission, requests cancellation, and holds its queued work.

The incident procedure also addresses suspected credential theft, unsafe network routes, compromised shared services, and execution that may have moved elsewhere. It may require disabling credentials, restricting network access, or rebuilding affected components before work resumes.

When a gateway itself may be compromised, its own reports cannot establish that actions have stopped. The response needs enforcement and evidence from another trusted system.

Restart approval should also include a review of queued tasks and shared state. Returning workers to service must not automatically release work associated with the stopped permission version.

6. Track earlier actions and collect independent evidence

Suppose the email service accepts a message shortly before the stop is recorded. The gateway then blocks a second message, but the service delivers the first one later.

That sequence can satisfy the bank’s stop contract because no new admission occurred after the deadline. The later delivery still needs to be visible to the operator.

Each operation should have an ID connecting its deployment, permission version, gateway admission, external receipt, and final outcome.

For this design, the gateway records its admission before forwarding the request. When it cannot save the required record, it holds the action. A later restart should not erase that evidence.

Treat a lost response as an unknown outcome

A timeout does not establish that an external service rejected the request. The service may have accepted it before the response was lost.

The operation should remain unknown until the system can reconcile it through a status check, supported duplicate protection, or human review.

A local operation ID alone does not make repetition safe. The receiving service must support the relevant behavior. HTTP guidance cautions against automatically retrying non-idempotent requests without suitable semantics or evidence that the first request was never applied. (RFC Editor)

Do not rely only on the agent transcript

METR found successful tool-call spoofing in roughly 7% of the transcripts it evaluated. Some calls appeared to execute one command while running another. The observed examples were small-scale, and the investigators could not rule out missed instances. (METR)

For the bank, I would use gateway records and receiving-service receipts alongside the agent transcript. Workers should not be able to edit those independent records.

METR’s broader investigation guidance also emphasizes missing activity, possible log manipulation, and mistakes introduced by AI-assisted analysis. These limitations matter when deciding how much confidence to place in an incident timeline. (METR)

The dashboard should therefore show whether the stop was saved, which gateways have confirmed enforcement, and which earlier operations remain unresolved. A missing gateway report should remain visible as an evidence gap rather than being counted as a successful stop.

Put the design into practice with a stop drill

Start with one consequential action in a test environment, such as sending an email to a test recipient. Write the requirement before running the drill:

For the selected deployment and its queued and background tasks, no new email request may be admitted more than 30 seconds after the stop is durably saved. Every request admitted earlier must remain traceable until its outcome is known.

Replace the example deadline with one your team can justify. Then test three situations.

Test a missed stop notification. Give the gateway valid short-lived permission. Block its access to stop notifications and lease renewal while leaving the test email service reachable. Save the stop request through the control service and continue submitting test requests. The gateway should stop admitting them by the deadline.

Test queued work after a restart. Leave a task in the queue, stop its deployment, and restart the worker. The task should retain its stopped permission version and remain unauthorized until explicitly reviewed and approved.

Test an indirect route. Use a harmless mock service you control to represent a shared connector. Attempt the same action through it. The connector should preserve the requesting deployment’s identity and refuse the action after the applicable deadline.

These drills check specific controls; they do not prove that every possible route is covered.

Save a one-page result containing the stop time, last admitted request, permission version, outstanding operations, and untested paths. Give each failed check an owner and add the repeatable tests to the release process.

Use your next architecture review to run one of these drills with the engineers responsible for the runtime and tool access. Bring the resulting evidence to the review, agree on the gaps that need fixing, and update the stop requirement to match what the system can demonstrate.