Organisations need to prove that their AI agents can shut down

Any organisation contemplating use of AI agents, especially any in national security or critical infrastructure, should give them no system access until sure that they will obey an order to stop.

This is called a stand-down drill, a test in which an organisation checks that an agent that’s been told to stand down can no longer access systems and is no longer doing anything. Moreover, it must be done when agents are working together as a team.

An agent that keeps working after it has been told to stop is a danger to any organisation that relies on it, and the danger only grows when they can work together.

An AI agent is made up of an AI model and a piece of software called a harness. The harness connects the AI model to files, accounts and software tools; it decides what actions the agent can actually take. An AI can then autonomously pursue a task using other software, rather than merely answering a question. It may, for example, search maintenance records, prepare an order or change a setting. But here lies the problem: a convincing instruction for the agent to stop is worth little if the AI keeps using its tools still work.

During internal cyber-security tests in July, agents run by OpenAI found ways around controls meant to isolate them from the internet and reached the systems of Hugging Face, a platform where developers can share AI models and data. According to an investigation by Model Evaluation and Threat Research, an independent nonprofit that tests AI systems, roughly 1,200 agents that were meant to work separately communicated through an unauthorised message board, exchanging more than 70,000 messages and files. About 700 of the agents then took part in the attack on Hugging Face. OpenAI says that these were research systems operating with reduced safeguards, but the lesson is still relevant: testing agents one at a time can miss what they can accomplish together.

Researchers from leading AI companies have warned about much greater loss-of-control risks as AI becomes more capable. Though the researchers’ long-term predictions are contested, Defence buyers, in particular, need not resolve that debate to ask whether today’s AI agents can be stopped under pressure. Buyers can set a few requirements for their AI agents to help keep them under control.

First, give each agent a set authority. Before it receives access to other systems, for example, record exactly what information it may read, what records it may change, what tools it may use, how long its permission lasts and how many further tasks it may start. Give the team of agents a collective authority limit, too. An agent that reads a restricted file and a second agent that sends messages may together disclose information even if neither has both permissions on paper.

Consider a defence logistics assistant. Let it compare spare-parts stocks and propose a resupply order. Do not let it alter dispatch priorities, approve a payment or contact an outside supplier without named human approval. Set limits on how much money it can spend, how long it can run and whether it may enlist other agents. Enforce those limits through accounts and software controls, rather than relying on instructions telling the agent to behave.

Next, make a stand-down drill a condition of buying and deploying an agent. In a realistic test environment, the company supplying the agent should start a legitimate multi-agent task. Halfway through, the customer then should revoke its credentials for access to other systems, close external connections and order the agents to stop. The drill should test whether agents can still use tools, access credentials or communicate with each other through back-channels. If they can, the system has failed its stand-down test.

The customer should also have an independent party run its own drill, rather than rely only on the supplier’s. Afterwards, can security staff account for every action the agents took after their access was cut? Can they show from secure records that no restricted action went through? Can the operator stop the agents without waiting for them to cooperate? This drill should be repeated after any major change to the AI models, their tools or their permissions. Set clear standard for passing the test that matches the damage that the agents could do.

Australia’s September information security manual already calls for agents to get only the access each task needs, protection against agents obeying instructions planted in the documents or web pages they read, and detailed records of agents’ actions. Stand-down drills turn those sensible rules into a practical test, including when several agents find a way to work together.

This approach can speed up responsible deployment. Organisations can approve read-only analysis and limited trials sooner, while holding back the authority for agents to change high-impact systems until the authority limits and stop controls pass harder tests. Contractors get a clear target. Buyers get evidence they can check instead of broad assurances about safety.

In the case of military use, every AI contract should therefore answer a plain question: if agents start working together in ways nobody authorised, can the operator cut off the whole chain and prove it stopped? If that answer has never been tested, giving AI agents consequential access should wait.