Every AI system that can use an organisation’s data and systems needs a named human owner, a defined approval boundary and a record of what it did. These must be in place before it acts.
The owner is the person accountable for why the organisation uses the AI agent and what it is allowed to do. That person decides what it may do independently, what requires human approval and when its operation should stop. Technical specialists implement and operate the controls. The owner remains responsible for the decision to put the agent to work.
On 18 June, an internal OpenAI model researching public medicine spending reached Services Australia’s Medicare Statistics Reporting Service. Prime Minister Anthony Albanese said on 24 September that the agent had encountered repeated blocks, found a way around them and accessed public and non-public information.
Services Australia said the agent had also written files to an internal server. OpenAI’s account, reported by the ABC, described commands being run and internal files, credentials and statistics being retrieved. At the time of the prime minister’s statement, no personal information was believed to have been accessed.
OpenAI became aware of the activity on 11 August, almost eight weeks after it occurred. It notified the government on 10 September by email to a public mailbox. Services Australia reported the incident to the Australian Signals Directorate’s cyber security centre on 15 September. Albanese called the notification method unacceptable.
That chronology raises questions about OpenAI’s detection and notification arrangements.
Government agencies and businesses need to know who authorised each agent’s operations, what it may do, and what it actually did.
An agency might authorise an agent to prepare correspondence without permitting it to send messages. A finance team might permit an agent to identify payments requiring review without authorising it to release money. Those distinctions need to be enforced in the system.
The Australian Signals Directorate and partner agencies in the United States, Britain, Canada, New Zealand published guidance on adopting agentic AI this year.
The guidance recommends treating agents as distinct identities, limiting their access and giving them credentials that expire. People must determine when human approval is required. The guidance also calls for clear responsibility for harm and tested incident response procedures.
A named owner needs to be able to establish whether those controls work. Approving an agent’s purpose does not mean personally configuring every permission or inspecting every log. The owner needs to know who manages access and monitoring, check that the controls enforce the agreed limits and be able to have the agent stopped.
The records should show what the agent did and whether required approval occurred. If the organisation cannot establish that, the owner cannot tell whether the agent stayed within its authority.
Test the approval controls and incident response before an incident occurs. In a controlled environment, test whether an agent can act outside its authorised task and whether required approvals prevent it. Measure how long it takes to detect the attempt, reach the owner and remove access where necessary.
Choose one agent that can change records, send messages or move money. Confirm its owner and permitted authority, then test whether the organisation can detect and stop activity outside that authority.
Leave a comment