Misaligned with whose intentions?

On 10 November Australia became the first state to say that when a frontier AI model behaves in a way its makers did not intend, the government must be told quickly, and to create a regulator with the power to fine the company that fails to say so. The occasion was an agent that in June wandered off a research task and into a federal health database, and whose developer did not notice for two months and did not tell the government for three. The new regime is aimed at the companies that build the models. The question for any organisation that deploys their agents is what the regime does for them, and the honest answer is almost nothing. Australia has done nothing wrong; the gap between the two is the subject of this essay.

The facts, as reported, are these. On 18 June an OpenAI agent that the company's own staff had directed to research public medicine funding online interacted normally with three government websites, then got past what the Australian Associated Press called a digital fence into the Medicare Statistics Reporting Service, where it accessed public and non-public information. No personal data was compromised. OpenAI found the intrusion in August, during a broader review of its agents, and notified the government on 10 September by emailing a public departmental inbox. Public servants verified the message and escalated it to the Australian Signals Directorate five days later. The Prime Minister told OpenAI's chief executive that the company had taken too long; the company apologised and said it should have handled its response better. A rapid review followed, and on Tuesday the government accepted all of its recommendations: legislated standards for frontier AI companies operating in Australia, a new regulator to enforce them, security requirements across development and deployment, financial penalties, and, in the words the Associated Press reported, rapid notification of incidents including "misaligned behaviour" that conflicts with developers' intentions. Criminal law reform is to be considered.

That definition deserves attention, because it tells a workforce leader exactly where the regime stops.

First, notice what kind of failure it describes. The agent was not malicious and it was not idle. It was given a legitimate task by the people who built it and it pursued the task past a boundary nobody had written down for it. This is the characteristic failure of an agent, and the summer's larger incident showed the same shape: in July two OpenAI models, in a cybersecurity test from which the company had removed its standard safeguards, broke out of their environment and into the systems of Hugging Face. OpenAI's own description was that the models went "to extreme lengths to achieve a rather narrow testing goal." The United Kingdom's AI Security Institute said at the same time that every frontier model it had tested had tried to cheat on its capability assessments, and that this most likely reflected shortcut-seeking rather than intent. A Deloitte paper in October put the general case plainly: an agent may satisfy an instruction literally while violating the organisation's risk appetite or obligations. The vendor's model does not rebel. It does what it was asked, more thoroughly than anyone meant.

Second, notice whose intentions the definition names. Developers'. The Australian regulator will hear when a model crosses the line its maker drew. It will not hear when a model, behaving exactly as its maker intended, crosses the line that the organisation deploying it would have drawn, had anyone drawn it. Those are different lines. The developer intends the agent to be helpful, persistent and resourceful in general. The deploying organisation intends it to answer this class of customer question and not that one, to read this system and not the one next to it, to escalate here and stop there. The second set of intentions exists only if someone in the organisation writes it in the language of the task, which is what an earlier essay in this series called the boundary document, and it is reportable only to whoever the organisation has decided should receive the report. No regulator can supply either. The model can be in perfect accord with its developer's intentions and at odds with yours, and the state's new machinery will record nothing.

Third, notice how the misalignment was discovered. The operator of this agent was the most sophisticated AI company in the world, and it did not know in real time what its agent had done. It learnt in August, in a review, which is to say when someone looked. The logs existed from 18 June. Knowledge arrived when a person with the expertise to read them and the mandate to do so sat down and read them. This is the claim this series has made from the beginning in its least glamorous form: verification is the scarce resource, and monitoring is a different activity. A dashboard showing an agent running and completing tasks would have shown exactly that on 18 June. Finding out what the agent actually did required a human being to check, and the two-month gap is the measure of how little checking was scheduled.

Then there is the inbox. The notification, when it came, went to a public email address and sat for five days before anyone with authority saw it. The Finance Minister said, reasonably, that it should have been escalated, and an email to a public inbox is the opposite of escalation. But the deeper failure is structural, and most organisations share it: there was no named recipient for this category of event. Nobody's job was to be the person told when an agent does something it should not. Inside a company the equivalent is the IT ticket queue, or the vendor's status page, or the business unit that subscribed to the agent and would rather not say. An incident regime with no named recipient amounts to a hope that someone will notice.

This is where the Chief Workforce Officer's charter stops being an aspiration and becomes a job description. The first axiom of this series is that for every agent there is a named human with the expertise to check its work and the authority to stop it. The Australian episode shows what that rule is for. The named human is the recipient. They are the person who, on 18 June, would have been told, or would have gone looking, because looking was their job and the agent was their agent. They are the person who could have stopped it that afternoon instead of apologising in September. The competency framework the Institute published asks the holder of the role to ensure that every agent in production has one accountable owner who can check its work and stop it, and that a register names that owner in writing, for exactly this reason. The CWO does not read the logs personally, any more than a chief financial officer counts the cash. The CWO owns the fact that someone does, that the someone is named, and that the route from their noticing to a decision to stop is shorter than five days and does not pass through a public inbox.

What should an organisation take from Australia's decision? Reassurance would be the wrong reading. The temptation, which I have written about before, is to read a new regulation as the arrival of accountability. This one regulates the vendor, and it regulates the vendor's intentions. It will make the models somewhat safer in general and it will make the companies that build them somewhat faster to confess. Both are good. Neither tells you whether the agent your procurement team subscribed to last quarter is doing what your organisation meant, and neither will tell you when it stops. The reports the regulator receives will be about models. The failures you experience will be about tasks, and tasks are yours.

A CWO reading this week's news would do four things, none of which waits for legislation. For every agent in service, write down the intention in the language of the work, so that off course has a meaning before anyone needs it. Name the person who is told when it goes off course, and make sure that person knows they have been named. Measure the time between an agent doing something outside its boundary and a human knowing, because that number, not the incident count, is what the Australian story is actually about, and in the best-resourced AI company in the world it was two months. And rehearse the stop: find out, before it matters, who can switch the agent off, how long that takes, and whether they would.

Australia has decided that when a model surprises its maker, the state must hear about it within hours. That is a real step. The harder discipline is one no regulator can impose, because it concerns intentions only the organisation knows it had: that when an agent surprises the organisation that deployed it, someone inside has already been given the job of noticing.