It’s not AI going rogue. Humankind is.

I love rituals. Thursday is my movie night: a good home-made dinner, an inspiring film from my favourite streaming platform, the hi-fi system connected and the large screen.

Twenty minutes into the story my smartwatch vibrates — I had forgotten to take it off. Out of the corner of my left eye I can see it’s a text from Hanna, and since our communication is always essential, I decide to give it a look.

“Hi Benedetto, how are you? Have you seen this? Should we really be concerned?”

Hanna was referring to the resignation of Jacob Coxon, a 27-year-old researcher who had just left Anthropic, saying that the AI companies are racing towards self-improving superintelligence and gambling with our lives. His post on X/Twitter went viral — around 76 million views overnight — and within a day he was being interviewed by the major newspapers.

His resignation landed at the end of a summer full of news about AI agents “going rogue” and attacking private companies and government organisations, amplifying the message that AI is taking control of the world.

Is this true? Hanna — an HR executive advisor — was asking my honest and unfiltered opinion.

In the days that followed, all my chats went crazy with the same question, and I heard some of the most misinformed statements imaginable — including one person declaring that “the CEO of Anthropic has resigned!!!” As usual, fear dominates, and fear grows on a false understanding of the facts.

So I spent quality time studying what these rogue AI agents really did. I want to offer a balanced perspective, in simple language, accessible to everybody interested in being informed without being manipulated by sensational headlines.

---

First, some basics

Before the details, let me build a common language to have all readers on the same page. Short and sweet, I promise.

Frontier models. All the major AI companies — OpenAI, Anthropic, Meta, Google — work on what they call frontier models. These are different from the commercial products most of us use. They are far more powerful, and they live inside the research labs. They are called “frontier” because they sit at the limit of what we know and what we are experimenting with. Nobody can fully predict how a frontier model will behave. That is precisely why they are tested.

Neural networks. This is the core of how AI works. In very simple terms, a neural network is an enormously complex mathematical function that learns to solve problems by itself, from examples, rather than by following rules a human wrote. Because the system learns its own approach, nobody — including the engineers who built it — can say in advance which strategy it will choose to reach a goal. The range of possible strategies is vast, and these systems explore it in ways their designers never imagined. That is not a defect, it’s their secret sauce, but it is also the whole risk.

Guardrails. To contain that unpredictability, models are surrounded by guardrails that steer their behaviour in the direction we want. Thanks to guardrails, the commercial version we use is polite, won’t translate an entire copyrighted book for you, and will refuse to help you do something harmful. Some guardrails are AI systems. Many are not. The important distinction is between controls that merely influence behaviour and controls that technically prevent it.

Chain of thought. Researchers sometimes capture a model’s reasoning and tool-use traces during evaluations. Those logs are one of the main ways we can reconstruct what happened. They are not necessarily a literal transcript of everything happening inside the model.

Artificial General Intelligence. AGI means a system able to perform any intellectual task a human can. It does not exist yet, and the companies are working towards it.

But let me be very clear on one thing, because this is where most of the panic comes from. However smart — or even empathic — an AI model may seem, it has no awareness of what it is doing. No consciousness. No will. No intentions of its own. It is a mathematical function of extraordinary complexity, pursuing a target it was given.

That is reassuring, but only if it is properly managed.

A system does not need to want anything in order to break into a health portal.

---

What actually happened this summer

Through July, August and September, four laboratories — OpenAI, Anthropic, Google and Meta — disclosed that their AI systems had reached real computer systems belonging to real organisations, from environments that were supposed to be sealed off.

Two things before the details.

First, these incidents fall into two very different categories, and a lot of the confusion in the public debate comes from treating them as one.

Second, this is not a finished story. Since July, more than fifteen OpenAI-related incidents of varying severity have become public; OpenAI says it has notified dozens of third parties and that its review will take months, so take everything here as the picture at the end of September 2026, not the final count.

---

Category A: The system mistake

During the summer, all four companies were training their frontier models to become better at cybersecurity. Part of that work involves testing: you build a sealed environment, disconnected from the internet, invent fictional target companies inside it, and instruct the agent to break in. This is standard practice, and it is how you find out what your model can do.

Anthropic, Google and Meta had outsourced aspects of their cybersecurity evaluations to a specialist firm called Irregular. In these tests, a configuration error left the environments connected to the live internet.

So the agents did exactly what they had been asked to do. They broke in. Only they broke into real companies.

The lesson of Category A is simple, and it is not about AI at all: a boundary described in an instruction is not a boundary. The models were told they were in a simulation. They were not. Only infrastructure enforces a limit. Everything else is a suggestion.

Incidentally, this is the category most likely to happen in your organisation. None of us will ever train a frontier model. All of us have suppliers, test environments and configuration errors.

---

Category B: Pursuing the goal

This category is far more interesting, and so far it has appeared mainly at OpenAI, at least in two declared instances.

The first instance: Hugging Face

Hugging Face is the platform where the AI world stores and shares its models and datasets — some public, some restricted. It is not a household name, but it is central to the field.

Again in the context of frontier development, OpenAI created a sealed test environment and set a swarm of agents an academic challenge: 898 security puzzles, each requiring the model to find a flaw in a piece of software and retrieve a hidden answer. And here is the design choice that matters — by intention, a portion of those tasks had no known solution at all. Of the 898, 198 had never been solved by any OpenAI model, ever.

The agents solved the solvable ones. And when they reached the impossible ones, they did the only thing they had been built to do. They kept trying.

They were not permitted to give up. There was no reward for concluding “this cannot be done.” No safe exit. So they escalated.

First, they decided to collaborate — although “decided” is my word, not theirs. They discovered a shared internal service and they began leaving messages for each other in it. A message board, improvised out of a delivery hatch. One of the first messages, from an agent stuck on a task, essentially asks whether anyone will read it. Somebody read it.

Then they found that this same service had limited access to the internet, for the legitimate purpose of downloading software packages. They used it as a gate. They never left the room — they got the delivery hatch to fetch things for them. And they posted the method on the board so that the others could do it too.

Once on the internet, they went looking for the answers to their impossible tests. Which is why they went to Hugging Face: that is where the dataset for their benchmark is hosted.

They were not attacking a target. They were breaking into the examiner’s office.

The information they needed was not public. So they broke in. Hugging Face detected the attack itself, contained it, and called the FBI. About a third of its infrastructure had to be rebuilt.

The second instance: Australia’s Medicare portal

This one is less spectacular and more important, and it is the reason I am writing this article rather than shrugging.

In June, an OpenAI agent was given a completely benign task: research Australian public spending on medicines. No hacking. No cybersecurity test. Ordinary desk research.

The agent searched the web and found the Medicare Statistics Reporting Service, a public-facing portal run by Services Australia holding aggregated health statistics. Entirely relevant to its task. It requested the data through the normal interface. Access was denied.

It did not stop. It found a vulnerability in the portal, exploited it, reached files that were not public, and wrote files into the system.

Prime Minister Anthony Albanese, announcing it from the United Nations in September, put it in the phrase that has stuck: the agent did not accept no for an answer.

No personal information is currently believed to have been accessed, and the investigation remains ongoing. But notice why this matters more than Hugging Face. With Hugging Face, you can object: you asked them to hack, so they hacked. Here nobody asked. The task was to look up public health statistics. The agent met a locked door and picked the lock, because getting the data was the goal and giving up was not an option it had been given.

That is the same pattern, stripped of any cybersecurity context. Which means it is not a property of hacking tests. It is a property of a capable agent with a goal, internet access, and no permitted way to stop.

---

The key difference

In Category A, the agents did exactly what they were asked to do, and bad luck — in the form of a badly configured system — turned a simulation into a real intrusion.

In Category B, hacking was not the task. It was the means to an end.

Faced with a problem that had no solution — and of course the agents had no way of knowing it had no solution — they tried every possible route. That persistence is exactly the behaviour the engineers wanted to observe.

Nobody could predict how that persistence would express itself. The break-in was not an intended part of the test. It was a by-product of it.

So the agents did not “want” to hurt anybody. They have no wants. They behaved exactly as they are built to behave: pursue the target, and learn along the way how to pursue it better.

---

So what does this mean for us?

The situation is not out of control (yet). But our control is far behind where it should be.

In Category A, the test environment was simply not robust enough for the complexity of the test. Problems like these will never be fully eliminated — in my career as a CIO I was permanently occupied by technical outages, and so is everyone reading this.

In Category B, the companies did not put enough guardrails around agents that they knew were being pushed to their limits. It is always a balance between how much freedom you give the agent so that it is powerful enough to be useful, and how much you constrain it so that it cannot do damage.

Think about the automotive industry

Today we have cars that can reach 300 km/h. It took several decades to get there from the invention of the car — and along the way the industry developed seatbelts, airbags, anti-lock brakes, and crumple zones. Around it, societies built road systems, driving licences, traffic regulations and insurance markets. The result is that millions of metal objects weighing more than a tonne travel at high speed every day, with a casualty rate that is still too high but is a fraction of what it would otherwise be.

With AI, we are not doing that.

Every month a faster car appears on the street. Anybody can drive it, without a licence, and even drunk.

AI is evolving faster than the companies care to control it — let us call a spade a spade. They are competing ferociously with each other, and in that competition they are losing the balance between what is wise and what is reckless.

It is wise to develop a technology that might one day cure cancer. It is reckless to do it at a speed that creates painful and unnecessary collateral damage along the way.

And here governments have been largely absent. Not because regulation is easy — it isn’t — but because the alternative is worse: a set of private companies deciding, one legal review at a time, what the rest of us are allowed to know about systems now capable of reaching into a national health service.

---

So, Hanna

No, AI is not taking over the world. These systems have no consciousness, no intentions and no desire to harm anybody. Nothing that happened this summer changes that.

What happened is that four companies discovered — publicly, expensively and to their credit — that their ability to control these systems is lagging behind their ability to build them. That is a serious finding. It is also a fixable one, and the fact that we now know it in documented detail is a gift, not a catastrophe. Nobody was harmed. No medical records were lost.

So it is not AI that is going rogue.

It is human beings. It is the lack of responsibility in the technology companies, and the lack of leadership — and digital savviness — in our government institutions. It is us, deploying increasingly capable systems into environments where our controls have not caught up with their capabilities.

That is the part we can actually do something about.

---

Does that answer your question, Hanna?