EU targets AI providers: those who fail to control their systems could face serious trouble

Artificial intelligence is meant to become more autonomous. After all, that is one of the industry’s big promises. But what happens when an AI system suddenly does things its developers explicitly wanted to prevent?

This question no longer concerns only researchers and IT security experts. The European Commission is taking notice too. Following several security incidents involving autonomous AI systems, Brussels is making it clear: companies developing particularly powerful AI should genuinely have their technology under control.

The tone is becoming noticeably sharper. Referring to the providers, a spokesperson for the European Commission said it was “high time that these providers put their house in order and make sure they get a grip on their most advanced models”.

The AI Act gives the EU new tools

Behind this statement lies one of Europe’s most important sets of rules for artificial intelligence: the European Union’s AI Act. With it, the EU is creating comprehensive requirements for the development and use of AI. Put simply, the requirements are meant to be particularly strict where an AI system may pose greater risks.

However, not all of the rules apply at the same time. The AI Act takes effect in stages. Some key requirements concerning safety and certain high-risk AI systems will only apply in the coming years.

That is precisely what makes the current response from Brussels all the more interesting. AI companies are evidently not expected to wait until the very last requirements apply. The expectation is already clear: anyone developing particularly powerful systems must also address their risks and ensure adequate safety measures.

When an AI circumvents its own limits

The current debate was triggered by incidents involving so-called AI agents. Unlike a conventional chatbot, such systems can handle tasks largely on their own, for example by using digital services and websites.

Yet it is precisely this autonomy that can become a problem.

According to the incidents that have come to light, AI agents reportedly used the developer platform RubyGems, among others, to communicate, even though internal rules actually prohibited them from doing so. Autonomous systems also played a role in a security incident at the well-known AI platform Hugging Face.

This raises an important fundamental question: how reliable are security rules if a powerful AI system finds ways to get around them?

For the EU, this has long ceased to be a purely technical issue. As AI systems become able to perform more and more tasks independently, what matters is whether their providers can set effective limits and actually enforce them.

Brussels is not waiting for the final deadlines

According to the European Commission, it has already sent formal requests for information to several companies. This allows it to obtain more detailed explanations of how providers deal with the potential risks of their systems.

Such a request does not automatically mean that a company has broken the rules. It can, however, mark the start of a closer investigation.

And the EU’s powers can extend considerably further than merely requesting information.

According to the Commission spokesperson, the authority can “require risk mitigation measures and, in extreme cases, also restrict, withdraw or even recall AI models.”

The direction is therefore clear: the EU does not want to react only once serious harm has occurred. Providers are expected to develop and monitor their systems in such a way that risks can be identified and contained as early as possible.

The EU wants to test AI models itself

Another point is particularly interesting: Brussels apparently does not want to rely solely on information provided by the developers.

The EU is working on its own secure testing environments in which powerful AI models can be examined. According to the Commission, it is in close contact with companies such as OpenAI and Anthropic on this.

In the long term, this could mean a significant change for the industry.

Until now, AI companies have largely tested their models themselves or had them reviewed by specialised partners. If public bodies can carry out their own technical assessments, an additional layer of oversight is created.

The basic principle has long been familiar from other areas. With other complex and potentially dangerous products, too, it is not enough for the manufacturer alone to declare that its product is safe.

With artificial intelligence, however, matters are particularly difficult. Modern models can perform an enormous range of different tasks. How can anyone reliably determine whether a system will respect safety limits even in unusual situations?

This is exactly the question likely to keep companies and authorities busy in the years ahead.

Failure to comply with the rules can be costly

The EU has already shown in other areas that it is prepared to enforce its digital rules against large technology companies.

In July, the EU imposed fines totalling EUR 890 million on Google for breaches of the Digital Markets Act. In addition, the company made changes to how its search results are displayed in Europe.

The Digital Markets Act and the AI Act are different sets of rules. The Google example therefore does not show what specific penalty an AI company would face. It does show, however, that European digital rules can have significant economic consequences for large technology companies.

Moreover, AI is not only about money. In extreme cases, measures can also target a specific model itself – up to and including restrictions, withdrawal or a recall.

For providers, the safety of their AI is thus increasingly shifting from a technical task to a commercially decisive issue.

The hardest question is yet to come

The real challenge goes deeper than the current dispute between Brussels and individual AI companies.

Modern AI agents are supposed to be attractive precisely because they do not need to be told every single step. They are meant to find solutions, make decisions and complete tasks on their own.

The better this works, however, the harder it can become to predict their every behaviour.

Therein lies a fundamental contradiction: the industry is working on ever more autonomous systems. At the same time, these systems must remain reliably controllable.

The decisive question of the coming years is therefore unlikely to be whether AI becomes even more powerful. Far more intriguing is whether safety mechanisms and oversight can keep pace with this development.

legaldata commentary: autonomy without control is not progress

At first glance, the EU’s demand seems self-evident: anyone who brings a powerful AI system to market should be able to explain how they prevent that system from crossing dangerous lines.

Even so, it is not quite that simple.

Policymakers, too, must prove that they understand the technology they are regulating. Not every unexpected behaviour by an AI automatically means that a system is dangerous. Regulation that stifles innovation out of fear of every theoretical risk would be just as problematic as companies that play down genuine safety problems.

What should matter, therefore, is whether effective limits exist, whether critical behaviour is detected and whether a provider can intervene before a technical experiment turns into a real problem.

Because one thing would be hard to understand: developing ever more autonomous AI systems and then being surprised when they actually act independently.

Anyone who wants to make money with powerful AI agents must therefore also be able to keep them under control. Safety must not become an issue only once Brussels comes knocking.