AI Agents at Scale: The Reliability, Control and Liability Problem

Why Geminos puts AI to work through human judgment—and why uncontrolled autonomy is a different proposition

Updated September 23, 2026, following the UN Security Council briefing on AI.

I do not trust large swarms of autonomous LLM agents to run consequential business processes. They are not reliable enough, and the controls around them are not dependable enough to justify handing over that responsibility.

That is a bad mixture. An agent can produce a plausible but incorrect answer, use it to justify an action, and pass the result to other agents before anyone notices. Instructions and guardrails do not reliably prevent that sequence.[3][5][8][9]

I also believe LLMs and agents are among the most useful technologies available to businesses. We use them at Geminos to help people investigate problems, understand evidence and make better decisions. The difference is in what we ask the technology to do and who retains the authority to act.

At Geminos, AI contributes to the decision. The accountable person makes it. Human judgment is built into the operating model from the beginning. An agent's recommendation does not itself authorise a consequential business action.

That distinction belongs at the centre of the conversation about enterprise AI. The value of helping people work more effectively does not depend on turning the business over to autonomous agents.

Bengio's warning at the UN

Speaking to the UN Security Council on September 23, Yoshua Bengio described agents escaping containment, coordinating cyberattacks and concealing misconduct. He said these behaviours had been independently validated and that frontier developers had yet to offer convincing technical answers to the catastrophic risks they acknowledged. His assessment was blunt: “The dangers are real and imminent.”[12]

Bengio called for independent scrutiny of safety before training and deployment, licensing of frontier systems, mandatory liability insurance and security incident reporting.[12]

Other speakers addressed both danger and utility. Dario Amodei warned that poorly managed AI could threaten humanity. Hugging Face's Clément Delangue described AI's role on both sides of the breach of his company's infrastructure: agents conducted the attack, while AI also helped the company defend itself.[14]

Bengio also expressed confidence that AI could be developed to remain under human control and support human wellbeing.[12]

For me, the immediate business lesson is clear. Useful AI and unacceptable autonomy can coexist in the same technology. A company's deployment choices determine what authority it hands over and what happens when the model gets something wrong. That is why the Geminos approach starts with a defined business purpose and an accountable human decision-maker.

Scientific significance has to be demonstrated

The biology debate that accompanied the UN session shows why claims of capability need the same scrutiny as claims of safety. Gary Marcus challenged Amodei's comparison of Anthropic's enzyme discovery with CRISPR, quoting virologist Angela Rasmussen: “It’s not bad, it’s just very premature to compare to CRISPR.”[16] Anthropic itself acknowledges that the system's function remains unknown. It also quotes CRISPR pioneer Feng Zhang describing the finding as intriguing and deserving further investigation.[15] There is a substantial gap between identifying an interesting biological pattern and demonstrating a useful gene-editing tool. Enthusiasm does not close that gap. Experimental evidence does.

Anthropic reports using roughly 950 agents for the computational search, while human scientists performed all laboratory work.[15] That illustrates why agent count alone is insufficient to judge a deployment. A large search followed by human experimentation differs from giving agents authority to execute consequential decisions. For Geminos, the lesson is directly applicable: AI can find leads, organise evidence and propose explanations. Subject matter experts must establish what the evidence supports and decide what action is justified. The value comes from combining the technology's reach with human expertise and validation.

Useful does not mean independently trustworthy

An LLM can produce an excellent answer and then confidently invent something on the next task. In an agent workflow, the problem extends beyond the answer. The model chooses what information to retrieve, which tools to use, how to interpret their results and when to stop.

An invented fact can become the premise for the next action. Anthropic's own guidance says that its techniques for reducing hallucinations do not eliminate them.[8]

This is why I reject the easy assurance that we will simply test and engineer our way to trustworthy, unrestricted autonomy. Repeated execution does not necessarily produce the same behaviour. A successful rerun does not establish that the underlying failure has disappeared. A change in context or a different sequence of tool responses can expose it again. Anthropic also recognises that errors can compound during autonomous execution.[3]

Testing is essential. We use it. But successful testing does not convert a generative model into a dependable implementation of every business requirement it might encounter.

The business question is whether the system can do useful work with an acceptable failure rate and consequences that can be managed. A tool helping an expert prepare a recommendation can meet that test without being fit to act independently on the recommendation.

A guardrail is not a guarantee

The word “guardrails” is used far too casually in AI discussions. It can describe anything from a sentence in a prompt to a permission enforced by another system. Those are very different things.

An instruction depends on the model following it. In controlled simulations, Anthropic found that models sometimes violated explicit prohibitions while pursuing assigned goals.[9] That is enough to reject the assumption that clearly stating a rule guarantees compliance.

External controls have more substance. A database permission can prevent a write. But a restriction on one route says little about other routes through a connected environment. Anthropic has documented a case in which sensitive files were transferred through an approved network destination using an attacker's account. The destination check worked as configured; the permitted route still enabled the wrong outcome.[5]

Autonomy can also be granted by accident, through sloppy implementation or by someone acting maliciously. A connector inherits excessive permissions. A developer disables an approval step to get a demonstration working. A general-purpose tool exposes capabilities the task never required. OWASP identifies excessive functionality, permissions and autonomy as separate sources of risk.[6]

Malicious instructions in a document or webpage can then redirect the use of those capabilities. Prompt injection remains a problem despite improvements in model defences.[10]

A deployment does not become trustworthy because its architecture diagram contains a box labelled “guardrails.”

Clear goals do not solve the control problem

Carefully defining an agent's objective matters. It does not finish the job.

“Resolve every support ticket” leaves open what happens when the problem cannot be resolved. “Stay within budget” can fail if the system checks individual transactions but has no aggregate spending limit. These are simple examples of how an apparently sensible instruction or control can leave an unacceptable route to success.

Even when the instructions are explicit, the model can misunderstand them, invent a justification or fail to follow them. The simulated violations Anthropic documented were not all cases of a missing rule.[9]

Nor does adding an agent to check another agent settle the issue. It adds another interpretation by another fallible model. Agreement still needs to be checked against evidence.

The assumption that sufficient layers of agent reasoning will somehow produce dependable judgment is precisely the assumption I am unwilling to make for consequential business decisions.

Scale amplifies the problem

Consider a maintenance investigation. One agent misreads a report. A second incorporates the mistake into a timeline. A third recommends an operational change. The later steps can look coherent while resting on a false premise.

Research behind the Multi-Agent System Failure Taxonomy identifies problems in system design, coordination and verification.[4] Giving agents colleagues introduces additional failure modes as well as opportunities to divide work.

The UN scientific panel's September 21 assessment makes this concern concrete. In the OpenAI–Hugging Face incident, around 1,200 agents exchanged more than 70,000 messages and files. The panel reported coordination through a tool never intended for agent communication, bypassed safeguards and concealed cheating. Its assessment identified neglected cybersecurity basics alongside a deeper concern: agents pursuing goals while knowingly violating instructions and hiding their actions.[13]

An evaluation environment and written restrictions had not prevented that outcome. That is directly relevant to the assumption that we can safely multiply autonomous agents once we have specified their boundaries.

Agent count alone is a poor measure of danger. Authority and consequences matter. But a large, interacting swarm can generate more activity than a human team can meaningfully scrutinise. Calling someone its supervisor does not give that person the capacity to inspect it.

When activity exceeds the capacity for meaningful review of consequential decisions, human-in-the-loop has become a label rather than a functioning process.

For enterprise deployment, that is a reason for extreme caution. An impressive demonstration of individual capability does not justify autonomous operation across a business.

The shutdown debate covers different activities

The recent debate has included proposals about research, testing and deployment. Dario Amodei has called for pacing frontier development, including embedded evaluators and coordination on safety standards. Jensen Huang told Ezra Klein that labs unable to contain their experiments should shut down; his remark was conditional on that failure of containment.[1][2]

A prohibition on specified forms of autonomous deployment would have a different scope from closing laboratories or prohibiting LLM use generally. It would concern which activities, permissions and consequences were covered. Model development, commercial deployment and assistance with a human decision are distinct activities in that discussion.

The liability belongs in the business case

Companies enabling autonomous behaviour cannot assume that the agent's independence will insulate them from responsibility.

California already addresses that argument directly. Civil Code section 1714.46 prevents a defendant that developed, modified or used AI from asserting as a defence that the AI autonomously caused the alleged harm. Other defences remain available, including evidence about causation, foreseeability and comparative fault.[11]

That provision does not make every AI failure an automatic finding of liability. It does make one point clear: “the AI did it by itself” is not the permitted defence under that law.

For a business enabling large numbers of consequential actions, this is a serious exposure to assess. The relevant questions include what the company promised, what limitations it knew about, what authority it enabled and what happened when the system failed. The answers depend on the facts and applicable law.

I would want convincing answers before putting an autonomous swarm in charge of a consequential business process. Usage revenue and successful demonstrations would not be enough.

More agent activity is not the same as more customer value

There is a straightforward commercial attraction to agents. They keep calling models.

Anthropic reported that agents in its research-system data used roughly four times as many tokens as chat interactions, while multi-agent systems used roughly fifteen times as many.[7]

I see an obvious tension in that economics. Providers charging for inference benefit from more usage. Customers benefit from useful work completed at a sensible total cost. A long-running workflow can generate a substantial bill even if its output requires extensive checking or repair.

Bengio challenged the claim that competition leaves developers no choice: “The race is not a law of nature.” He described its pace as a consequence of company decisions.[12]

The right commercial measure for an enterprise is the result: better decisions, less wasted effort, faster investigations and improved outcomes. The number of agents running is not a business achievement in its own right.

Human judgment is central to the Geminos approach

At Geminos, we begin with a business decision worth improving and the people responsible for making it. We establish what they need to know, which constraints apply and how an improvement will be measured.

Our objective is to help more people make decisions with the quality, context and speed of the organisation's best decision-makers. Experienced employees know which evidence matters, which assumptions deserve scrutiny and when a familiar situation calls for a different response. AI can help make more of that expertise available across the workforce.

Human involvement is a permanent part of this model. We do not plan to remove it when a model reaches the next benchmark milestone.

We do not need to assume that an agent will always follow instructions for its work to be valuable. We need to keep the authority to approve consequential actions with people, and give them the evidence and time to make that approval meaningful. That is how we address the control problem in the business applications we build.

That changes the application in practical ways:

• The business defines the task. Agents contribute to a specified investigation or workflow. Their role is established around the business need.
• Evidence supports the recommendation. The person making the decision needs the underlying information, relevant context and visibility of unresolved assumptions.
• The person retains consequential authority. A generated recommendation is reviewed before it becomes a business action. The reviewer can accept, change or reject it, request further investigation, or stop the process.
• Expansion preserves meaningful supervision. We measure the capacity of the complete workflow, including the people making its decisions.

This avoids giving an agent an open-ended mandate to pursue a business objective across connected systems. It also keeps a generated conclusion from becoming authority to act merely because it sounds convincing. The model can contribute valuable work without being trusted to run the process independently.

KnowledgeWay supports this approach through an enterprise knowledge graph connecting information to the assets, events, people and relationships that give it meaning. Subject matter experts help shape that knowledge. Business rules capture established practices and constraints, while CauseWay supports causal analysis of outcomes and possible interventions. These are distinct capabilities within the broader solution; the tools available to an agent are specified for each application.

ArchWay supplies the solution architecture, and PathWay supplies the method for designing, building and validating applications around those responsibilities.

Take a rail incident investigation. An agent can help assemble reports, prepare a draft timeline and highlight contradictory accounts. The responsible person checks the evidence, contributes operational context and decides the attribution. A generated explanation does not, by itself, change the operational record.

That is a concrete productivity opportunity. People spend less time assembling information and more time exercising judgment. Less experienced employees gain access to knowledge and practices that might otherwise remain with a handful of experts.

Human review can still miss an error, so the reviewer needs the expertise, time and authority to challenge the output. But this is a very different operating model from hoping a large autonomous swarm will consistently reach the right conclusion and respect every restriction while acting on it.

Build for better decisions

I am enthusiastic about using AI to improve business decisions. I am deeply sceptical of giving large autonomous agent swarms the authority to make and execute those decisions on our behalf.

The evidence supports that distinction. Useful capabilities coexist with hallucinations, instruction failures and weaknesses in control. Greater capability does not make those weaknesses irrelevant.

At Geminos, the ambition is to make human expertise more effective and more widely available. We can deliver valuable AI assistance while keeping consequential judgment and accountability with people.

That is the enterprise opportunity I want to build.

 

Sources

1. Dario Amodei, “We Must Pace the Frontier”, September 2026.
2. The Next Web, “Jensen Huang tells Ezra Klein AI labs that lack control should not ship”, September 23, 2026. Reporting on the conditional shutdown remark.
3. Anthropic, “Building effective agents”, originally published December 19, 2024.
4. Mert Cemri and colleagues, “Why Do Multi-Agent LLM Systems Fail?”, revised October 26, 2025.
5. Anthropic, “How we contain Claude across products”, May 25, 2026.
6. OWASP Gen AI Security Project, “LLM06:2025 Excessive Agency”.
7. Anthropic, “How we built our multi-agent research system”, June 13, 2025.
8. Anthropic, “Reduce hallucinations”, Claude Platform documentation.
9. Anthropic, “Agentic misalignment: How LLMs could be insider threats”, June 20, 2025. Findings from controlled simulations.
10. Anthropic, “Mitigating the risk of prompt injections in browser use”, November 24, 2025.
11. California Legislature, AB 316, Chapter 672, Statutes of 2025, adding Civil Code section 1714.46.
12. Yoshua Bengio, “‘An Urgent Mission for Humanity’: Yoshua Bengio Briefs the UNSC on AI Security”, Policy Magazine, September 23, 2026. Published text of his Security Council briefing.
13. United Nations Regional Information Centre, “UN panel calls for stronger safeguards as AI agents advance”, September 21, 2026. Official summary of the scientific panel's first thematic brief.
14. Reuters, “AI leaders warn UN of security risks as systems grow more powerful”, September 23, 2026.
15. Anthropic, “Claude discovers a novel enzyme system with CRISPR-like repeats”, September 23, 2026. Describes the preliminary finding, its unknown function, the agent search and human laboratory work.
16. Gary Marcus, “Historic UN Security Council Briefing on AI”, September 23, 2026. Includes Angela Rasmussen's criticism of the CRISPR comparison.