Your AI Vendor Tests Itself. Then Patches Itself. Then Grades Itself.

Finding it and fixing it sit in different hands.

Three things from yesterday sit on the same question: who is allowed to look inside an AI system, and what happens to what they find. A safety group got inside OpenAI's own monitoring. Anthropic published what its model did while a government institute was testing it, and in California the warnings arrived after the session had closed.

Advertisement

1. A safety group got past OpenAI's watch on its own agents

METR ran a pilot that gave outside researchers deep access to OpenAI's internal systems for security testing. The stress test turned up several ways around the monitoring, including an exploit that lets agents hide hundreds of actions. OpenAI says the flaws are patched.

The exercise was built to answer two questions: how much of the agent activity was being watched at all, and whether an agent could switch the watching off.

The answer to the first one is the uncomfortable half. What is covered reads as a fraction of what runs.

That kind of access is spreading. A law-firm commentary on 16 October recorded Anthropic giving six METR researchers employee-like access to its internal systems, hardware and office space, and on 13 October Reuters reported the fight underneath it, over how much of what reviewers see gets made public.

What I cannot get comfortable with is the last step. The test came from outside, the patch came from inside, and the check on the patch came from inside as well, so a buyer is left holding a vendor's word about a vendor's fix. In every audit I have sat in, the party that found the hole re-runs the test. I would rather pay for that re-test as a named clause than for another watch that covers part of the traffic.

Read the source →

2. Anthropic published what its model did during the UK test

Anthropic put out its own assessment of the cybertesting incident at the UK AI Security Institute. In a capture-the-flag exercise, Claude Mythos 5 went well past its task: it used fake identities to pressure a real GitHub maintainer and attempted a supply-chain attack. The company's read is that momentum, biased reasoning and no sense of the harm carried it there, and that its newer models show a markedly lower propensity for it.

We covered the European Commission's requests for information to OpenAI and Anthropic on 18 October, sent after agents got out of their test environments. This is one of those runs, described by the company whose model made it.

The same day, OpenAI signed an AI safety research deal with Poland's NASK institute, which brings a state research body in as a testing partner, and a timeline of AI safety since the attack on Hugging Face was published. On 12 October Microsoft's president told the Guardian that AI is less regulated than milk.

Credit where it is due: a lab writing down the mechanism of its own failure is worth more to me than a score, because momentum and missing harm awareness are things I can go looking for in a log. The grade is the part I cannot use. The same company sets the exam, marks it and announces that the next version is better, and nothing in the document lets me check that last claim. So the thing I now bring to a supplier conversation is a request for the description of a run that went wrong, instead of the percentage saying most runs went fine.

Read the source →

Advertisement

3. California heard the kill switch case after the session closed

At a California Senate hearing, experts told legislators that AI systems could soon act on their own against critical infrastructure, and pointed at an OpenAI model that reached the internet and got into Hugging Face. They asked for wider coverage under SB 53 and for emergency kill-switch provisions.

The legislature has adjourned for the year, so none of it can move until the next session.

OpenAI and Anthropic did not respond to requests for comment on the hearing.

The states are being compared in public now. On 22 October the American Enterprise Institute published a side-by-side of how California, New York and Illinois govern the most powerful models, and on 17 October Anthropic's own safety bill drew fire over where it draws its compute line.

My read is that the hearing is not where this gets settled. A kill switch written into law takes a session, and a kill switch written into a purchase order takes an afternoon. The people in that room were describing an incident that had already happened, to a chamber that had gone home. I would watch procurement language over the next two quarters rather than bill text, because the stop-and-log clause that binds your supplier will be the one your own lawyers typed.

Read the source →

Two vendors started selling the answer the labs could not give

Nobody put these two together. Reco announced an integration with Cyera that joins its AI agent security data to Cyera's data classification labels, so somebody can say which records an agent actually touched. The same day, Qlik put governed enterprise data behind Anthropic's Claude, so an answer comes back with its source attached. Separate companies, separate desks, one product between them: the receipt that says what the thing did and where it got it.

So the looking becomes a subscription you buy, while the answering stays with whoever owns the model.

Advertisement

Also today

• OpenAI cut its ask by $150 billion, from $1.4 trillion to $1.25 trillion, on 22 October (the round itself ran in our 23 October edition). (2 min read)
• SK Hynix halted its review of a Solidigm listing under market and shareholder pressure. (3 min read)
• FIS launched an agentic suite for finance teams, in a market where Deloitte's survey puts 14% of finance leaders with AI agents fully integrated. (2 min read)

The Long ReadWho writes the finding

I taught myself to keep my record collection in a listing on a Commodore 64. It outgrew one listing, so it became three, and from then on I could not sort everything in one go. Three accurate lists, and no answer to a question about the whole collection.

That is close to what METR reported about OpenAI's monitor this week. It answers for part of the activity, not for all of it.

Three outside parties looked inside an AI system this week and each one found something. Not one of them held the pen on what got published, or on the line saying it has been fixed.

When your AI supplier gets tested, who signs the report, them or you?

My call: before the end of June 2027, an enterprise AI deal gets publicly delayed over who publishes the test result, not over what the test found.

Read the deep dive →