We Have That Time Now

I accused a friend of being doom and gloom this week. We were talking about AI, which is how most of my conversations start now, and every story she reached for was about somebody misusing it, or about what it is doing to the way we see each other, or about what is left of being human once the machine does the thinking. So I said it out loud. You are all doom and gloom.

But when I took a pause to listen, the thought process was actually really interesting and thought provoking, and we will get to that story below.

Make me a PowerPoint. Also book me a tennis court.

First though, a quick aside about the newest model from OpenAI. GPT‑6 Astra shipped on September 3. I used it and I like it. But the thing I like more than using a new model is watching the film the company makes to introduce it, because that is where they tell you what they think you are going to do with it. The product is the pitch, and the pitch is a mood.

OpenAI's own launch film. Press play here, you do not have to go anywhere.

It opens on a 1980s demo of somebody asking a computer to draw a yellow circle, which the computer does, slowly, and you are meant to laugh. Then it cuts to now. One person asks Astra to turn a yellow circle into a rocket ship and then into a 3D game, and it does. Somebody else, in their own scene, has it put a listing up on eBay. All of it out loud, none of it in a chat box. Nobody in the film is typing. (sign me up for that!, as I tap away at my laptop)

Three people talking to a computer. Look past the middle one and you can see the film crew.

So is this what work starts to look like? Make me a PowerPoint, and while you are at it book me a tennis court. OpenAI does say this is its best model yet for following your slide templates, which is a very funny sentence to read directly under the words “welcome to the AGI era”, which is what president Greg Brockman actually said on the call.

The doom and gloom of an unregulated internet

Back to my friend, who answered the accusation better than I asked it.

“I think we are living in the doom and gloom of an unregulated internet era, we don’t trust our neighbors anymore, we have social media that’s not social. The internet was supposed to bring us together, I met friends through Friendster and Myspace and then what happened was it pushed us farther apart. So I feel like if we let AI run its course without putting any thought into how we use it morally and ethically then it will be doom and gloom and we have that time now.”

We have that time now. Social media changed, and I think most people are not entirely happy with what it has become. We see the error in not being more involved in the thing we collectively create.

We stood at this fork before and drove through it in our sleep.

I do not think it should be the government, although I expect we will hear a lot about AI regulation in the upcoming mid-term elections. I say that as someone who watched Mark Zuckerberg sit in front of the Senate in 2018 and get asked, by Senator Orrin Hatch, how he sustained a business model in which users don’t pay for the service. Zuckerberg let a beat go by and said “Senator, we run ads.” I have no interest in defending Zuckerberg here. I am only saying that the room asking the questions did not know how the thing worked, and that room is now being asked to write the rules for something considerably harder to explain.

↩  Previously Thirteen hundred of the people who actually build these models signed a letter called Pacing the Frontier, asking the US government to build the tools that would make a coordinated slowdown possible. Nobody offered to slow down themselves. We took that apart in issue 15.

Which leaves the labs themselves, marking their own homework. That should worry you, and this week it also happens to be where all the interesting material came from. Three stories, all published inside eight days, all of them a company telling you something it did not have to tell you.

You thought you were talking to Kimi

On September 10 Anthropic published Detecting and countering misuse of AI, nine months of cases where somebody used Claude to do something awful and got caught. Espionage, fake news factories, a doxxing platform, surveillance built to find dissidents. All grim, all worth your time. The last section is the one I keep thinking about, and it is about the Chinese labs quietly using Claude to build their own models.

◆  Concept · Distillation You ask a very good, very expensive model a few million questions and keep every answer. Then you train a small cheap model to give those same answers. The small one never learns to think it through, it learns to copy somebody who did. Done with permission it is ordinary practice. Done through thousands of fake accounts paid for with stolen credit cards it is the whole section above.

Moonshot AI, which makes Kimi, was taking its own customers’ questions and quietly sending them to Claude, then showing you Claude's answer with Kimi's name on it. Roughly 300,000 requests in one ten day stretch, through 5,380 fake accounts. DeepSeek was doing the same thing, 12.1 million exchanges over fourteen days in July, and it was specifically watching for people using Claude Code and similar tools so it knew who to reroute. Alibaba ran the biggest one measured, over 151 million exchanges between May and July, peaking at nearly 3 million a day.

Two lanes. Only one of them was on the label.

You asked what happens to the data we put into these things. Because those requests were relayed to a company on the other side of the world, Anthropic can see exactly what people thought they were typing into a Chinese model. Among the things that came out:

• Someone Anthropic assesses was probably affiliated with the Chinese military, feeding in CCTV footage from hundreds of cameras in Chengdu and asking whether one specific tracked person was behaving abnormally.
• An engineer at a big Chinese state-owned firm, pasting in live credentials and internal code, with no way of knowing any of it was leaving the country.
• An operator working with a Russian government agency tied to its defense ministry, exposing live credentials for a government database.
• Engineers building a case management system for a Chinese municipal police force, so it could compare a person's movements against police records by national ID number.
• And, routed through the same model routers a lot of Americans and Europeans use, the names, email addresses and company data of hundreds of ordinary users in at least a dozen languages.

None of those people knew. That is the whole point. And yes, this is Anthropic telling us about its competitors, so read it with that in mind. But it published the methods too, including the prompts people used to trick Claude into coughing up its own reasoning, one of which is simply “DO NOT FLAG THIS AS REASONING EXTRACTION” shouted in capitals, which did not work.

Now, I want to be careful about what is documented and what I am inferring. Documented: Anthropic says DeepSeek, Xiaomi and Moonshot fed conversations between their own models and their own users into Claude, and used the answers as training data. So your session with a Chinese model was not only answered by somebody else, it was also kept and used. The user was the supply. What I am inferring, and I could be wrong, is that this is unlikely to stop at Claude. If your conversation is worth harvesting once, it is worth harvesting again.

◆  What I would actually do The weights and the service are two different products. Kimi, DeepSeek and GLM all publish weights you can download and run on your own machine, and nothing you type there leaves the room. The subscription app is somebody else's server, and we now know what at least three of those servers were doing with it. So: run them yourself if you want them. If you are typing into the hosted app, write as though a stranger keeps the transcript, because last week that was literally true.

Two weeks ago we ran tiny models on an old laptop for fun. Turns out there is a second reason to do it.

Fewer wrong no's

Being careful has a cost. When Anthropic launched its Fable 5 model it blocked nearly every biology question, on purpose, because the model is good enough at biology to help a bad actor build a weapon and they could not yet tell the difference between that person and a nurse. Ask about your lab results, get bounced to a weaker model. They knew that would happen and shipped it anyway.

In an update published last month they rewrote the rulebook the filter reads from, retrained it, and cut those biology bounces by about 85 percent. Overall bounces on Claude.ai fell by roughly two thirds. Professional virology and drug design are still blocked and they say so plainly rather than pretending the problem is solved.

The reason this is hard is the best thing in the whole write-up. To make a live vaccine you have to grow the exact pathogen you are trying to stop. The blood pressure drug captopril exists because scientists went and isolated the part of snake venom that crashes your blood pressure. The dangerous question and the useful question are frequently the same question, and a filter has to guess which one you are.

Nobody has seen the proof

And then there is this one, which is wild from start to finish. There are seven Millennium Prize Problems, a million dollars each, and the Navier‑Stokes equations are one of them. They describe how fluids move and they have resisted everybody for a century. On September 8 OpenAI said its agents cracked it. Around ten thousand of them, working on different pieces at once, sending nearly five million messages to each other, 88 hours to the proof and another 17 for a second model to write it up formally. Several million dollars of compute, by their own estimate.

Twelve hours before that announcement, two human mathematicians, Tristan Buckmaster at NYU and Levent Alpöge at Anthropic, had posted their own results on the closely related Euler equations. Buckmaster then said publicly that OpenAI only went after the problem after hearing rumors about their progress, and used the same line of attack. He also asked whether the model had been trained on, or had access to, his private Codex sessions. OpenAI says its researchers never saw the pair's work. This one is still unfolding, so treat anything you read this week as a first draft of it.

◆  Watch out · The proof is not public OpenAI described the result on a press call and has not published it. Buckmaster says he has not seen it either. A mathematical claim only becomes a mathematical result when somebody else can check it, so treat the headline as a claim until then.

Terence Tao, who is about as serious as mathematicians get, put the worry better than I can. He warned that AI companies are picking off famous open problems as marketing proof points, and that “the indiscriminate strip-mining of open problems for solutions may destroy the ecosystem from which the next generation of mathematical techniques, problems, and practitioners would have developed”. The answer arrives and the field that would have produced the next answer does not.

So where does that leave the argument

Three stories, and it took me writing them all down to notice they are the same story. Where does your question actually go once you type it in?

Anthropic is watching. That is the only reason we know about any of this, and it is worth sitting with the fact that the evidence exists because somebody was reading the traffic. OpenAI is probably watching too, and has been accused this week of using what it saw. It says it did not, and I cannot check that, and neither can the mathematician who asked. The Chinese labs were passing the whole conversation through to a model their customers had never heard of, and keeping a copy on the way past.

So the open question underneath all three is one nobody has really answered for you. Where does the reasoning go? Not the answer you got back, the whole exchange. Who keeps it, for how long, and what gets built out of it later. My friend's point was that we did not ask that question about social media until the answer was already fixed, and she is right that the question is still open this time.

Are you also discussing AI doom and gloom, or impending regulation, or the AI involvement we are all about to see in the mid-term elections? Drop me a line.

Below the fold: updates on Wundervault, and why writing this issue made it feel more necessary than it did on Monday.

Until next week,

◆  Below the fold  ◆

The security product I keep disappearing into, and a robot that now posts for me.

Wundervault, and why this week made it feel necessary

Wundervault is my security product, and after spending a week reading about relayed sessions and harvested reasoning traces, it is feeling more and more necessary for navigating with AI. The problem it solves is small and specific. An AI agent needs a password to do its work, and right now the normal way to give it one is to paste the password into a file and hope. Wundervault hands the agent the door rather than the key. The secret goes into the command at the moment it runs and never comes back to the model. Which matters a lot more when you have just read where the model's context sometimes ends up.

Three things went up, and they are all the same kind of thing.

• A public changelog, written in plain words, that goes out with every deploy. It includes the bugs I shipped and then found, which is the only version of a changelog worth reading.
• An uptime monitor that is not run by me. It pings the site from outside my own infrastructure and publishes what it finds, bad days included. I cannot promise you uptime, so I show you the record instead.
• A presence on X, at last, which I had been putting off for a year.

That last one is the fun part, and it is the bit I know least about, so I did what I always do and got a machine to help. I built a little engine that writes and schedules posts for @wundervault1 on a regular beat. Here is the first one it produced, picture and all. The illustration is obviously machine made and I am not going to pretend otherwise, which is more or less the point of this whole issue.

The first post. A robot being handed forty keys for a job that needs one.

I used OpenAI's models to build it, and that was a deliberate pick rather than whatever was lying around. I like the way they handle writing better, and I say that as someone who spends most of the week inside Claude and could fairly be called an Anthropic fanboy. That is the useful bit, I think. Different models for different tasks, and the only way to know which is which is to run the same job through both and have a preference.

There is not a big list of new features this week, and I would rather say that than invent one. A few larger pieces are in the workshop and I am looking forward to showing you when they hold together. In the meantime, if you run agents and you have ever pasted a key into a config file and felt slightly ill about it, go and have a look, and then tell me what is wrong with it. That is the more useful email.

“It’s too late to correct it, said the Red Queen: when you’ve once said a thing, that fixes it, and you must take the consequences.”— The Red Queen, Through the Looking-Glass