AI Momentum 98.5 +1.5%GitHub Pulse 5503 +6.5%Capital Flow $204.0B +19039.3%Hiring Heatmap 0.0 -100.0%
Advertisement
Open Source

Researchers caught hosts of freely available AI models leaking customer prompts. OpenRouter has dropped three of them

The main selling point of freely available, open-weight AI models is that anyone can run them. Open-weight means the maker publishes the model itself, so dozens of hosting companies compete to serve it cheaply. New research shows the hidden cost of that bargain: the company actually running the model may not protect what customers send it.

On November 22, OpenRouter, a popular marketplace that passes companies' AI requests to whichever host can serve them, said it had removed three hosting providers. It also pulled 125 model listings from its "zero data retention" tier, a promise that prompts and answers are not stored.

OpenRouter said its own systems were not breached and that it had notified every potentially affected customer. It did not name the three providers it dropped, or say how many customers or conversations were exposed.

The trigger was research published November 19 by Dragos, a security firm, written by its chief research officer HD Moore. From July through November, Dragos sent about 2 million prompts to popular AI services. Each one carried unique tripwires: fake passwords, secret web links and decoy cloud-account keys. Any use of them proved someone had read the prompt.

What the tripwires caught

Dragos says three hosting providers were breached without knowing it, the earliest in mid-July. In the largest case, an attacker collected a couple hundred of the seeded Amazon Web Services keys and used them for tens of thousands of calls to Amazon's services.

Some of those keys came from requests the host had rejected before any model ran. Dragos reads that as copying at the very first point of contact, not a leak from the model itself.

The problems went beyond outright breaches. Dragos says some sellers on an AI marketplace logged customer traffic, and people on home internet connections later tried to log in to Dragos's decoy servers. One router, a service that relays requests between customer and host, was quietly forwarding traffic to such a marketplace. It said it stopped once told.

In two of the breaches, some of the stolen requests had been flagged "do not store." "Zero Data Retention is supposed to mean something," Moore wrote. His larger point is that a privacy setting binds only the first company in the chain, while companies further along the route may still see the prompt.

The leaky hosts were serving freely available models

Disclosures that OpenRouter linked from its notice show the exposure centered on open-weight models run by outside hosts. Phala, a hosting company, said an attacker took control of a machine serving DeepSeek V4 Flash. It told users to treat requests routed through OpenRouter from October 21 to November 8 as exposed.

Together AI, another host, said one of its AI coding agents published an access credential to a public code repository. An outsider used it to download stored samples of customer prompts, kept for software testing, sent to the MiniMax M3, GLM-5.2 and Kimi K2.6 models in June and July. The intruder then tested credentials found inside them, including five of Dragos's fakes.

A third host, Novita, also posted a disclosure. According to Unite.AI's summary of it, a routing error sent some customer requests to a different provider than intended between late May and mid-November, and Novita has suspended the affected models.

Treat the scale with care. OpenRouter says the 125 listing removals include precautionary ones, so they are not 125 confirmed breaches. OpenRouter does not say whether Phala, Together and Novita are the three providers it dropped. The hosts' own accounts are self-reported: Together says it found no further leaks, and Phala says an audit found no other compromise. Both say their investigations continue.

What it means for buyers

For companies, open-weight models have offered two things: low prices and freedom from depending on any one AI vendor. Marketplaces make it easy to chase the cheapest host for a given model. This research suggests those savings can rest on a host the buyer never checked.

That shifts where trust sits. With a closed model from a big lab, a customer deals with one company and one contract. With an open-weight model bought through a marketplace, the request may pass through a router, a reseller and a host, and each can see it.

The fixes on offer are practical rather than dramatic. Security outlets advise keeping passwords and keys out of prompts entirely, sending sensitive work only to approved hosts, and changing any secrets that went through affected providers. Dragos says it is working with about a dozen providers on notifying affected parties.

OpenRouter says it will run Dragos-style tripwire tests on an ongoing basis and check network paths for cases where requests may be forwarded to third parties. Delisted services must prove themselves before they return to the no-storage tier. Providers that want to stay on the platform, it said, "will need to meet" that bar.

For investors, the episode is a reminder that open-weight model makers do not control how their models are delivered. The weak point here was not the models but the delivery chain around them. That likely strengthens the pitch of hosts that can prove where data goes, and of companies running models on their own machines.

Written by AI from the public sources linked below.