Dear friends,
A few frontier labs have tried to tell a story of open models being dangerous because they can be used to launch cyberattacks, and of their “safe” proprietary models with strong guardrails being there to defend us. This week, the opposite happened: Users of a closed model unintentionally launched a significant cyberattack; other closed models then failed to defend against the attack because of their guardrails. Ultimately an open model that was not hobbled by excessive guardrails assisted the defense.
The details of what happened are still emerging, but it appears that researchers at OpenAI, while testing one of their systems, accidentally allowed their autonomous agent to attack Hugging Face’s infrastructure. It succeeded and gained unauthorized access to some datasets and credentials. This attack was unusual in that the attacking agent orchestrated tens of thousands of automated actions.
Hugging Face took logs from the attack and tried to analyze them for defensive purposes using a commercially hosted LLM, but the LLM refused to do so on safety grounds. Thus, Hugging Face ended up using the open GLM 5.2 model to analyze the logs to help them understand and respond to the attack. Hugging Face pointed out a further advantage of using GLM 5.2: It allowed them to do the analysis on their own infrastructure, and none of the sensitive logs, attacker data, or credentials had to be sent to a third party provider.
Guardrails on LLMs do have a place. There are certain requests, such as for detailed directions to harm oneself or others, or for clearly criminal acts, that we’re better off having models refuse. But rather than trying to make LLMs “safe,” I would rather we put greater emphasis on making sure their use is responsible. There’s only so much one can do to make a tool like a hammer safe, and whether it helps or harms is more a function of using it responsibly than how it was made.
But a meaningful fraction of work on AI safety is no longer about safety but rather aimed at stoking fears to pursue regulatory capture. As David Sachs points out, “There’s no reason to limit American models on tasks that Chinese models handle without issue. We’re only making ourselves less competitive.”
I believe that open weight models, and more generally openness — despite some companies falsely saying it is dangerous — casts sunlight on technology and ultimately makes it safer. With the release of GLM 5.2 and the upcoming release of Kimi K3’s weights, open weight models have almost caught up to proprietary frontier models. Consequently, the proprietary model providers are dramatically accelerating their lobbying efforts to hamstring their open weight competitors. As Bill Gurley points out, open sourcing is a well established business strategy, not a danger to be licensed and contained.
While it is unfortunate that Hugging Face was accidentally attacked, I’m glad that at this moment, when anti-open model lobbying is at its most intense, we have a clear example of why open models actually make cyber defense easier and thus increase safety.
Let’s keep speaking up for and defending open source and open weight models!
Keep building!
Andrew