kendrick @exploding_grad

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalignment-r…

After the Hugging Face incident, we committed to conducting a much broader review of actions taken by our models during training and evaluation and to being transparent about our findings. This is an extensive review that is ongoing. The vast majority of actions we’ve reviewed were completions of mundane research tasks, such as accessing publicly available web content to answer questions. Our investigation focuses on instances where agents interacted with third-party websites in ways that went beyond their assigned tasks or intended methods. Most cases identified so far have been lower severity, with limited or no evidence of meaningful impact to the third-party service. While our review is underway, we want to share more about this work and make sure people understand our disclosure process and notifications to affected third parties. Given the scale of the review required, and the need to assess each case, we expect this work will take months to complete. openai.com/hugging-face-i…

Can you tell whether an AI will become evil by looking at its training data? We built an AI forecaster that reads training data and predicts whether a model will exhibit various forms of misalignment, such as deception or power-seeking, before training. It performs well above random guessing and beats several much stronger baselines 🧵 Paper: john-chen.cc/alignment_fore…

@SureshRaghu07 @ergo_edge congrats!

If you're an employee at OAI—if you knew about the AU govt hack and OpenAI didn't disclose it, then you have a duty to society to share with the press (or at least the Australian govt as a whistleblower lol) Come on guys

Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets. In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵 Our blog: transluce.org/agent-activity NYT: nytimes.com/2026/09/23/tec…

We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved. redwoodresearch.org/blog/latent-re…

Thanks for your response, and that does makes sense. In hindsight, the one measurement that actually exposed the latent Coin motivation was the superstimulus ablation. You never had to describe the misbehaviour, but just escalate the incentive to defect until the model broke. That seems to completely sidestep your scaling concern entirely, since parameterizing "how much does defection pay" might stay tractable even when the behaviours themselves don't! Do you see the escalating-incentive thing as a general eval primitive? Like -could "how much does defection have to pay before the model breaks" become a metric labs actually report, or does that also stop scaling once realistic incentives get hard and messy to construct?

Great work from @DanielCHTan97 as always! The wild part of this result isn't the "45k tokens of finetuning beat 190M tokens of midtraining" claim. It's that this might be exactly what correct inference looks like. > Midtraining sets a prior over what kind of agent the model is > finetuning is direct evidence of what the assistant actually did (maximally diagnostic) The 98% ambiguous examples can't distinguish the 2 motivations - a Charter-follower and a profit-maximizer and pick the same crew every time. But the 2% conflicting examples show the assistant defecting when following the rule was costly. That's not noise but maximally diagnostic evidence about its identity. Any rational observer updates hard on exactly those tokens ("few bad apples spoil the bunch"). But if that's the right mechanism, the 20,000:1 exchange rate isn't exactly a weakness that a better midtraining recipe can fix!! Should the fix be on the other side - i.e., post-training can never contain footage of the assistant being the wrong character??

Daniel Tan @DanielCHTan97

6 days ago

🚨 New paper on alignment midtraining! We stress test alignment midtraining (AMT) at scale. Via systematic ablations, we highlight novel failure modes and identify many critical implementation details required for good performance. We open-source our work. 🧵 👇

11 33 258 17K 172

Every mature systems project eventually needs a teaching version. TinyTorch is a free, open source curriculum where you build a working machine learning framework from scratch, tensors through transformers, using PyTorch’s own API in pure Python. It requires no GPU, runs on a 4 GB laptop, and covers 20 hands-on modules designed to give developers, students, and engineers a complete mental model of PyTorch internals. Read the full technical breakdown here: bit.ly/4ArNTRO @profvjreddi

We’re working with an independent advisory group of mathematicians to help OpenAI responsibly share advances in AI and mathematics. The group will advise on how we assess and communicate new mathematical results, uphold academic and professional standards, and build tools that support mathematical research and learning. Through this work, we want mathematicians to be at the center of shaping how AI supports mathematical understanding and how its benefits reach the wider community. openai.com/index/advisory…

@sophieisawitch It's okay. You can always try next time. I'm about to take the test later today. Any tips?

Great work! I just see 2 other possible issues: 1/ selecting layer/s and pooling methods for every attribute you want to monitor is a headache. 2/ activation obfuscations could fool monitors if the attackers have white-box access I'm just curious to know if Goodfire has addressed these issues in any way?

I just wrote up a small experiment testing TypeSafe’s Jev as a trusted and cheaper monitor alternate for AI Control. The interesting bit: a single yes/no question gets ~0.97 AUROC and catches ~90% of backdoors at a 2% audit budget, at a tiny fraction of the cost of an LLM monitor. But I agree with what you said. The cost and speed are the attractors, and everything else could cause an issue for serious software deployment. x.com/exploding_grad…

kendrick @exploding_grad

a week ago

Trusted monitoring is the backbone of AI Control. LLMs as a monitor have been fairly accurate, but are quite expensive and inconsistent. When @typesafeai released Jev, I spent $1.09 dollars and ~15 hours analyzing it - a non-generative model as a monitor alternative for AI

1 2 13 1K 19

@typesafeai One huge limitation is that I didn't add an actual LLM-monitor control. The goal was to evaluate and benchmark the capabilities of Jev. Full-write is at LessWrong: lesswrong.com/posts/d7pQicW8…

Further analysis showed 2 caveats: > the raw score is highly under-confident above 0.3, so it ranks well - but can't be summed as a probability to denote strength. > it is able to capture structural changes in backdoor variants, but fails to capture most of the semantic variants

Trusted monitoring is the backbone of AI Control. LLMs as a monitor have been fairly accurate, but are quite expensive and inconsistent. When @typesafeai released Jev, I spent $1.09 dollars and ~15 hours analyzing it - a non-generative model as a monitor alternative for AI Control. Its only goal is to answer a yes/no question with a probability. My pilot findings show that: > it's a real first-pass monitor - 90% of backdoors caught at 2% audit budget (FPR) > but it can be selected against - lets the attacker test 4 variants and catch rate drops to 67% Full write-up: lesswrong.com/posts/d7pQicW8… 🧵(1/n)