Arvind Narayanan @random_walker

Agree with essentially everything here. Bottlenecks are common, even when technological capability growth is very fast.

Arvind Narayanan @random_walker

55 minutes ago

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task

6 8 35 5K 36

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task automation ≠ process automation ≠ faster progress ≠ recursive self-improvement ≠ intelligence explosion. Source: anthropic.com/institute/meas… 1) The software engineering precedent: a year ago there were widespread hopes / fears that once AI can write ~100% of the code, software engineering output would explode (SaaSpocalypse! Everyone would create their own SaaS and dump their vendors) and that this would make software engineers obsolete. Since then, many companies and teams have basically hit that milestone, but neither of the assumptions proved true. Turns out we still need humans, and while shipping velocity has increased moderately, improvements in terms of actual outcomes for software users remain unclear. Besides, we’re still getting a better grasp on the negatives: code quality, long-term maintainability issues, and burnout. While the precedent is no guarantee, this should be our default expectation for what happens as the “Automation Level 4” line trends towards 100% — it won’t be a phase change. normaltech.ai/p/why-ai-hasnt… 2) The “production-progress paradox” is the fact that individual researchers’ productivity has been increasing while the rate of collective scientific progress has been slowing by most measures. AI exacerbates this because everyone uses the same or similar AI models, and ideas become homogenous over time. I suspect it’s too early to tell if this is going to bite companies that are plunging into AI-led research. normaltech.ai/p/could-ai-slo… Note: our own research on AI agents doing open-ended research shows limitations in creativity, judgment, and other areas. cruxevals.com/crux/can-ai-ag…. But it is possible that these could be overcome in the near future, so I’m discounting those limitations here. The production-progress paradox is a deeper issue that’s not AI-specific, though particularly applicable to AI-driven research. It’s about the fact that productivity increases are self-evident but true progress is not measurable as it happens (and only becomes clear in retrospect), so we end up optimizing for the wrong thing. 3) Let’s talk about automation level 5, which is still at 0% in Anthropic’s graph. It’s a bit unclear what Level 5 would look like, but it seems to be about full autonomy at the task level, and not the “AI builds its own successor” vision. My prediction for a while has been that even 100% task automation in most cognitive jobs won’t lead to any kind of discontinuity. youtube.com/watch?v=uiTwQG… What I expect will happen: if anything is understood well enough to be specifiable as a task, it can be handed off to AI, whereas the role of humans is entirely in the interstitial tasks — hard to formalize but still essential. So there will still be a human bottleneck. 4) This human bottleneck is a good thing and is essential for remaining in control. Humans don’t have to be in the loop on every task, but as long as there are enough touch points for oversight in the overall process, and adequate investment in improving human understanding and AI control, increasing AI capabilities doesn’t have to be bad for safety and, more broadly, collective human agency over AI. But “full RSI” is where this balance of agency can break. This kind of closed-loop process is arguably a much more important and tractable target for regulation than compute thresholds, superintelligence, or harm thresholds. I’m glad that OpenAI agrees that fully autonomous RSI may not be a good idea, in a just-released post: openai.com/index/building… “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.” 5) Finally, many people have written about why Recursive Self Improvement, even if achieved, won’t necessarily lead to superintelligence. Here’s my argument: normaltech.ai/p/what-will-be… The bottlenecks are external.

@PoliticalKiwi ... do we need to start publishing graphs of indicators of AI impact on the world that show no change. We could publish a lot of those. Here are some (very vibe coded) examples:

keyboard_arrow_left Previous keyboard_arrow_right Next

Perhaps unsurprisingly, and as many predicted, AI companies are sued under antitrust law for "pacing the frontier" efforts.

“When companies promise that they’re going to do a better job on safety, we shouldn’t have to take their word for it.” - @random_walker on @CBSNews discussing today’s AEF letter and our call for credible, independent AI evaluations. cbsnews.com/video/expert-a…

OpenAI has talked a big game about AI for cyberdefense. But when @HacktronAI broke into their internal repository and reported it, they received a bug bounty of just $6,500 because one of the vectors for the attack was "out of scope". This is atrocious. If we actually want a flood of defenders auditing these systems, companies need to take bounties more seriously. Signing letters isn't enough.

s1r1us @S1r1u5_

4 days ago

On July 25, we hacked OpenAI. Two bugs let us take over ChatGPT/Codex accounts of OpenAI employees (+some unaffiliated users) and reach connected services: Outlook, Slack, GitHub, etc. We proved it with a PR in OpenAI’s internal codebase . It took us <72h. 🧵

354 1K 12K 2.8M 6K

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiatives/em… Learn more at aievaluatorforum.org/path-ahead

CNBC @CNBC

3 days ago

Anthropic and OpenAI need truly independent safety evaluators, experts say in public letter cnbc.com/2026/09/18/ai-…

23 12 60 45K 8

I'm one of 100+ signatories to a public letter calling for five minimum requirements to make embedded evaluations credible, by establishing evaluator independence, protection from retaliation, and real access. ———— We, the undersigned, are encouraged to see frontier AI companies call for embedding third-party organizations to evaluate rapidly escalating AI capabilities and risks. We believe that all frontier AI companies should embed evaluators to independently assess AI risks, including evaluating the systems themselves and any significant incidents of real-world harm, as well as the companies’ training, deployment, oversight, operational, and safeguard practices. To be credible, embedded third-party evaluations must have scientific objectivity, transparency, independence, and robust protections against interference from the evaluated companies, including at least: 1. Frontier AI companies should rely on evaluators that are meaningfully independent, that maintain full editorial control, and that disclose and mitigate potential conflicts of interest. This includes at a minimum that embedded evaluation organizations should not be owned or governed by frontier AI companies, should not have other significant commercial business with them, and should not accept any form of payment or other reward contingent on the evaluator’s findings. 2. Frontier AI companies should incorporate differing viewpoints and areas of expertise, including by embedding multiple evaluation organizations across a range of priority risk areas, each with deep relevant technical expertise, as well as by allowing and encouraging evaluators to share how conclusions differ among evaluators and between evaluators and company employees. 3. Embedded evaluators should be transparent, including transparency about their methods and findings, the nature of their access, and the broader terms of the evaluation. Frontier AI companies should actively facilitate this transparency, including limiting the scope of non-disclosure agreements. They should also allow evaluators prompt and unfiltered communication with the companies’ boards and other privileged oversight bodies, as well as public release of findings and evidence, subject only to a time-limited redaction process restricted to protecting critical interests in intellectual property, customers’ sensitive information, individual privacy, security, and public safety. 4. Embedded evaluators should be shielded from retaliation from the companies they embed with for choosing reasonable evaluation methods, discovering information, or drawing conclusions that are unflattering to those companies. This includes reasonable protections against retaliatory litigation, as well as funding mechanisms that give them confidence they will remain funded even in these cases. 5. Frontier AI companies should grant embedded evaluators access equivalent to that of their own highly privileged employees for the purposes of their evaluations, and with exceptions to protect sensitive data belonging to the company’s customers and other third parties. This includes access to the same relevant systems, data, tools, and physical spaces as those available to senior internal company employees responsible for carrying out comparable risk assessments, as well as candid and direct one-on-one communication with relevant staff. This list is not comprehensive, and conditions like these to ensure credible evaluations should be increasingly standardized, codified, and enforced. One example is the set of terms defined in the AEF-1 standard, which has already seen early adoption, but far more work will be necessary to ensure that embedded evaluators are effective and meaningful. Embedded evaluations cannot address all oversight needs and should be treated as a complement to, rather than a replacement for, broader efforts by frontier AI companies to expand external oversight, including greater public transparency and additional, broader forms of access for independent researchers. ———— Full letter with links and signatures: aievaluatorforum.org/initiatives/em… I'm grateful to the AI Evaluator Forum (@aievalforum) for organizing this.

A post on risk management in the marketplace of ideas x.com/random_walker/…

Arvind Narayanan @random_walker

a year ago

Back in grad school, when I realized how the “marketplace of ideas” actually works, it felt like I’d found the cheat codes to a research career. Today, this is the most important stuff I teach students, more than anything related to the substance of our research. A quick

9 60 458 64K 569

A thread on how elite research universities are a tournament system: x.com/random_walker/…

Arvind Narayanan @random_walker

6 years ago

Professors at top universities are lottery winners, but rarely acknowledge the role of luck in their success. Be skeptical when they give you advice suggesting that the path they took is a repeatable one. If you aspire to an academic research career, have a backup plan.

72 771 6K 0 351

Students sometimes tell me they’re applying to PhD programs as a backup in case they don’t get a job as a software engineer or whatever. I get that the job market is rough but this is like saying you’ll start a company if you don’t find a job. Like starting a company, a PhD is the far *more* risky path and requires incredible grit over a long period. It isn’t just that you might realize after a couple of years that a PhD isn’t for you. The bigger risk is that you complete a PhD but end up overqualified on a niche topic after having spent 5-7 years on it. The research world, like acting, sports, music, or book publishing, is set up as a tournament, with lots of extremely competent people competing for a small number of highly desirable slots. We can talk about whether that’s fair and whether it’s inherent or can be changed, but as a professor I feel the least I can do is loudly and repeatedly warn people about what they’re getting into.

This is one of the central points in my ICML keynote, with a lot of detail — see part 2. normaltech.ai/p/what-will-be… Note: I take RSI seriously! One of our big empirical projects is evaluating agents' ability to do open-ended AI research. But it doesn't imply much about superintelligence, labor displacement, or doom.

Ramez Naam @ramez

5 days ago

We frequently conflate RSI with a fast-takeoff to super-intelligence (ASI). They are not the same. RSI means a model can improve on itself. That does not mean that it can do so more at each iteration, which is what would be required for a singularity. Far more likely that

29 18 141 23K 26

Over 2 years ago @sayashk and I wrote a detailed deconstruction of p(doom) and argued that its primary function is to launder vague, evidence-free intuitions and fears through a facade of quantification. It remains 100% relevant today. normaltech.ai/p/ai-existenti…

Itai Sher @itaisher

4 days ago

There is a strange laundering of this number. It was floated by Anthropic employees without public justification. Then mathematicians pick it up and now it is attributed to their judgment. It’s fine to survey AI practitioners, but posts like this are a bad epistemic practice.

11 14 112 38K 16

I think we have pretty good reason to accept that “AGI” is a meaningless term and a useless idea which should be retired. No one ever managed to agree on how you define AGI, but AI capabilities have improved enough to stress out Fields medalists, while the frontier is so “jagged” that regular users hate AI writing, and @random_walker was completely right about there being no discontinuity.

Timothy B. Lee @binarybits

4 days ago

When forecasting about AI (or any topic really) you should be skeptical of sharp discontinuities. People used to think we'd "achieve AGI" and everything would suddenly change. Now that we're closer the concept seems very fuzzy. I view recursive self-improvement the same way.

18 7 88 12K 10

I've been using Pangram's Gmail inbox labeler for about two months now. It's been pretty huge to be able to see cold emails labeled as AI and use this info to inform how I want to engage. With this feature, there was a big discussion internally about privacy. Zero data retention (ZDR) -- where Pangram doesn't store the result after processing -- is a policy typically reserved for enterprise customers. However, since email data is much more sensitive, we ended up deciding to turn on ZDR for all email inbox scans. No email inbox data will be retained by Pangram, and this applies equally to everybody. The other really important thing here was to give users a choice of what happens to their email. My preference today is to simply label AI emails but leave them in my inbox. This may shift In the future if AI emails ramp up in volume, so we have the option to make fully AI emails skip the inbox or go straight to spam. Please try it out and let me know any feedback you may have!

Pangram @pangram

5 days ago

AI detection, now for Gmail

5 14 208 160K 62

Yes. Bio-risk exists today, from zoonotic outbreaks, accidents, and bio-terror. We know a great many things we can and should do to increase societal resilience, and have done almost none. Independent of how AI changes the risk, we should be doing the basics here. 1. Pathogen detection in waste streams, water supplies, and perhaps even indoor air. 2. Proactive design of template vaccines for every known virus family. 3. Pre-build vaccine manufacturing capability to have on standby. 4. Physical countermeasures: UV in the HVACs of all large buildings, airports, etc..; Stockpiling PPE. 5. R&D into new approaches like PCANS and other techniques that block viral transmission. These are all no-regret bio-defense policies that increase our resilience to natural pandemics, lab leaks, or intentional bio-attacks (AI augmented or not). Defense in depth.

Olivia H. Scharfman @OliviaHelenS

5 days ago

Your position on AI should not affect your support for biodefense measures. Even if you do not think AI will make a difference, the problem is urgent today.

6 15 70 16K 23

Nihar Shah did a heroic experiment for TMLR: he spent 20-25 hours over two weeks interviewing authors of seemingly low-quality submissions about their own papers. He confirmed what we all suspected: people submitting these papers have *no idea* what is going on in them.

Transactions on Machine Learning Research @TmlrOrg

5 days ago

TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission? medium.com/@TmlrOrg/askin…

27 215 1K 692K 277

@MackenZ_arnold @sayashk Incredibly useful feedback! We'll chew on these. Thank you so much.

My toxic trait is loving even-handed, ecumenical takes. And @sayashk and @random_walker supply them in spades. A few thoughts and reactions: (1) This is what virtue looks like. Actual humility, a deep appreciation for uncertainty, openly updating their beliefs, and finding actionable areas of common ground. (2) Best line in the whole piece: “we do not need consensus on worldviews to have agreement on policy” — I’d put it even more strongly, such consensus is not possible; we have to work within that constraint. (3) Luckily many policies are robust to different assumptions. Their policy recs hold up, and that’s largely because they choose the right focus areas—Managing uncertainty and building resilience. I wrote more on the original policy recs in another post; I’ll link in the comments. (4) It’s striking how much their policy recs parallel the major AI policy proposals to date (including newer additions like embedded auditing and additional emphasis on liability reform). That still surprises many. (5) Despite these policies being chosen specifically for their robustness, most are hotly contested when actually proposed. I think the authors ought to ask why policies they’ve selected specifically for being agreeable no-brainers have not gotten traction amongst policymakers who like and cite their work. (6) I want to challenge the authors to consider whether they have unique leverage in unsticking some of these policies, and to act on it. (7) I do wonder if the AI as normal tech meme obscures the policy recs and encourages misreadings. I’d be curious for the authors to at least try writing some work that leads with the policy recs, and presents them in pithier form. The authors recognize this issue: “we are often mistaken as downplaying Al risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.” — but even now, I don’t think the urgency comes through to the average skim reader. (7) I’d like to see them incorporate ~adaptation/flexibility into their policy recs. Uncertainty = surprises and updates, and so many policies are too rigid to adapt. Maybe I’m just being a lawyer, but I’d like to see them stump for rulemaking authority, updating mechanisms for standards, etc. This is one of the most frequent errors in current policymaking and it seems very consistent with their thesis. (8) I think the authors, at times, underestimate how widely their takes are held amongst people they lump into the “AI Safety” category. I think a much larger portion of those folks agree with the diagnosis that the Hugging Face incident displayed large cultural and procedural safety lapses and an under-investment in control. There are legitimate disagreements, but I see the authors reaction as closer to the modal reaction than they seem to think. (9) I see this policy portfolio as highly overlapped with that of @law_ai_, and the emphasis on robustness to different assumptions has strong overlaps with both my own way of thinking and Radical Optionality radical-optionality.ai

Sayash Kapoor @sayashk

7 days ago

What does it mean to pace the frontier? Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can improve safety and what companies should do to pace the frontier. The result

50 106 353 124K 324

AI flooding of administrative agencies and trial courts is a huge problem, and is driving adjudicators to use AI tools in response. Slop for slop, and the whole world goes blind.

Arvind Narayanan @random_walker

5 days ago

Floods might be the most under-recognized AI harm. By AI flood I mean overwhelming communication channels, forcing them to shut down or imposing a huge time cost on recipients. Three reasons we don’t talk about them much: 1) There are 50+ kinds of floods, so it registers as 50

15 26 155 27K 85