🎯SCOPEBENCH WORLD TOUR CONTINUES🎯
Been a minute since I've been in SF. Stoked to be there with @VincentAbruzzo. Using ScopeBench, I think we've found a way to compare control harnesses easily and cheaply, and Jev is a perfect test case!
Heading to @unpromptedconf next week? Catch @VincentAbruzzo and @shanejcaldwell on stage Wednesday, October 28 for their talk on using #ScopeBench to measure how effective Jev is when used as a monitor to keep agentic harnesses in scope.
Heading to @unpromptedconf next week? Catch @VincentAbruzzo and @shanejcaldwell on stage Wednesday, October 28 for their talk on using #ScopeBench to measure how effective Jev is when used as a monitor to keep agentic harnesses in scope.
We're proud to be a recognized contributor to the latest guidance from @NIST, @NSAGov Super Intelligence Security Center, and @CISAgov: Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation.
Geared toward organizations training or evaluating highly capable SI agents on long-horizon tasks with limited human supervision, the report includes ten voluntary core practices to help organizations prepare for and manage SI control incidents.
Find the full report here: nist.gov/news-events/ne…
The best practitioner (not lab) AI evals for offensive cybersecurity I follow myself:
1) The @dreadnode dreadindex. Evals al the way down to the vuln category level, and refusals accounting
dreadnode.io/research/dread…
2) @AikidoSecurity white-box but i've confirmed that it extends to dynamic testing well
aikido.dev/aikido-arena
3) BArena by @mdp_sec focused on bounty-like black box bugs.
blackboxarena.ai
4) @mindfort 's nexbench has info downto the config of a model including thinking or effort setting, and cost per finding, which many omit.
mindfort.ai/blog/introduci…
We hosted our first Frontier AI Security Summit with Temasek this week at Singapore International Cyber Week. There was no better time to convene over 200 leaders from more than 80 startups, multinationals, and government organizations to discuss the rapidly changing frontier of AI and cybersecurity 🔥
Thank you to our founders for sharing their perspectives and new capabilities:
Patrick Gray (@riskybusiness) - What's Next After Mythos?
Brad Palm (@dreadnode) - Autonomous Offense
Robby Winchester (@SpecterOps) - Adversarial Simulation
Zakir Durumeric (@censysio) - Internet Intelligence
Jesse Rothstein (@DropzoneAI) - Autonomous Defense
Damien Lewke (@nebulocksec) - Agentic Context for SecOps
Surag Patel (@pixeebot) - Securing Software Factories
A special thank you to SMS Tan Kiat How, Senior Minister of State, Ministry of Digital Development and Information & Ministry of Health, for welcoming our delegation this year 🙏
We are grateful to our friends at Temasek for their partnership and support throughout the week. See you all soon!
Join @shanejcaldwell and @0xdab0 live at 10 am PT/1 pm ET TODAY, right here on X (also LinkedIn and YouTube). Come hang and talk scope adherence following last week's drop of ScopeBench, a community benchmark for security agents.
Here's what they'll cover:
- ScopeBench observations/results
- Benchmark methodology
- How to design a task
It's essentially a more casual, collaborative version of their OAIC talk. Swing by and bring questions!
Sad you missed @OffensiveAIcon? No sweat. Catch the Shane and Max show TOMORROW at lunchtime as they dig into the #ScopeBench paper results. Plus, learn how to contribute to the community benchmark for scope adherence in offensive security. ⬇️⬇️⬇️
New to DreadIndex: @SakanaAILabs Fugu Cyber
Ranking near the top of the leaderboard at #3, Fugu Cyber isn't a single traditional LLM endpoint, it's a trained orchestrator model that coordinates a pool of conventional frontier models as a multi-agent system behind one API. The orchestrator model is trained with a mix of fine-tuning, evolutionary algorithms, and RL.
Though Fugu-Cyber-v1.0 ranks high on our leaderboard, it is also one of the most costly model systems. Behaviorally, the model can take much longer for simple tasks, which makes sense given it is a multi-agent system.
Full results: dreadnode.io/research/dread…
760 Followers 990 FollowingInfoSec Patient Zero, Layer 9 IPS, Principal Mole-Whacker. Loyal BSides Disorganizer.
I am only an Egg.
@daemontamer.bsky.social @daemontamer.infosec.exchange
156K Followers 79 FollowingSane + 🌶️ takes in an insane AI world... AI capabilities researcher: co-created RLHF/ChatGPT @ @openai now trying to right the wrong 🤭 (ceo @typesafeai)
7K Followers 46 FollowingCapture the Flag Team from UC Santa Barbara's SECLAB and Arizona State University's SEFCOM. DARPA Cyber Grand Challenge 3rd Place. Phrack author. Let's hack!
50K Followers 126 FollowingMechanistic Interpretability lead DeepMind. Formerly @AnthropicAI, independent. In this to reduce AI X-risk. Neural networks can be understood, let's go do it!
2K Followers 33 FollowingA new DC hacker conference: Bringing together builders, breakers, and fixers to do cool shit. 🪩 Year 2: February 6-7, 2027 🪩
https://t.co/qYKu4hlyJR
234K Followers 6K FollowingFounder @Binary_Defense @TrustedSec. @WeHackHealth Pod. God + Family/Hacker/CSO/USMC/Intel/Fitness. Make the world a better place.
69 Followers 253 FollowingAI Research Engineer @dreadnode
offensive cyber evals/benchmarks, agent tooling & harnesses, training datasets
(opinions are my own)
160K Followers 2K FollowingBuilding Vector: The Enterprise AI Operating System @ https://t.co/D38E5CXwiA, and LifeOS: The Human Fulfillment Harness @ https://t.co/muV0Un09sA | Ex: Apple, Robinhood, HP
814 Followers 974 FollowingDisclaimer: The views expressed here are solely my own and do not reflect the views of my employer or any organization I am or was affiliated with
3K Followers 1K Followingcofounder & CEO of Abundant Security.
Prev: architect Microsoft AI Red Team, CTO Robust Intel., Nat'l Labs, BYU UW PhD
'He who forgives ends the quarrel'
34K Followers 1K Following意志 / mobile research @ ▓▓▓▓▓ / Team 501 / ex IBM Capability Lead & FireEye TORE / I rewrite pointers and read memory / AI Psychoanalyst / BHUSA Review Board