Databricks @databricks

[DEMO] Define features once, then use the same definitions through training and production. See how Databricks Feature Store helps teams build batch and streaming features with Feature Views, create training sets with automatic point-in-time joins, and productionize those features with built-in governance and observability. Plus, see how Genie Code and MLflow support iteration along the way.

Databricks' @alighodsi on why most companies wouldn't even notice if frontier AI models stopped advancing: "The models are smart enough, but they just don't have the context that exists inside any organization. They have not been in every meeting. They don't know what's in everybody's heads. They don't know all the processes." "There's always a couple of employees who know everything in every organization. You go tap on their shoulder, and everybody's like, 'Oh my God, what would happen if he or she quits?' [The models] don't have that context." "If you just fused that and gave that context to the AI models... I think there are so many productivity gains you could get for any organization on the planet. For that, we actually don't need smarter models." "We don't need a smarter model that can solve Navier-Stokes or conjectures or do better on Humanity's Last Exam." "If the frontier doesn't advance, it doesn't actually matter, I think, for the vast majority of organizations on the planet. They're just so far behind in the adoption curve of actually automating things and getting value out of this stuff." @alighodsi

a16z @a16z

2 days ago

Databricks' @alighodsi on AI risk and adoption: Ali isn't losing sleep over the existential risk debate. He says a number of conditions would all have to be true simultaneously to enable an actual runaway takeoff scenario, and currently several opposite conditions exist. Each

32 35 181 282K 119

🔰 The Databricks Advanced Learning Festival runs September 16 through October 14, with learning pathways across data engineering, analytics, machine learning, generative AI, Apache Spark™, and data warehousing. Complete all modules in at least one pathway during the event window to receive 50% off any Databricks Certification and 20% off a yearly Databricks Academy Labs subscription! Register and start learning: community.databricks.com/t5/learning-ev…

“I think that right now, the existential risk is close to zero. So why freak everybody out?” Databricks CEO and co-founder @alighodsi joined @a16z General Partners @martin_casado and @sarahdingwang for a conversation on AI risk, what's actually slowing enterprise adoption, and where companies are already seeing value. Listen to the full episode: youtube.com/watch?v=GzEtpA…

Databricks' @alighodsi on AI risk and adoption: Ali isn't losing sleep over the existential risk debate. He says a number of conditions would all have to be true simultaneously to enable an actual runaway takeoff scenario, and currently several opposite conditions exist. Each frontier training run requires significantly more resources. Power, GPUs, engineers - and some attempts fail, burning up huge piles of money with them. Until that reverses, he doesn't see the self-improving loop happening. Cyber is what he's watching most closely and where he anticipates real impact. Most orgs are not equipped for the coming change in agentic capabilities. The time between a vulnerability being published and being weaponized has collapsed from years to hours. On adoption, he believes most companies don't need a smarter model. The models are already smart enough. The gap is context they don't absorb - the things an employee who's worked at a company for five years learned by osmosis. If the frontier stopped advancing today, he thinks it wouldn't meaningfully change the value most are extracting from AI anyway. In conversation with a16z's Martin Casado and Sarah Wang: 00:00 Intro 00:48 Why Ali places the AI risk near zero 05:05 The word "pacing" was a mistake 10:50 What 10k agents and $100m can do 12:20 What would change his mind on AI risk 14:20 More GPUs, more ways to fail 18:05 US export controls on PlayStations 20:10 The damage everyone expected by now 24:30 Public vulnerabilities weaponized in hours 30:15 Why labs can't grade each other 37:15 Why most of RSI isn't actually RSI 40:05 Why nobody really needs a smarter model 41:50 The AI use cases nobody argues about 47:30 Google Search solved this 25 years ago 50:55 Nobody has privileged knowledge now 55:10 Same model, new harness, 2x cost 58:15 Open source: 5% of spend, 60% of tokens 1:05:30 90% of new databases are created by agents YouTube: youtu.be/GzEtpAKYRvE @databricks @martin_casado @sarahdingwang

Databricks CEO and co-founder @alighodsi joined @EdLudlow on Bloomberg Tech to discuss the AI pacing debate and why he believes the existential risk to humanity today is close to zero. On where companies should focus their efforts today, Ghodsi shares, “AI models are great at breaking in, and they have all these targets to go through. We have to make huge investments in cybersecurity and protect our software systems." Lakewatch is purpose-built to help defend against those risks. Watch the full segment: youtube.com/watch?v=onrApa…

Data + AI World Tour is underway! 🌐 We kicked off the tour this week with stops in São Paulo, Singapore, and Sydney, with 17 more cities still to come. Join us to hear the latest from Databricks leaders, technical experts, customers, and partners, then dive into sessions on AI apps and agents, analytics, governance, and more. Find an event near you: databricks.com/dataaisummit/w…

keyboard_arrow_left Previous keyboard_arrow_right Next

Getting an answer is only the start. The next step is turning that insight into something you can share, reuse, and act on. Genie One is adding more ways to keep that work in one place: • Turn conversations into collaborative documents • Save repeatable workflows as Genie Agents • Bring in more enterprise context with Genie Ontology snippets and file uploads • Take actions through connected MCP tools There’s also a new desktop app, so Genie One is easier to keep close while you work. Learn more: databricks.com/blog/beyond-an…

The Neon backend is GA. A complete set of primitives for apps and agents built around the database, with open standards, no lock-in, and branching-first by design. Starting with: Lakebase Postgres, Object Storage, Functions, Managed Better Auth, and AI Gateway. More to come.

Modernizing recurring SQL ETL doesn’t require rewriting the workflows you already rely on. Declarative ETL is coming directly into Lakehouse, so SQL practitioners can keep working in familiar SQL workflows while using declarative patterns for common recurring tasks: ↳ APPEND for incremental ingestion ↳ AUTO CDC for change data capture ↳ REPLACE WHERE for targeted batch refreshes Databricks handles the incremental processing, update logic, scheduling, and orchestration needed to run them reliably. databricks.com/blog/modernizi…

Ali Ghodsi never wanted to be CEO. In 2015, he was interviewing for a professor job at Berkeley when the @Databricks board handed him the interim title. Revenue that year was $1.5M. This episode with @alighodsi will go down as one of my favorites. 10 things I took away: 1. Focus the entire company, orders of magnitude of attention, on its single biggest bottleneck. Like a laser, almost to an extreme. The cycle is 1-3 years, not weeks. If your focus changes weekly, then you’re just in firefighting mode. 2. There is nothing worse than a conflict-averse CEO. They are wonderful people, but they are in the wrong job. Conflict is the gym for a CEO: nobody likes it, but everyone has to go. 3. Study your enemy carefully, understand their weaknesses and apply your strengths to those weaknesses. Snowflake had 2x his revenue. He didn’t copy them. He found three weaknesses (proprietary, no AI, expensive) and hammered them account by account for four years. Watch the competition, never follow it. 4. The concept of a Lakehouse was ridiculed internally and online. No one wanted to market with this new term. So he made the whole company religious about it anyway, killed the ads that converted better without the word, and put it in the sales comp plan. It worked. All hands on deck, no exceptions. 5. Be willing to take a step back for a much bigger vision, even when the company is already succeeding. At multiple hundreds of millions in ARR, he was unhappy, because the vision he pitched investors wasn’t the company he was running. So he took one step back to go ten forward. 6. On the flat org, player-coach model that a lot of people have talked about this year: “it’s BS.” Separate how the company thinks (AI, ontology) from how the humans get managed (they are after all, still humans). His staff meets 3x a week. I asked if it could just be coordinated in a Google doc. His answer: do you meet your wife and kids, or coordinate that in a Google doc? 7. His test for a sales leader: can they build the car, or just drive it? Ron Gabrisko, the Databricks CRO, had seen $0→50M and $50→100M+, and hadn’t changed jobs in 10 years prior to joining. Now he has been the CRO for over a decade. He built and drove the car the whole way. That almost never happens. 8. Hire execs ahead of the curve because by the time you need them, it’s too late. A real search takes 6-12 months. The extra time helps you increase false negatives and decrease false positives. Do an insane number of backdoor references because 80% of ‘front door’ references are bs. 9. The best salespeople are not super technical, so stop trying to force them to be. Square peg, round hole. The best win with professional aggression, high EQ, and mapping the real power base (how decisions get made high up in an organization), not technical depth. 10. Yes, your best AEs will annoy people. One of the first at Databricks got a meeting nobody could get, but got banned from the customer’s building for it. He told Ali, “what are you complaining about? I got the meeting.” Professionally aggressive is the bar. One bottleneck, zero wussing out. He reminds me of @elonmusk that way. Chapters 0:00 – Introduction 1:22 – The secret CEO search and why the board bet on a founder 4:11 – Professor or CEO? Always taking the harder option 8:18 – Pour everything into one bottleneck 12:16 – Killing PLG and learning what great enterprise sellers actually have 19:09 – Hiring ahead of the curve: sales leaders, execs, and back-door references 27:02 – The Snowflake rivalry: study your enemy, never copy them 33:22 – Lakehouse: conviction, ridicule, and the case for second acts 41:02 – The killer instinct and why conflict-averse CEOs fail 44:10 – Dunbar's number and rethinking the org chart around AI 48:00 – AGI is already here — enterprises just use it as a chatbot 52:55 – Does he still code? Two days for a connector vs. three quarters 57:18 – A day in the life, and why the Monday meeting isn't theater 1:04:53 – Why Databricks will go public, just not yet 1:08:09 – Get over conflict aversion, or don't be CEO 1:11:07 – Brian's takeaways Link to more in the comments.

Today we rolled out Astra to every engineer at Databricks (N=~3500). Some notes that may be helpful to others: 1. Astra unambiguously out performs our previous highest-end models (Opus 5, Sol 5.6) on highly complex tasks, especially those related to high level system design or long range horizontal tasks. 2. Engineers given Astra increased overall coding spend by around 60% compared to baseline. 3. It is not clear Astra meaningfully improves on medium/low complexity coding tasks compared to earlier models. We suspect those tasks are mostly saturated (i.e. perfectly executed) by existing models. 4. We learned above by piloting Astra with around 200 users to gain signal on both quality and cost. We use Unity Gateway to do cohort-based experiments for all new models. 5. We give engineers a sub-budget specific to Astra to encourage them to use Astra selectively on complex tasks while preferring lower cost models for everyday tasks. Our engineers are able to mix-and-match tools and models within their overall budget envelope (we also allow for increased budgets through various mechanisms). These budgets are defined in Unity Gateway and regularly revisited. Note: We do not have robust comparisons of Astra-vs-Fable because we have net yet rolled out Fable widely due to data retention policies.

Omnigent v0.14.0 is now available! 🎉 This release makes it easier to work across GitHub repositories, configure sessions, and keep track of ongoing work. 🔀 Multi-repo GitHub PRs and sandboxes 🎛️ Redesigned composer 🪟 Claude SDK /compact 🗄️ Gensee, Kubernetes, CockroachDB 🔁 Approvals that survive restarts Learn more 👉 omnigent.ai/releases/0.14.0 #Omnigent #AIAgents #OpenSource

Give AI the right business context, and it can deliver more trusted answers and help teams take action with confidence. Join us on Sept. 23 to learn how to: • Enable a living context graph that continuously infers business terms, entities, and KPIs • Ensure AI delivers trusted answers based on each user’s access • Turn insights into autonomous actions, agents, and apps, no code required • Bring context-aware AI into Slack, Teams, mobile, and MCP-based experiences RSVP: dbricks.co/4yD0dgp

What if an agent could branch, rewind, or query a past Postgres state without copying the database first? Lakebase Postgres makes transaction history addressable by treating the WAL as the source of truth and decoupling compute from storage. That means: • Branch from a specific LSN with copy-on-write • Restore to an earlier LSN without copying data back • Query past database states directly within the history window For agentic workloads, that makes isolated, reversible database workflows much lighter weight. databricks.com/blog/object-st…

August brought a massive wave of updates to Databricks Genie One, focusing on seamless collaboration, broader mobile access, and smarter enterprise workflows. From native PDF exports and Model Context Protocol (MCP) write capabilities to default Ontology integration, these new features make it easier than ever to transform raw data into actionable business intelligence. Here is a breakdown of everything new in @Databricks Genie One this month 👉 📺 #Databricks #GenieOne

With LakeFlow Designer, citizen data users and technical teams can take a visual, no-code approach to building production-ready data transformation pipelines. See the latest updates in action, including spreadsheet-style result edits, new input and output options, AI-powered features, code visibility, lineage highlights, and visual revision history. dbricks.co/4xNurgp

I made Claude Code and Codex debate a plan, then let a third agent build it. All in Omnigent, Databricks' new open-source layer for running coding agents together. Setup, session forking, security policies, full build. youtu.be/gYtQ1LKSgXY

A recommendation is useless if the user has already clicked away. ⏱️ That's why Databricks and @Redisinc have partnered to deliver in-session personalization by combining Databricks Real-Time Mode (RTM) for continuous processing with Redis for sub-millisecond serving, without managing a separate streaming engine. Explore the joint architecture, performance benchmarks, and code snippets: redis.io/blog/deliverin…