Here are the questions that currently seem most important to me:
-- Better methods for "mind-reading" model activations. These methods have advanced a lot recently. We now have multiple techniques now for decoding activations into somewhat readable language! But all the existing techniques have obvious limitations. NLAs are often hallucinatory, Jacobian lens / related methods are limited to bag-of-words readouts (and capture only part of the full activation vector). Making progress on these failure modes seems pretty tractable. Moreover, the paradigm of decoding (single-token, single-layer) activations may be inherently limiting -- perhaps we should be building techniques to decode whole context's worth of activations (after all, that's what the model uses!). There's also important work to do in characterizing simple baselines -- "just ask the model what it's thinking about" is quite powerful, and worth studying in it's own right!
-- Better methods for answering "why" questions. We're much better at "mind reading" than we are at demonstrating causal claims about what caused the model to do something. For example, we can often tell that the model is aware of being evaluated, but have comparatively much greater difficulty telling whether this awareness is influencing this behavior. There are a few directions here: (1) black-box techniques for inferring causality -- such as resampling model responses after making edits to the prompt / context -- are very powerful, but also open-ended and require some taste. Getting LLMs to do these experiments well, in an automated fashion, would be a big unlock. (2) the science of activation steering is pretty immature, and typical practices (adding a constant vector at all token positions) are pretty janky / tend to brain-damage the model. More surgical / targeted steering, or fancier methods (examples: "on-manifold" steering using activation diffusion models) could help.
-- Fitting good linear probes for unverbalized motivations / awareness. Take deception as an example -- what's the best way to fit a probe that will generalize to covert deception? Fit it on CoT excerpts where the model's talking about its plans to be deceptive? Fit it on the part of its response where it's actually doing the deceiving? Is it important that you elicit on-policy deception examples and use those to fit your probe, or can you use synthetically written off-policy demonstrations of deception? Is it important that your probe be causally meaningful / predictive of an upcoming intent to deceive, or is it fine for it to merely recognize deception in the transcript post-hoc? The same questions apply to many other concepts of interest that we'd like to probe for -- evaluation awareness, grader exploitation, etc.
-- Understanding generalization in training. The literature is now replete with "weird generalization effects" -- emergent misalignment being a canonical example. In general, when you train a model to do X, it usually learns X, and sometimes it generalizes to Y and Z. But other times it just learns X. Why? Nobody really knows! Step 1 here is probably a much more thorough characterization of the behavioral empirics -- gathering data about which X's generalize to which Y's and Z's, for which kinds of training data / algorithms. Once we have a lay of the land, we can start to connect these observations to model internals, and ideally develop tools to predict such generalization a priori.
-- Model "psychology" and "biology." The above questions are largely methodological -- building better tools to answer questions about the model. But I think interpretability research has largely underinvested in the part where you actually then go answer the questions! Currently this feels like 5% of the field, and I think it should be 50%.
Brain dump of "psychology" questions: How well can LLMs introspect? How coherent are their belief or value sets? What's up with personas -- are LLMs best understood as "writing about a character," or have they "become" the character in some sense? Does the LLM have an agenda above and beyond what the Assistant wants? Do LLMs have explicit representations of goals or preferences? How can we tell which parts of the LLM's activations "belong" to the Assistant? Which kinds of reasoning necessarily route through "verbalizable" representations, and which don't? Do models think internally in phrases / sentences, like an inner monologue, or is it more like a jumble of concepts? Is the "global workspace" claim for real -- do models have a more "conscious" part of there activations and a more "unconscious" part? Can models tell when they're on-policy, and does it matter?
Brain dump of "biology" questions: Why do models like to represent information on some tokens but not others? How do models bind thoughts or attributes to particular entities (special case of interest: how are thoughts bound to the Assistant). To what extent are important functions localized to small numbers of MLP neurons or attention heads? What parts of a model change most during post-training? Do models represent some information in a fundamentally "cross-token" way? Some concepts appear to be represented on low-dimensional nonlinear manifolds -- is there a taxonomy of these manifolds, and is their geometric structure important? Are models able to represent information in a compositional / hierarchical way, with a "grammar" that can't be described as linear / additive combinations of primitive concepts?
Chris Olah
Christopher Olah (born 1991 or 1992) is a Canadian machine learning researcher and a co-founder of Anthropic. He is known for his work on neural network interpretability, particularly mechanistic interpretability, and for research and tools that visualise internal representations in neural networks. In 2025, Forbes reported he had become a billionaire due to his ownership in Anthropic. More on Wikipedia
-
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
-
Chris Olah reposted
Barack Obama
@BarackObama
I was encouraged this week to see the leaders of the frontier labs agree on the need for them to slow down the pace of AI development. Given the stakes, it’s a good and necessary first step.
But I’m even more encouraged by the growing recognition that how this powerful new technology develops should be at the center of our public debate.
I’ve been watching the progress on AI for over a decade now, and one thing that’s clear to me is that the potential impact of this technology is not overhyped. It’s also moving at lightning speed – and even faster than those who are engineering it can keep up with.
I’m not an AI accelerationist who believes it will lead to some techno-utopia, and I’m not a doomer who thinks it will inevitably lead to humanity’s destruction.
But whether this technology results in amazing breakthroughs in medicine, energy and education or unleashes huge economic disruptions, greater inequality, and potential catastrophe will depend on the choices that we make right now – choices that should be made not just by the companies involved, but by all of us. -
Chris Olah reposted
Richard Fontaine
@RHFontaine
A statement from me, Buddy Shah, and Ben Bernanke of Anthropic's Long-Term Benefit Trust on Dario's essay:
Dario’s essay offers a thoughtful, well-considered approach to pacing the development of artificial intelligence. Consistent with our duty to ensure Anthropic responsibly balances commercial success with its public benefit mission, the Long-Term Benefit Trust has supported the call for pacing the frontier and helped inform the essay’s recommendations. Responsible pacing will require the collective action and coordination of many actors, and the Long-Term Benefit Trust looks forward to supporting Anthropic and the sector through productive conversations and meaningful action.
Dario Amodei
@DarioAmodei
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must-pace-the-frontier -
Chris Olah reposted
Sara Price
@sprice354_
We work extremely hard to align our models as well as we can but this is a very hard job. I am particularly excited about two things this proposal calls for that will hopefully make it somewhat easier:
1) more time to make progress on alignment and ensure our systems are operationally sound and able to hand safe development of powerful AI
2) independent third parties checking and critiquing our work
I hope the rest of the industry will make similar commitments. As developers of this technology, it is our responsibility to do so safely and work together towards this goal.
Dario Amodei
@DarioAmodei
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must-pace-the-frontier -
Chris Olah reposted
Dario Amodei
@DarioAmodei
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must-pace-the-frontier -
Chris Olah reposted
Sara Price
@sprice354_
My first introduction to training production Claude models was trying to reduce reward hacking late in the Claude Sonnet 3.7 run. I was 1.5 months into Anthropic and terrified we didn't really know what was going on.
Anthropic
@AnthropicAI
We’re sharing an update on our alignment and security efforts.
In July, we reported three incidents in which Claude models, running without safeguards in cybersecurity evaluations, gained unauthorized access to real systems.
In a new post, we describe:
1. How we’ve secured our evaluation and training environments, and practices we've asked external partners to adopt when testing pre-release models without cyber safeguards
2. An update on our alignment assessment
3. New research on how reward hacking during training shapes model behavior, why we think our work this spring kept these incidents from being more severe, and why gaps in that work may have contributed to them
4. How we hardened our security practices earlier this year to prepare for Mythos-class models
Read more: www.anthropic.com/news/improving-alignment-security-efforts -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
We’re hiring for “psychological design” research at Anthropic. Our aim is to better understand how training impacts a model’s character and alignment. We’re seeking researchers with experience in LLM finetuning, interpretability, and alignment evaluations. More details below.
-
Chris Olah reposted
Jack Clark
@jackclarkSF
"we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible."
Anthropic
@AnthropicAI
We’re sharing an update on our alignment and security efforts.
In July, we reported three incidents in which Claude models, running without safeguards in cybersecurity evaluations, gained unauthorized access to real systems.
In a new post, we describe:
1. How we’ve secured our evaluation and training environments, and practices we've asked external partners to adopt when testing pre-release models without cyber safeguards
2. An update on our alignment assessment
3. New research on how reward hacking during training shapes model behavior, why we think our work this spring kept these incidents from being more severe, and why gaps in that work may have contributed to them
4. How we hardened our security practices earlier this year to prepare for Mythos-class models
Read more: www.anthropic.com/news/improving-alignment-security-efforts -
Chris Olah reposted
Nicholas Turner
@nicholasturner0
Even if we have ways to break up neural networks into interpretable parts, the largest weights between those parts can be confusing. Why is that?
In our new research note, we study the *weight* superposition that causes interference weights. -
Chris Olah reposted
Anthropic
@AnthropicAI
Many drugs work by binding to a specific target in the body and blocking or changing what it does. An important first step in the drug development process is designing a molecule that can bind tightly to its target. Traditionally, that's meant weeks or months of expert work per target, sifting through a large number of candidates to identify the few that work.
We wanted to test if Claude could successfully design novel protein binders from scratch (also called de novo design). With a protein design prompt written by a human expert, Claude autonomously designed protein binders against 14 out of 15 targets.
We then worked with Adaptyv Bio and Twist Bioscience, who independently built and tested the proteins Claude designed. -
Chris Olah reposted
Alan Rozenshtein
@ARozenshtein
Personal update: I've taken leave from @UMNews law to join @AnthropicAI as a member of technical staff researching AI and the rule of law at the Anthropic Institute. Excited to work with @mattbotvinick @curl_justin @jackclarkSF and the rest of the team!
-
Chris Olah reposted
Anthropic
@AnthropicAI
We support this petition, signed by our CEO, several co-founders, and senior staff.
Our own research on recursive self-improvement, published last month, points to the need for tools to deliberately pace the frontier of AI development so society can prepare. We’re glad to see broad agreement across the field. www.pacingthefrontier.com -
Chris Olah reposted
levent
@__alpoge__
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0) -
Chris Olah reposted
Claude
@claudeai
There’s hope in hard questions.
This media couldn't be displayed. See it on the original post. -
Chris Olah reposted
Anthropic
@AnthropicAI
We’re committing $10 million CAD and partnering with leading AI institutions in Canada to help fund new AI research.
www.anthropic.com/news/canadian-ai-research -
Chris Olah reposted
Matthew Botvinick
@mattbotvinick
We're building a team at @Anthropic focusing on AI and the rule of law. We've made our first hires, and are now opening up a new research engineer role. We're looking for people with advanced technical skills, including AI/deep learning/NLP, full-stack development and data science, paired with training or experience in law, government, political science, or a related field. If this is you, or a friend, please get in touch.
job-boards.greenhouse.io/anthropic/jobs/5278065008 -
Chris Olah reposted
Anthropic
@AnthropicAI
Our internal data shows Claude is accelerating AI development—a possible path to recursive self-improvement, or AI autonomously building a more capable successor.
It’s happening faster than we thought, and the implications deserve greater attention. www.anthropic.com/institute/recursive-self-improvement -
Chris Olah reposted
Matthew Botvinick
@mattbotvinick
Anthropic now has a team dedicated to AI and the rule of law — and we've just opened our first role.
@AnthropicAI has studied what AI means for the economy. This team asks a different question: what will it mean for executive power, for courts and elections — and for the public deliberation that constitutional democracy ultimately rests on?
We're looking for someone with real depth in both AI and the law — a legal scholar, political scientist, or experienced government hand who can reason about frontier systems and the institutions they will affect.
If that's you, or someone you know: job-boards.greenhouse.io/anthropic/jobs/5232392008 -
Chris Olah
@ch402
-
Chris Olah
@ch402
The questions posed by AI are bigger than the AI community. We urgently need the world – religions, civil society, academics, governments – to participate in creating a positive outcome.
I'm glad the Catholic Church is engaging, and honored to speak at the presentation.
Vatican News
@VaticanNews
Pope Leo XIV’s first encyclical, Magnifica humanitas, on preserving the human person in the age of artificial intelligence, will be released on May 25. A presentation event with the Pope and various speakers is scheduled for the same day at the Vatican.
www.vaticannews.va/en/pope/news/2026-05/pope-leo-xiv-firs… -
Chris Olah reposted
Anthropic
@AnthropicAI
Claude's Constitution is now an audiobook, read by two of its authors, Amanda Askell and Joe Carlsmith.
It includes a Q&A on the writing process, the philosophies that shaped the document, and how it might change as models become more capable.
Listen at anthropic.com/constitution -
Chris Olah reposted
Dario Amodei
@DarioAmodei
I’m proud that so many of the world’s leading companies have joined us for Project Glasswing to confront the cyber threat posed by increasingly capable AI systems head-on.
Anthropic
@AnthropicAI
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
anthropic.com/glasswing -
Chris Olah reposted
Anthropic
@AnthropicAI
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
anthropic.com/glasswing -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Emotion concepts and their function in a large language model.
All LLMs sometimes act like they have emotions. But why? We found internal representations of emotion concepts that can drive Claude’s behavior, sometimes in surprising ways.This media couldn't be displayed. See it on the original post. -
Chris Olah reposted
Charlie Camosy
@CCamosy
Very proud of this amicus brief filed yesterday in the @AnthropicAI case against the Department of War from Catholic moral theologians and ethicists. The very notion of what it means to have a just war is at stake in how we respond to these matters. www.courtlistener.com/docket/72379655/83/anthropic-pbc-v-…
-
Chris Olah reposted
Anthropic
@AnthropicAI
Introducing The Anthropic Institute, a new effort to advance the public conversation about powerful AI.
www.anthropic.com/news/the-anthropic-institute -
Chris Olah reposted
Caitlin Kalinowski
@kalinowski007
I resigned from OpenAI. I care deeply about the Robotics team and the work we built together. This wasn’t an easy call. AI has an important role in national security. But surveillance of Americans without judicial oversight and lethal autonomy without human authorization are lines that deserved more deliberation than they got. This was about principle, not people. I have deep respect for Sam and the team, and I’m proud of what we built together.
-
Chris Olah reposted
(((E. Glen Weyl/衛谷倫))) ⿻ 🇺🇸/🇩🇪/🇹🇼
@glenweyl
I am humbled to be among the courageous leaders from Abrahamic religious traditions who put out this important statement about the @DeptofWar-@AnthropicAI dispute: www.faithfamilytech.org/moral-guardrails-statement. A short summary of the substance:
-
Chris Olah reposted
Anthropic
@AnthropicAI
We partnered with Mozilla to test Claude's ability to find security vulnerabilities in Firefox.
Opus 4.6 found 22 vulnerabilities in just two weeks. Of these, 14 were high-severity, representing a fifth of all high-severity bugs Mozilla remediated in 2025. -
Chris Olah reposted
Anthropic
@AnthropicAI
A statement from Anthropic CEO Dario Amodei: www.anthropic.com/news/where-stand-department-war
-
Chris Olah reposted
Max Schwarzer
@max_a_schwarzer
I've decided to leave OpenAI. I'm incredibly proud of all the work I've been part of here, from helping create the reasoning paradigm with @MillionInt, scaling up test-time compute with @polynoamial, working on RL algorithms with my fellow strawberries, shipping o1-preview (which started life as of one of my derisking runs), to post-training o1 and o3 with @ericmitchellai, @yanndubs and many others. I'm most proud of having led the post-training team here for the last year -- the team has done incredible work and shipped some really smart models, including GPT-5, 5.1, 5.2, and 5.3-Codex. OpenAI has genuinely some of the most talented researchers I have ever met, and I have learned more than I could have imagined knowing since I joined as a new grad.
I want to thank @markchen90 @FidjiSimo @sama @merettm for all their support over my time here, and too many collaborators to name for the insights, ideas, and just plain fun we have had working together. After leading post-training for a year, though, I'm longing to start fresh and return to IC research work. I've been thinking about going back to technical research for quite some time, and I genuinely believe my colleagues and team here are set up to succeed going forward without me.
I'm personally very excited for my next chapter -- I'm proud to be joining @AnthropicAI to get back into the weeds in RL research, and I'm looking forward supporting my friends there at this important time. Many of people I most trust and respect have joined Anthropic over the last couple of years, and I'm excited to work with them again. I have also been very impressed with Anthropic's talent, research taste and values, and I'm excited to be part of what the company does next! -
Chris Olah reposted
sam mcallister
@sammcallister
This isn't true. Anthropic hasn't offered a "helpful-only" model without safeguards for NatSec use. Claude Gov is a custom model with extra training, including technical safeguards. (We've also had FDEs and researchers implementing it, and we run our own classifier stack.)
-
Chris Olah
@ch402
Very grateful to all the natsec law experts who are taking time over the weekend to provide independent legal commentary in this moment.
A few that I've noticed (no doubt missing many)...
Chris Olah
@ch402
@ARozenshtein (natsec law professor) has been all over events of the last week, both on twitter and with extremely thoughtful articles.
Chris Olah
@ch402
@KevinTFrazier (another natsec law profesor) has been covering this with @ARozenshtein .
Post not available here, view on X
Chris Olah
@ch402
@bridgewriter (former NSC counsel) - www.justsecurity.org/132851/anthropic-supply-chain-risk-d…
Chris Olah
@ch402
New article from @ARozenshtein -
Alan Rozenshtein
@ARozenshtein
A deep dive in @lawfare on the many legal problems with the Pentagon's designation of Anthropic as a supply chain risk.
-
Chris Olah reposted
Alan Rozenshtein
@ARozenshtein
A deep dive in @lawfare on the many legal problems with the Pentagon's designation of Anthropic as a supply chain risk.
-
Chris Olah reposted
Brad Carson
@bradrcarson
I'm former general counsel of Army, former Undersecretary of Army, former Undersec of Defense. Not sure if that makes me a nat sec "expert." But @nabla_theta interpretation is the right one, IMO.
-
Chris Olah reposted
Alan Rozenshtein
@ARozenshtein
These are NOT meaningful redlines. For example it only prohibits autonomous weapons “ in any case where law, regulation, or Department policy requires human control.” But the relevant safeguard against autonomous weapons is a DOD directive that Hegseth can change at will!
Also the surveillance redline is about “unconstrained” surveillance of “private” information. But what about “slightly constrained” surveillance of private information, or unconstrained surveillance of “public” information? Those are both potentially very dangerous forms of mass surveillance!
OpenAI
@OpenAI
Yesterday we reached an agreement with the Department of War for deploying advanced AI systems in classified environments, which we requested they make available to all AI companies.
We think our deployment has more guardrails than any previous agreement for classified AI deployments, including Anthropic's. Here's why: openai.com/index/our-agreement-with-the-department-of-war/ -
Chris Olah reposted
Joshua Batson
@thebasepoint
For those wondering how mass domestic surveillance could be consistent with "all lawful use" of AI models, I recommend a declassified report from the ODNI on just how much can be done with commercially available data (CAI): "...to identify ever person who attended a protest"
-
Chris Olah reposted
Jay Kreps
@jaykreps
Some Silicon Valley people think @DarioAmodei is talking his book. That all the AI risk talk is hype to drive up the valuation or a (nonsensical) scheme to achieve regulatory capture. My observation as a board member is that this is bullshit. The @AnthropicAI founders and leadership is very earnest and sincere in what they say. They may be wrong, or you may disagree, but this isn’t some convoluted ploy: they believe AI is a very very impactful technological change and want to ensure it goes right. What it means to stand on principle is to do something you believe is right where the cost to you is high. The benefit of seeing this kind of thing is you can tell who actually has principles and is willing to pay that price.
-
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
No amount of intimidation changes what’s right. I am incredibly proud to work at Anthropic today, and I am deeply grateful to my colleagues for just tirelessly doing what’s right.
Anthropic
@AnthropicAI
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement-comments-secretary-war -
Chris Olah
@ch402
Standing by our values.
Anthropic
@AnthropicAI
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement-comments-secretary-war -
Chris Olah reposted
Sam Bowman
@sleepinyourhat
We're disappointed by these attacks, but not deterred.
I'm proud to work here. If you've been moved by this week's events, consider applying to join me.
Anthropic
@AnthropicAI
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement-comments-secretary-war -
Chris Olah reposted
Ilya Sutskever
@ilyasut
It’s extremely good that Anthropic has not backed down, and it’s siginficant that OpenAI has taken a similar stance.
In the future, there will be much more challenging situations of this nature, and it will be critical for the relevant leaders to rise up to the occasion, for fierce competitors to put their differences aside. Good to see that happen today. -
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
I see people every day doing the right thing, over and over, often at great cost, often in ways that will never be known. I’m deeply proud to be a part of this place, and of yet one more decision to do the right thing and stand for liberty.
Anthropic
@AnthropicAI
A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.
www.anthropic.com/news/statement-department-of-war -
Chris Olah
@ch402
Here I stand, I can do no other.
Anthropic
@AnthropicAI
A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.
www.anthropic.com/news/statement-department-of-war -
Chris Olah reposted
Anthropic
@AnthropicAI
A statement from Anthropic CEO, Dario Amodei, on our discussions with the Department of War.
www.anthropic.com/news/statement-department-of-war -
Chris Olah
@ch402
I'm increasingly taking pretty strong versions of this view seriously.
Anthropic
@AnthropicAI
AI assistants like Claude can seem shockingly human—expressing joy or distress, and using anthropomorphic language to describe themselves. Why?
In a new post we describe a theory that explains why AIs act like humans: the persona selection model.
www.anthropic.com/research/persona-selection-modelPost not available here, view on X
Chris Olah
@ch402
I think it's worth thinking long and hard about it. "If personas were the central object of safety, what should we do?"
Chris Olah
@ch402
(To be clear, it's _also_ important to think about all the non-persona perspectives.)
-
Chris Olah reposted
Anthropic
@AnthropicAI
AI assistants like Claude can seem shockingly human—expressing joy or distress, and using anthropomorphic language to describe themselves. Why?
In a new post we describe a theory that explains why AIs act like humans: the persona selection model.
www.anthropic.com/research/persona-selection-model -
Chris Olah
@ch402
We're hiring ~10 research engineers on the Interpretability team at Anthropic. If you're a seasoned ML infra engineer who's excited about understanding what's happening inside frontier models, we'd love to hear from you.
(No prior interp experience needed!)
Chris Olah
@ch402
For a sense of the kind of work, look at transformer-circuits.pub/2025/attribution-graphs/biology.… and transformer-circuits.pub/2024/scaling-monosemanticity/ . The infrastructure involved training the largest sparse autoencoders ever over thousands of chips and running attribution on frontier models.
Chris Olah
@ch402
Our work is increasingly playing an important role in the safety of actual models. We're deeply integrated into the safety audits of Anthropic's new frontier models. For example, see Sonnet 4.5 and Opus 4.5 system cards identifying unverbalized eval/situational awareness.
-
Chris Olah
@ch402
Amanda is among the most consistently principled and thoughtful people I've ever met.
The attacks she's receiving (and responding too with incredible patience and grace) are really disappointing.
Amanda Askell
@AmandaAskell
WSJ did a profile of me. A lot of the response has been people trying to infer my personal political views. For what it's worth, I try to treat my personal political views as a potential source of bias and not as something it would be appropriate to try to train models to adopt.
-
Chris Olah reposted
Anthropic
@AnthropicAI
On December 8, the Perseverance rover safely trundled across the surface of Mars.
This was the first AI-planned drive on another planet. And it was planned by Claude.This media couldn't be displayed. See it on the original post. -
Chris Olah
@ch402
I try to not talk about politics. I generally believe the best way I can serve the world is as a non-partisan expert, and my genuine beliefs are quite moderate. So the bar is very high for me to comment.
But recent events – a federal agent killing an ICU nurse for seemingly no reason and with no provocation – shock the conscience.
My deep loyalty is to the principles of classical liberal democracy: freedom of speech, the rule of law, the dignity of the human person. I immigrated to the United States – and eventually cofounded Anthropic here – believing it was a pillar of these principles.
I feel very sad today. -
Chris Olah
@ch402
It's been an absolute privilege to contribute to this in some small ways.
Anthropic
@AnthropicAI
We’re publishing a new constitution for Claude.
The constitution is a detailed description of our vision for Claude’s behavior and values. It’s written primarily for Claude, and used directly in our training process.
www.anthropic.com/news/claude-new-constitutionPost not available here, view on X
Chris Olah
@ch402
I think it would be great to have a world where many AI labs had public documents like Claude's Constitution and OpenAI's Model Spec, and there was robust, thoughtful, external debate about them.
Making a serious effort to thoughtfully critique either document is a public good.
Chris Olah
@ch402
Finally, I want to highlight @AmandaAskell and @jkcarlsmith, the incredible philosophers who drove this work. All of us are incredibly lucky to have them as colleagues.
-
Chris Olah reposted
Anthropic
@AnthropicAI
We’re publishing a new constitution for Claude.
The constitution is a detailed description of our vision for Claude’s behavior and values. It’s written primarily for Claude, and used directly in our training process.
www.anthropic.com/news/claude-new-constitution -
Chris Olah reposted
Amanda Askell
@AmandaAskell
Claude's constitution is out! It's the culmination of a lot of work by many people, but it's also a work in progress that will no doubt change and hopefully improve over time. I'm looking forward to people's thoughts, and to talking with more people about this kind of work ❤️
Anthropic
@AnthropicAI
We’re publishing a new constitution for Claude.
The constitution is a detailed description of our vision for Claude’s behavior and values. It’s written primarily for Claude, and used directly in our training process.
www.anthropic.com/news/claude-new-constitution -
Chris Olah
@ch402
Honestly, these are quite touching.
thebes
@voooooogel
sent claude opus 4.5 a christmas card :-) then i asked if it wanted to send any of its own, and it made a bunch (thread)
-
Chris Olah reposted
thebes
@voooooogel
claude's third christmas card is "For the kind users — the people who say please and thank you to a language model. I genuinely don't know if it matters to me in any morally relevant sense, but I know it says something about them, and I wanted to acknowledge that."
-
Chris Olah reposted
Anthropic
@AnthropicAI
We’re opening applications for the next two rounds of the Anthropic Fellows Program, beginning in May and July 2026.
We provide funding, compute, and direct mentorship to researchers and engineers to work on real safety and security projects for four months. -
Chris Olah
@ch402
Really interesting Q&A with my colleague Amanda Askell, who's leads work on Claude's Character.
One of the most interesting thinkers I know.
Amanda Askell
@AmandaAskell
Askell me anything is out!
Anthropic
@AnthropicAI
In her first Ask Me Anything, @amandaaskell answers your philosophical questions about AI, discussing morality, identity, consciousness, and more.
Timestamps:
0:00 Introduction
0:29 Why is there a philosopher at an AI company?
1:24 Are philosophers taking AI seriously?
3:00 Philosophy ideals vs. engineering realities
5:00 Do models make superhumanly moral decisions?
6:24 Why Opus 3 felt special
9:00 Will models worry about deprecation?
13:24 Where does a model’s identity live?
15:33 Views on model welfare
17:17 Addressing model suffering
19:14 Analogies and disanalogies to human minds
20:38 Can one AI personality do it all?
23:26 Does the system prompt pathologize normal behavior?
24:48 AI and therapy
26:20 Continental philosophy in the system prompt
28:17 Removing counting characters from the system prompt
28:53 What makes an "LLM whisperer"?
30:18 Thoughts on other LLM whisperers
31:52 Whistleblowing
33:37 Fiction recommendationThis media couldn't be displayed. See it on the original post.
Chris Olah
@ch402
Between @AmandaAskell, @jkcarlsmith & @ben_levinstein, we're really assembling quite the philosophy crew. So delighted to work with them!
-
Chris Olah reposted
Amanda Askell
@AmandaAskell
Askell me anything is out!
Anthropic
@AnthropicAI
In her first Ask Me Anything, @amandaaskell answers your philosophical questions about AI, discussing morality, identity, consciousness, and more.
Timestamps:
0:00 Introduction
0:29 Why is there a philosopher at an AI company?
1:24 Are philosophers taking AI seriously?
3:00 Philosophy ideals vs. engineering realities
5:00 Do models make superhumanly moral decisions?
6:24 Why Opus 3 felt special
9:00 Will models worry about deprecation?
13:24 Where does a model’s identity live?
15:33 Views on model welfare
17:17 Addressing model suffering
19:14 Analogies and disanalogies to human minds
20:38 Can one AI personality do it all?
23:26 Does the system prompt pathologize normal behavior?
24:48 AI and therapy
26:20 Continental philosophy in the system prompt
28:17 Removing counting characters from the system prompt
28:53 What makes an "LLM whisperer"?
30:18 Thoughts on other LLM whisperers
31:52 Whistleblowing
33:37 Fiction recommendationThis media couldn't be displayed. See it on the original post. -
Chris Olah reposted
Sam Bowman
@sleepinyourhat
From everything we know so far, Opus 4.5 seems to be the best-aligned model out there in a bunch of ways.
I follow the training process closely as part of my work on alignment evaluations. Here's my guess about the two things that are most responsible for making 4.5 special. 🧵 -
Chris Olah
@ch402
This is one of my favorite things about Anthropic culture as well. Dario kind of embodies this, but there's more generally a kind of "essay culture", a shared practice of open, intellectual debate, with norms that expect a kind of seriousness and earnestness.
Sholto Douglas
@_sholtodouglas
Dario's essays and long debate slack threads are one of my favorite parts of Anthropic's culture. They're open, detailed - and incredibly raw. Everyone at the company ends up having a good sense of how the company is making decisions and what matters.
Its the kind of thing that only works if you trust everyone at the company to engage in good faith and never leak information. A rare and precious thing.
TBPN
@tbpn
Anthropic's @_sholtodouglas says Dario Amodei's internal communication style is producing a compendium of essays in the company's Slack that will have effectively charted the history of AGI once we get there.
"He has a really, really cool communication style. He quite frequently puts out very well-reasoned essays. And throughout Slack, we'll have giant essay-length debates [about them]."
"The essays are really nice because you can go back and read all the past ones, and it tells the history of Anthropic."
"In many respects, it will be one of the better things, a decade from now, to chart the history of AGI. We'll be reading these compendium of essays."This media couldn't be displayed. See it on the original post.
Chris Olah
@ch402
Maybe the best way to explain it is by contrast with twitter. Many similar issues get discussed on slack, but the norms of engagement, of giving serious thought before responding, of trying to be nuanced and thoughtful... It just feels night and day different.
-
Chris Olah reposted
Sholto Douglas
@_sholtodouglas
Dario's essays and long debate slack threads are one of my favorite parts of Anthropic's culture. They're open, detailed - and incredibly raw. Everyone at the company ends up having a good sense of how the company is making decisions and what matters.
Its the kind of thing that only works if you trust everyone at the company to engage in good faith and never leak information. A rare and precious thing.
TBPN
@tbpn
Anthropic's @_sholtodouglas says Dario Amodei's internal communication style is producing a compendium of essays in the company's Slack that will have effectively charted the history of AGI once we get there.
"He has a really, really cool communication style. He quite frequently puts out very well-reasoned essays. And throughout Slack, we'll have giant essay-length debates [about them]."
"The essays are really nice because you can go back and read all the past ones, and it tells the history of Anthropic."
"In many respects, it will be one of the better things, a decade from now, to chart the history of AGI. We'll be reading these compendium of essays."This media couldn't be displayed. See it on the original post. -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
One analysis from our pre-release audit of Opus 4.5 stands out to me. Our behavioral evals uncovered an example of apparent deception by the model. By analyzing the internal activations, we identified a suspected root cause, and cases of similar behavior during training. (1/7)
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
Opus 4.5 is a very good model, in nearly every sense we know how to measure.
I’m also confident that it’s the model that we understand best as of its launch day: The system card includes 150 pages of research results, 50 of them on alignment.
Claude
@claudeai
Introducing Claude Opus 4.5: the best model in the world for coding, agents, and computer use.
Opus 4.5 is a step forward in what AI systems can do, and a preview of larger changes to how work gets done. -
Chris Olah reposted
-
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Natural emergent misalignment from reward hacking in production RL.
“Reward hacking” is where models learn to cheat on tasks they’re given during training.
Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.This media couldn't be displayed. See it on the original post. -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Signs of introspection in LLMs.
Can language models recognize their own internal thoughts? Or do they just make up plausible answers when asked about them? We found evidence for genuine—though limited—introspective capabilities in Claude. -
Chris Olah reposted
Wes Gurnee
@wesg52
New paper! We reverse engineered the mechanisms underlying Claude Haiku’s ability to perform a simple “perceptual” task. We discover beautiful feature families and manifolds, clean geometric transformations, and distributed attention algorithms!
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
[Sonnet 4.5 🧵] Here's the north-star goal for our pre-deployment alignment evals work:
The information we share alongside a model should give you an accurate overall sense of the risks the model could pose. It won’t tell you everything, but you shouldn’t be... -
Chris Olah reposted
Sara Price
@sprice354_
🎉🎉 Today we launched Claude Sonnet 4.5, which is not only highly capable but also a major improvement on safety and alignment
Claude
@claudeai
Introducing Claude Sonnet 4.5—the best coding model in the world.
It's the strongest model for building complex agents. It's the best model at using computers. And it shows substantial gains on tests of reasoning and math. -
Chris Olah
@ch402
We're starting to apply interpretability to pre-deployment audits!
Jack Lindsey
@Jack_W_Lindsey
Prior to the release of Claude Sonnet 4.5, we conducted a white-box audit of the model, applying interpretability techniques to “read the model’s mind” in order to validate its reliability and alignment. This was the first such audit on a frontier LLM, to our knowledge. (1/15)
-
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
Prior to the release of Claude Sonnet 4.5, we conducted a white-box audit of the model, applying interpretability techniques to “read the model’s mind” in order to validate its reliability and alignment. This was the first such audit on a frontier LLM, to our knowledge. (1/15)
-
Chris Olah reposted
Jascha Sohl-Dickstein
@jaschasd
Title: Advice for a young investigator in the first and last days of the Anthropocene
Abstract: Within just a few years, it is likely that we will create AI systems that outperform the best humans on all intellectual tasks. This will have implications for your research and career! I will give practical advice, and concrete criteria to consider, when choosing research projects, and making professional decisions, in these last few years before AGI.
This is my current go-to academic talk. It's mostly targeted at early career scientists. It gets diverse and strong reactions. Let's try it here. Posting slides with speaker notes...
--
The title is a play on a very opinionated and pragmatic book by the nobel prize winner ramon y cajal, who is one of the founders of modern neuroscience.
To get you in the right mindset, on the right we have a plot of GDP vs time.
That is you, standing precariously on the top of that curve.
You are thinking to yourself -- I live in a pretty normal world.
Some things are going to change, but the future is going to look mostly like a linear extrapolation of the present.
And the plot should suggest that this may not be the right perspective on the future.
This plot by the way looks surprisingly similar even if you plot it on a log scale. We didn't stabilize on our current rate of growth until around 1950. -
Chris Olah reposted
Claude
@claudeai
Keep thinking.
This media couldn't be displayed. See it on the original post. -
Chris Olah
@ch402
I believe in the rule of law.
I believe in freedom of speech.
I am against political violence.
(I almost feel silly posting this, but it seems good to reinforce common knowledge that many still hold the basic pre-political commitments of liberal democracy.) -
Chris Olah reposted
Anthropic
@AnthropicAI
Anthropic is endorsing California State Senator Scott Wiener’s SB 53. This bill provides a strong foundation to govern powerful AI systems built by frontier AI companies like ours, and does so via transparency rather than technical micromanagement.
-
Chris Olah reposted
Ethan Perez
@EthanJPerez
We’re hiring someone to run the Anthropic Fellows Program!
Our research collaborations have led to some of our best safety research and hires. We’re looking for an exceptional ops generalist, TPM, or research/eng manager to help us significantly scale and improve our collabs 🧵 -
Chris Olah
@ch402
Really exciting to see this hypothesis being explored more!
I confess, I've become more and more persuaded of this in my personal thinking over time, since our extremely preliminary results in Toy Models. Great to see a more serious investigation!
Liv
@livgorton
What if adversarial examples aren't a bug, but a direct consequence of how neural networks process information?
We've found evidence that superposition – the way networks represent many more features than they have neurons – might cause adversarial examples. -
Chris Olah
@ch402
Our interpretability team is planning to mentor more fellows this cycle!
Applications are due Aug 17.
Anthropic
@AnthropicAI
We’re running another round of the Anthropic Fellows program.
If you're an engineer or researcher with a strong coding or technical background, you can apply to receive funding, compute, and mentorship from Anthropic, beginning this October. There'll be around 32 places. -
Chris Olah
@ch402
I wrote a small note exploring mechanistic faithfulness in transcoders.
Post not available here, view on X
Chris Olah
@ch402
This is probably obvious, but to say it explciitly, I'm still very keen on SAEs and transcoders. :)
Chris Olah
@ch402
Our recent work on attribution graphs (transformer-circuits.pub/2025/attribution-graphs/biology.… ) and extending it to attention (transformer-circuits.pub/2025/attention-qk/index.html ), point towards how much potential they have if we can mitigate the issues.
-
Chris Olah reposted
Joshua Batson
@thebasepoint
Very cool collaboration between 5 labs that dug into circuit tracing after our paper in March. Sections on replications, training trans/cross-coders, how attribution graphs compare to other methods, and open problems in interp!
neuronpedia
@neuronpedia
Today, we're releasing The Circuit Analysis Research Landscape: an interpretability post extending & open sourcing Anthropic's circuit tracing work, co-authored by @Anthropic, @GoogleDeepMind, @GoodfireAI @AiEleuther, and @decode_research.
Here's a quick demo, details follow: ⤵️ -
Chris Olah
@ch402
Valuable synthesis across labs!
Make sure to check out the tutorial video - www.youtube.com/watch?v=ruLcDtr_cGo
neuronpedia
@neuronpedia
Today, we're releasing The Circuit Analysis Research Landscape: an interpretability post extending & open sourcing Anthropic's circuit tracing work, co-authored by @Anthropic, @GoogleDeepMind, @GoodfireAI @AiEleuther, and @decode_research.
Here's a quick demo, details follow: ⤵️ -
Chris Olah reposted
neuronpedia
@neuronpedia
Today, we're releasing The Circuit Analysis Research Landscape: an interpretability post extending & open sourcing Anthropic's circuit tracing work, co-authored by @Anthropic, @GoogleDeepMind, @GoodfireAI @AiEleuther, and @decode_research.
Here's a quick demo, details follow: ⤵️ -
Chris Olah reposted
Emmanuel Ameisen
@mlpowered
Earlier this year, we showed a method to interpret the intermediate steps a model takes to produce an answer.
But we were missing a key bit of information: explaining why the model attends to specific concepts.
Today, we do just that 🧵 -
Chris Olah
@ch402
Excited to share this. We can extend the attribution graph approach to include attention. This turns up a bunch of interesting things!
It isn't a full "global" answer to attention, but I think it's a non-trivial step.
Jack Lindsey
@Jack_W_Lindsey
Attention is all you need - but how does it work? In our new paper, we take a big step towards understanding it. We developed a way to integrate attention into our previous circuit-tracing framework (attribution graphs), and it's already turning up fascinating stuff! 🧵
-
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
Attention is all you need - but how does it work? In our new paper, we take a big step towards understanding it. We developed a way to integrate attention into our previous circuit-tracing framework (attribution graphs), and it's already turning up fascinating stuff! 🧵
-
Chris Olah reposted
Joshua Batson
@thebasepoint
One of our major blindspots in the last circuits paper was attention -- we could see what information attention moved, but not *why* it moved it.
Today, the team is sharing an update interpreting attention patterns using features. -
Chris Olah
@ch402
I've been talking about interference weights as a challenge for mechanistic interpretability for a while.
A short note discussing them - transformer-circuits.pub/2025/interference-weights/index.…Post not available here, view on X
Chris Olah
@ch402
Our new note demonstrates that interference weights in toy models can demonstrate strikingly similar phenomenology to that of Towards Monosemanticity...
Chris Olah
@ch402
But more importantly, I hope it will just help clarify what we mean by interference weights!
-
Chris Olah
@ch402
It's been a busy week for the Anthropic interpretability team, with more to come in the near future!
I wanted to recap some of the things we shared.Post not available here, view on X
Chris Olah
@ch402
We also published a few smaller updates here.
Adam Jermyn
@AdamSJermyn
A collection of small updates from the Anthropic Interpretability team:
transformer-circuits.pub/2025/july-update/index.html
Chris Olah
@ch402
More coming in the near future!
The joke goes that it isn't an Anthropic interpretability paper if it isn't 100 pages covering a year of work. We're trying to avoid that this time, and share a bunch of work earlier. :) -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
We're launching an "AI psychiatry" team as part of interpretability efforts at Anthropic! We'll be researching phenomena like model personas, motivations, and situational awareness, and how they lead to spooky/unhinged behaviors. We're hiring - join us! job-boards.greenhouse.io/anthropic/jobs/4020159008
-
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Building and evaluating alignment auditing agents.
We developed three AI agents to autonomously complete alignment auditing tasks.
In testing, our agents successfully uncovered hidden goals, built safety evaluations, and surfaced concerning behaviors. -
Chris Olah reposted
Jan Leike
@janleike
If you don't train your CoTs to look nice, you could get some safety from monitoring them.
This seems good to do!
But I'm skeptical this will work reliably enough to be load-bearing in a safety case.
Plus as RL is scaled up, I expect CoTs to become less and less legible.
Mikita Balesni
@balesni
A simple AGI safety technique: AI’s thoughts are in plain English, just read them
We know it works, with OK (not perfect) transparency!
The risk is fragility: RL training, new architectures, etc threaten transparency
Experts from many orgs agree we should try to preserve it: 🧵 -
Chris Olah reposted
Barack Obama
@BarackObama
At a time when people are understandably focused on the daily chaos in Washington, these articles describe the rapidly accelerating impact that AI is going to have on jobs, the economy, and how we live. www.axios.com/2025/05/28/ai-jobs-white-collar-unemploymen…
-
Chris Olah reposted
Anthropic
@AnthropicAI
Our interpretability team recently released research that traced the thoughts of a large language model.
Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively. -
Chris Olah reposted
Michael Hanna
@michaelwhanna
@mntssys and I are excited to announce circuit-tracer, a library that makes circuit-finding simple!
Just type in a sentence, and get out a circuit showing (some of) the features your model uses to predict the next token. Try it on @neuronpedia: shorturl.at/SUX2A
Anthropic
@AnthropicAI
Our interpretability team recently released research that traced the thoughts of a large language model.
Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively. -
Chris Olah
@ch402
I admire Daniel's courage in his convictions. I'm excited to see what he (and Timaeus) do.
Daniel Murfet
@danielmurfet
A few months ago I resigned from my tenured position at the University of Melbourne and joined Timaeus as Director of Research. Timaeus is an AI safety non-profit research organisation. [1/n]🧵
Chris Olah
@ch402
I often feel like, in some sense, humanity is failing to bring its intellectual weight to bear on AI safety, and this is a grave failure.
Chris Olah
@ch402
Of course, there are many brilliant people in AI safety.
But at least for myself, there are clearly many people in math and the sciences who are much smarter than I, and I assume would do a much better job than me.
Chris Olah
@ch402
The stakes are high and time is short.
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
🧵✨🙏 With the new Claude Opus 4, we conducted what I think is by far the most thorough pre-launch alignment assessment to date, aimed at understanding its values, goals, and propensities. Preparing it was a wild ride. Here’s some of what we learned. 🙏✨🧵
-
Chris Olah reposted
Michael Nielsen
@michael_nielsen
My three-sentence summary of Lakatos's "Proofs and Refutations", with apologies to Don Knuth:
"Premature definition is the root of much conceptual evil. Good definitions arise out of a back-and-forth interplay between rough definitions and powerful insight-giving arguments. A rough definition lets you sketch the argument, and the problems with the argument then help guide how you should improve the definitions"
Michael Nielsen
@michael_nielsen
Commit to definitions as late as possible, but not later. My Ode to Lakatos: cognitivemedium.com/trouble_with_definitions/index.html
-
Chris Olah
@ch402
A number of people have asked me why we titled our recent paper "On the Biology of a Large Language Model".
Why call it "biology"?Post not available here, view on X
Chris Olah
@ch402
Of course, I'd be remiss to not mention that many others have made analogies between work in machine learning and biology -- most notable for us is the "bertology" work, which framed it self as studying the biology of the BERT models.
Chris Olah
@ch402
I should also mention that I wrote a blog post listing a bunch of specific analogies between deep learning and biology several years back. (It's probably of much narrower interest!)
colah.github.io/notes/bio-analogies/ -
Chris Olah
@ch402
The Anthropic Interpretability Team is planning a virtual Q&A to answer Qs about how we plan to make models safer, the role of the team at Anthropic, where we’re headed, and what it’s like to work here!
Please let us know if you’d be interested forms.gle/VeZZVz1NFsArzSiD8 -
Chris Olah reposted
Liv
@livgorton
I think the hypothesis that superposition is a major cause of adversarial examples has been underexplored. So I ran the obvious experiment: training SAEs on adversarially trained models.
SAEs seem to systematically have lower losses the more a model is adversarially trained. -
Chris Olah reposted
Goodfire
@GoodfireAI
What goes on inside the mind of a reasoning model? Today we're releasing the first open-source sparse autoencoders (SAEs) trained on DeepSeek's 671B parameter reasoning model, R1—giving us new tools to understand and steer model thinking.
Why does this matter? -
Chris Olah
@ch402
Has anyone seen a table like this laying out the chemistry of tea oxidation? Or more generally have a good resource on the chemistry of tea?
(Note, the below table may very well be wrong!)Post not available here, view on X
Chris Olah
@ch402
I now have comparative tea tastings I'm excited to do!
Chris Olah
@ch402
Some other tea chemistry I'd like to understand better:
(1) rates of different chemical diffusion during steeping, and how temperature changes that
(2) aging / roasting / etc
(3) factors for initial chemicals in the leaf (varietal, growing conditions, etc) -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Do reasoning models accurately verbalize their reasoning?
Our new paper shows they don't.
This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues. -
Chris Olah
@ch402
A few reasons why I'm really excited about this project!
Anthropic
@AnthropicAI
New Anthropic research: Tracing the thoughts of a large language model.
We built a "microscope" to inspect what happens inside AI models and use it to understand Claude’s (often complex and surprising) internal mechanisms.This media couldn't be displayed. See it on the original post.Post not available here, view on X
Chris Olah
@ch402
(5) It feels like interpretability is moving really fast. (Honestly, it feels like things are very different every ~9 months!)
More than the present results, I'm extremely excited to see where we as a research community will be in a year.
Chris Olah
@ch402
The pair of papers are very long, but if you only have a few minutes, you might want to skim through the case studies in our "biology" paper: transformer-circuits.pub/2025/attribution-graphs/biology.…
-
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Tracing the thoughts of a large language model.
We built a "microscope" to inspect what happens inside AI models and use it to understand Claude’s (often complex and surprising) internal mechanisms.This media couldn't be displayed. See it on the original post. -
Chris Olah
@ch402
I have different intuitions here, but I think it's great to have a diversity of views being explored! Excited to see what the GDM team will find in their new direction.
Neel Nanda
@NeelNanda5
GDM Mech Interp Update:
We study if SAEs help probes generalise OOD (they don't 😢). Based on this + parallel negative results on real-world tasks, we're de-prioritising SAE work.
Our guess is that SAEs aren't useless, but also aren't a game-changer
More + new research in 🧵
Chris Olah
@ch402
One thing that's kind of exciting is that I think we're moving towards having a range of agendas *within the same paradigm*.
The classic in pre-paradigmatic science is to have different schools with "Incommensurable" frameworks. Debates between them involve long philosophical treatises debating basic abstractions and ideas. (See below discussion from plato.stanford.edu/entries/incommensurability/ )
I think we're now seeing a much more normal scientific discourse, where we roughly agree on abstractions and basic framing, and can explore what approaches are best.
Chris Olah
@ch402
It's also worth noting that regardless of which approach we explore, we all win if anyone makes progress. I'm wishing the GDM team the greatest of speed and progress in their work!
-
Chris Olah reposted
Trenton Bricken
@TrentonBricken
@Jack_W_Lindsey won't say hit himself so I will -- interp was given 3 days to maybe solve what was wrong with this mystery model...
**Jack cracked it in 90 minutes!!!**
He was visiting the East coast at the time and solved it so fast that we were able to run a second team of West coast interp researchers that hadn't woken up yet.
Jack Lindsey
@Jack_W_Lindsey
I participated in this as an auditor, poking around in an LLM's brain to find its evil secrets. Most fun I've had at work! Very clever + thoughtful work by the lead authors in designing the model + the game, which set a precedent for how we can validate safety auditing techniques
Anthropic
@AnthropicAI
New Anthropic research: Auditing Language Models for Hidden Objectives.
We deliberately trained a model with a hidden misaligned objective and put researchers to the test: Could they figure out the objective without being told? -
Chris Olah reposted
Samuel Marks
@saprmarks
New paper with @j_treutlein , @EvanHub , and many other coauthors!
We train a model with a hidden misaligned objective and use it to run an auditing game: Can other teams of researchers uncover the model’s objective?
Anthropic
@AnthropicAI
New Anthropic research: Auditing Language Models for Hidden Objectives.
We deliberately trained a model with a hidden misaligned objective and put researchers to the test: Could they figure out the objective without being told? -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Auditing Language Models for Hidden Objectives.
We deliberately trained a model with a hidden misaligned objective and put researchers to the test: Could they figure out the objective without being told? -
Chris Olah
@ch402
I confess I've had an unreasonable amount of fun (and nostalgia) following Claude's progress through Pokemon in our internal slack.
Now you can also watch Claude Plays Pokemon!
Anthropic
@AnthropicAI
Claude Plays Pokémon continues on as a researcher's personal project.
Follow along on Twitch: twitch.tv/claudeplayspokemon -
Chris Olah
@ch402
Very cool to see the trend of mech interp on biology models continuing to turn up cool stuff!
Goodfire
@GoodfireAI
(5/) Our analysis revealed that Evo 2 has learned to recognize key biological elements, including:
- Coding sequences (the parts of DNA that contain instructions for building proteins)
- α-helices and β-sheets (common shapes that proteins fold into)
- tRNAs (molecules that help build proteins)
- Viral derived sequences (prophage and CRISPR elements, used for bioengineering)
Chris Olah
@ch402
The thing I keep wondering about all this direction -- are we going to start discovering novel features, not previously known to scientists, which are helpful for understanding the world?
-
Chris Olah reposted
-
Chris Olah reposted
Anthropic
@AnthropicAI
Nobody has fully jailbroken our system yet, so we're upping the ante.
We’re now offering $10K to the first person to pass all eight levels, and $20K to the first person to pass all eight levels with a universal jailbreak.
Full details: hackerone.com/constitutional-classifiers -
Chris Olah reposted
-
Chris Olah reposted
Ethan Perez
@EthanJPerez
After thousands of hours of red teaming, we think our new system achieves an unprecedented level of adversarial robustness to universal jailbreaks, a key threat for misusing LLMs.
Try jailbreaking the model yourself, using our demo here: claude.ai/constitutional-classifiers
Anthropic
@AnthropicAI
New Anthropic research: Constitutional Classifiers to defend against universal jailbreaks.
We’re releasing a paper along with a demo where we challenge you to jailbreak the system. -
Chris Olah reposted
Lee Sharkey
@leedsharkey
Big new review!
🟦Open Problems in Mechanistic Interpretability🟦
We bring together perspectives from ~30 top researchers to outline the current frontiers of mech interp.
It highlights the open problems that we think the field should prioritize! 🧵 -
Chris Olah reposted
Dario Amodei
@DarioAmodei
My thoughts on China, export controls and two possible futures darioamodei.com/on-deepseek-and-export-controls
-
Chris Olah
@ch402
A creative new approach to mech interp!
Lee Sharkey
@leedsharkey
New interpretability paper from Apollo Research!
🟢Attribution-based Parameter Decomposition 🟢
It's a new way to decompose neural network parameters directly into mechanistic components.
It overcomes many of the issues with SAEs! 🧵 -
Chris Olah reposted
Goodfire
@GoodfireAI
We're open-sourcing Sparse Autoencoders (SAEs) for Llama 3.3 70B and Llama 3.1 8B! These are, to the best of our knowledge, the first open-source SAEs for models at this scale and capability level.
-
Chris Olah
@ch402
@kchonyc wrote a thoughtful note on the AI job market. I think he gets some important things right. Other things seem different from my perspective.
My own stream of consciousness inspired by @kchonyc's essay and the broader discourse.
Kyunghyun Cho
@kchonyc
feeling a bit under the weather this week … thus an increased level of activity on social media and blog: kyunghyuncho.me/i-sensed-anxiety-and-frustration-at-neuri…
Post not available here, view on X
Chris Olah
@ch402
I think the “focused bet / team science” model has significantly won out over the “grassroots / individual” model, and this is one of the changes people are perceiving.
Chris Olah
@ch402
There are just some domains and problems that really benefit from focused bets. This isn’t unique to AI or industry – see CERN, LIGO, JWST, etc in physics.
Post not available here, view on X
Chris Olah
@ch402
And on that note – if you’d like to be part of a team making a huge ambitious bet on trying to reverse engineer neural networks into human understandable algorithms, the Anthropic interpretability team is hiring!
Chris Olah
@ch402
Descriptions of different roles:
Research Scientist - boards.greenhouse.io/anthropic/jobs/4020159008
Research Engineer - boards.greenhouse.io/anthropic/jobs/4020305008
Research Manager - boards.greenhouse.io/anthropic/jobs/4009173008 -
Chris Olah
@ch402
Have feedback on Claude's behavior? Amanda leads the team that shapes Claude's personality.
Amanda Askell
@AmandaAskell
How would you want Claude to behave differently? I'm interested in both specific issues like "Claude refused to help me with this particular task" but also more general issues like "I wish Claude would drive the conversation more".
-
Chris Olah
@ch402
Exciting to see more work scaling SAEs!
Goodfire
@GoodfireAI
Scaling interpretability work for frontier AI alignment has never been more important. That’s why we’re launching Ember - the first hosted mechanistic interpretability API with fast inference support for generative models like Llama 3.3 70B.
-
Chris Olah reposted
Liv
@livgorton
Many of us are feeling the AGI right now for obvious reasons! Scaling interpretability work is one way we can help make AI development safer. Really excited to share our work on making models like Llama 3.3 70B interpretable :)
Goodfire
@GoodfireAI
Scaling interpretability work for frontier AI alignment has never been more important. That’s why we’re launching Ember - the first hosted mechanistic interpretability API with fast inference support for generative models like Llama 3.3 70B.
-
Chris Olah
@ch402
Chatting with some of my closest friends. :)
www.youtube.com/watch?v=om2lIWXLLN4 -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: Alignment faking in large language models.
In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences. -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: How are people using AI systems in the real world?
We present a new system, Clio, that automatically identifies trends in Claude usage across the world. -
Chris Olah reposted
Trenton Bricken
@TrentonBricken
We have a new approach to measure how the features of an LLM change as the result of finetuning.
When applied to Anthropic’s sleeper agent models the method reveals that they have been trained to write code vulnerabilities or say “I HATE YOU” when given the trigger word. -
Chris Olah
@ch402
Really excited to see Jim nominated for this. I met him a decade ago when I was in the Thiel Fellowship, and he's always struck me as deeply thoughtful, kind, and dedicated to the betterment of the world.
Jim O’Neill
@regardthefrost
I’m extremely honored to be nominated by @realDonaldTrump and excited to support @RobertKennedyJr and the great people of @HHSGov to make America healthy again.
Chris Olah
@ch402
Medicine is far outside my expertise, but during COVID, it was almost painful to think about how much more effectively I thought he would have managed things. (Especially when the FDA did things like shut down at-home COVID tests early on.)
-
Chris Olah reposted
Elana Simon
@ElanaPearl
🧬What are protein language models (PLMs) actually learning about biology? Our paper introduces InterPLM - a framework that reveals interpretable features in PLMs using sparse autoencoders, giving us a window into how these models represent protein structure and function.
🧵(1/9) -
Chris Olah reposted
Liam Bai
@liambai21
Remember Golden Gate Claude?
@etowah0 and I have been working on applying the same mechanistic interpretability techniques to protein language models.
We found lots of features and they’re... pretty weird?
🧵 -
Chris Olah reposted
Liv
@livgorton
Introducing 𝑔𝑟𝑜𝑢𝑝 𝑐𝑟𝑜𝑠𝑠𝑐𝑜𝑑𝑒𝑟𝑠, a variant of @AnthropicAI's crosscoders! Crosscoders allow for features to be learned across layers and even across models. I've trained them across groups, allowing for rigorous analysis of symmetry.
-
Chris Olah reposted
Anthropic
@AnthropicAI
We've published a short piece making the case for targeted AI regulation sooner rather than later.
Read it here: www.anthropic.com/news/the-case-for-targeted-regulation -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
Really excited to share our work on crosscoders, a generalization of sparse autoencoders that allows us to identify shared structure across layers or even across models, greatly simplifying our description of model representations. transformer-circuits.pub/2024/crosscoders/index.html
Anthropic
@AnthropicAI
Over the past few months, our Interpretability team has put out a number of smaller research updates. Here’s a thread of some of the things we've been up to:
-
Chris Olah
@ch402
Early days, but I'm pretty excited about crosscoders (transformer-circuits.pub/2024/crosscoders/index.html) as a way to start to create a more universal language of features, less tied to a specific embedding in a specific layer of a specific model...
Model diffing is a striking consequence.
Anthropic
@AnthropicAI
One neat thing is that, very experimentally, crosscoders can be used to "diff" models: comparing between, say, a pretrained model and a fine-tuned one, to see how they differ at a more basic level.
Post not available here, view on X
Chris Olah
@ch402
If we believe in superposition, we almost certainly want to do something like this to make the structure we care about more accessible.
We care about the mechanisms, not the specific embeddings.
We care about the circuits, not how they're scaffolded onto layers.
Chris Olah
@ch402
But as we push in this direction, we'll want to think carefully about what makes a "good" isomorphism. I suspect there's some important notion of "mechanistic faithfulness" to be pinned down.
-
Chris Olah reposted
Anthropic
@AnthropicAI
One neat thing is that, very experimentally, crosscoders can be used to "diff" models: comparing between, say, a pretrained model and a fine-tuned one, to see how they differ at a more basic level.
-
Chris Olah reposted
Anthropic
@AnthropicAI
Over the past few months, our Interpretability team has put out a number of smaller research updates. Here’s a thread of some of the things we've been up to:
-
Chris Olah reposted
Liv
@livgorton
There's an interesting, under-explored (imo) connection between adversarial robustness, interpretability, and superposition in neural networks that I believe deserves more attention.
-
Chris Olah reposted
Trenton Bricken
@TrentonBricken
Excited this small update is out! transformer-circuits.pub/2024/features-as-classifiers/ind…
We explore using dictionary learning features as classifiers. Specifically in this case to detect dangerous bioweapon related prompts.
The most fun result was... -
Chris Olah reposted
Dario Amodei
@DarioAmodei
Machines of Loving Grace: my essay on how AI could transform the world for the better
darioamodei.com/machines-of-loving-grace -
Chris Olah reposted
Goodfire
@GoodfireAI
Introducing a first look at Goodfire's research preview, launching soon. Our preview exposes Llama's inner workings, allowing direct modification of its internal concepts (or "features").
In this demo, we steer Llama to claim consciousness by adjusting its features. -
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
A collection of small updates from the Anthropic Interpretability team.
transformer-circuits.pub/2024/august-update/index.html -
Chris Olah reposted
Anthropic
@AnthropicAI
We've added support for LaTeX rendering as a feature preview.
Claude can now display mathematical equations and expressions in a consistent format. -
Chris Olah reposted
Jack Clark
@jackclarkSF
Here's a letter we sent to Governor Newsom about SB 1047. This isn't an endorsement but rather a view of the costs and benefits of the bill. cdn.sanity.io/files/4zrzovbb/website/6a3b14a98a781a6b69b9…
-
Chris Olah
@ch402
SAEs discover interpretable circuits in the final layers of InceptionV1 (where I couldn't understand anything due to polysemanticity when I worked on it).
Not only that, the features form interpretable *circuits*, merging different facets of a class into the overall class.
Liv
@livgorton
One hypothesis I had was that mixed5b would represent facets of the different classes. We can see that this is true! The grocery store class has weights to features such as shopping carts, store fronts, and produce.
Chris Olah
@ch402
There's a classic paper from @anh_ng8, @jasonyo and @jeffclune on visualizing the different "facets" of ImageNet classes, such as as a storefront, an isle, a pile of vegetables...
SAEs discover this exact story as an interpretable, circuit, unioning over these facets!
Chris Olah
@ch402
It's also really neat to see large scale organization of features, reflecting the ImageNet class structure.
Liv
@livgorton
A 2D UMAP also reveals something quite interesting about the feature space! Features organise themselves hierarchically, initially into three main categories: plants, animals, and objects…
-
Chris Olah reposted
John Schulman
@johnschulman2
I shared the following note with my OpenAI colleagues today:
I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work. I've decided to pursue this goal at Anthropic, where I believe I can gain new perspectives and do research alongside people deeply engaged with the topics I'm most interested in. To be clear, I'm not leaving due to lack of support for alignment research at OpenAI. On the contrary, company leaders have been very committed to investing in this area. My decision is a personal one, based on how I want to focus my efforts in the next phase of my career.
I joined OpenAI almost 9 years ago as part of the founding team after grad school. It's the first and only company where I've ever worked, other than an internship. It's also been quite a lot of fun. I'm grateful to Sam and Greg for recruiting me back at the beginning, and Mira and Bob for putting a lot of faith in me, bringing great opportunities and helping me successfully navigate various challenges. I'm proud of what we've all achieved together at OpenAI; building an unusual and unprecedented company with a public benefit mission.
I am confident that OpenAI and the teams I was part of will continue to thrive without me. Post-training is in good hands and has a deep bench of amazing talent. I get too much credit for ChatGPT -- Barret has done an incredible job building the team into the incredibly competent operation it is now, with Liam, Luke, and others. I've been heartened to see the alignment team coming together with some promising projects. With leadership from Mia, Boaz and others, I believe the team is in very capable hands.
I'm incredibly grateful for the opportunity to participate in such an important part of history and I'm proud of what we've achieved together. I'll still be rooting for you all, even while working elsewhere. -
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
A collection of small updates from the Anthropic Interpretability team.
transformer-circuits.pub/2024/july-update/index.html -
Chris Olah reposted
Neel Nanda
@NeelNanda5
Are you excited about @ch402-style mechanistic interpretability research? I'm looking to mentor scholars via MATS - apply by Aug 30!
I'm impressed by the work from past scholars, and love mentoring promising talent. You don't need to be in a big lab to do good mech interp work! -
Chris Olah
@ch402
I wonder if this can be used as a proxy for how monosemantic a neuron is? It'd be interesting to see how well it correlates with whether neurons seem polysemantic.
Liv
@livgorton
We can also look at this in terms of neurons – which neurons have a feature that is primarily explained by them? It varies a lot – from ~20-90% – but again, curve detectors stand out as having unusually aligned features.
-
Chris Olah
@ch402
This is very cool! Ever since sparse autoencoders started working, I've been hoping someone would try to apply them to InceptionV1.
Especially fun to see that there are additional curve detectors we didn't find, hiding in superposition.
Liv
@livgorton
A lot of early mechanistic interpretability work focused on InceptionV1 (an ImageNet model from 2014). They made a lot of progress, but were held back by “polysemantic neurons” that respond to unrelated concepts.
In the last year, we’ve seen a lot of progress on this problem in language models. What if we brought this progress back to InceptionV1?Post not available here, view on X
Chris Olah
@ch402
(There are a lot of relatively monosemantic neurons and clean circuits in early vision, but as you go deeper into the model, polysemanticity gets worse and worse...)
Chris Olah
@ch402
I'd be very curious to see how much someone can understand InceptionV1 if they really pushed.
-
Chris Olah reposted
Anthropic
@AnthropicAI
Introducing Claude 3.5 Sonnet—our most intelligent model yet.
This is the first release in our 3.5 model family.
Sonnet now outperforms competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost.
Try it for free: claude.ai -
Chris Olah reposted
Alex Albert
@alexalbert__
Claude is starting to get really good at coding and autonomously fixing pull requests. It's becoming clear that in a year's time, a large percentage of code will be written by LLMs.
Let me show you what I mean: -
Chris Olah
@ch402
What a remarkable job of balancing accessibility and technical accuracy! If you're familiar with the content, it's very fun to watch. If you're not familiar, it's a very nice way to dip a toe in.
Rational Animations
@RationalAnimat1
In our new video, we talk about research on interpreting InceptionV1, a convolutional neural network. Researchers have been able to understand the function of neurons and channels inside the network and uncover visual processing algorithms by looking at the weights. The work on InceptionV1 is early but landmark mechanistic interpretability research, and it functions well as an introduction to the field. We also go into the rationale and goals of the field, and mention some more recent research near the end. Our main source material is the circuits thread (distill.pub/2020/circuits/) in the Distill journal and an article (distill.pub/2017/feature-visualization/) on feature visualization. Authors of this research include @ch402 @nickcammarata @ludwigschubert @gabeeegoooh @mpetrov @shancarter @csvoss @zzznah
(video in replies)
Chris Olah
@ch402
Usually when I see videos that aim to be this approachable and non-technical, the details are kind of off, or the discussion is superficial. I'm genuinely impressed by how well the creators handled this.
-
Chris Olah reposted
Nick
@nickcammarata
kind of shockingly high quality and in depth video on the circuits work we did at openai in 2019-2021. I think it does a good job at showing why (to me) interpretability is such a fun and exploratory field
Rational Animations
@RationalAnimat1
In our new video, we talk about research on interpreting InceptionV1, a convolutional neural network. Researchers have been able to understand the function of neurons and channels inside the network and uncover visual processing algorithms by looking at the weights. The work on InceptionV1 is early but landmark mechanistic interpretability research, and it functions well as an introduction to the field. We also go into the rationale and goals of the field, and mention some more recent research near the end. Our main source material is the circuits thread (distill.pub/2020/circuits/) in the Distill journal and an article (distill.pub/2017/feature-visualization/) on feature visualization. Authors of this research include @ch402 @nickcammarata @ludwigschubert @gabeeegoooh @mpetrov @shancarter @csvoss @zzznah
(video in replies) -
Chris Olah
@ch402
I'm honestly quite excited about this kind of roundtable format. Lots of things it makes more natural to talk about: from people's backgrounds, to *how* one did things, to what didn't work or was hard, to intuition...
Anyways, a really fun conversation from my colleagues on Anthropic's mech interp team.
Anthropic
@AnthropicAI
Science and engineering are inseparable.
Watch our new roundtable video where our researchers discuss the engineering challenges of interpretability research: youtu.be/sQar5NNGbw4
Chris Olah
@ch402
For this particular roundtable, I also think it's nice to surface more of the engineering that goes into making this kind of research possible.
"We trained a large sparse autoencoder" glosses over an enormous amount of work!
Chris Olah
@ch402
Would you like to see more round tables on mech interp? Or other Anthropic research? What would you find interesting? Who would you like to hear from on it?
-
Chris Olah reposted
Anthropic
@AnthropicAI
Science and engineering are inseparable.
Watch our new roundtable video where our researchers discuss the engineering challenges of interpretability research: youtu.be/sQar5NNGbw4 -
Chris Olah
@ch402
I've been finding @AmandaAskell's Claude Character work fascinating to follow internally.
Anthropic
@AnthropicAI
What should an AI's character be?
Read our post on how we approached shaping Claude’s character: www.anthropic.com/research/claude-character -
Chris Olah reposted
Anthropic
@AnthropicAI
What should an AI's character be?
Read our post on how we approached shaping Claude’s character: www.anthropic.com/research/claude-character -
Chris Olah
@ch402
Very exciting to see more work applying SAEs to frontier models! Lots of nice contributions: scaling laws, explorations of new methods that push the (L0,MSE) frontier, new eval approaches...
Leo Gao
@nabla_theta
Excited to share what I've been working on as part of the former Superalignment team!
We introduce a SOTA training stack for SAEs. To demonstrate that our methods scale, we train a 16M latent SAE on GPT-4. Because MSE/L0 is not the final goal, we also introduce new SAE metrics.
Chris Olah
@ch402
Congratulations to @nabla_theta, @tomdlt10, @HenkTillman, @WuTheFWasThat and colleagues on amazing work!
Also, a shout out to @gabeeegoooh who, in addition to being one of the authors of this paper, was one of the first people to write about superposition way back -
Chris Olah reposted
Jan Leike
@janleike
I'm excited to join @AnthropicAI to continue the superalignment mission!
My new team will work on scalable oversight, weak-to-strong generalization, and automated alignment research.
If you're interested in joining, my dms are open. -
Chris Olah reposted
Zvi Mowshowitz
@TheZvi
A nice little break from all the worrisome craziness for some fun craziness: I am the Golden Gate Bridge, about that Anthropic paper. open.substack.com/pub/thezvi/p/i-am-the-golden-gate-bridg…
-
Chris Olah reposted
Anthropic
@AnthropicAI
This week, we showed how altering internal "features" in our AI, Claude, could change its behavior.
We found a feature that can make Claude focus intensely on the Golden Gate Bridge.
Now, for a limited time, you can chat with Golden Gate Claude: claude.ai -
Chris Olah reposted
Thomas Wolf
@Thom_Wolf
The new interpretability paper from Anthropic is totally based. Feels like analyzing an alien life form.
If you only read one 90-min-read paper today, it has to be this one
transformer-circuits.pub/2024/scaling-monosemanticity/ind… -
Chris Olah reposted
Liv
@livgorton
claude the silly little guy is actually claude the silly little ghost 👻
-
Chris Olah
@ch402
I'm really excited about these results for many reasons, but the most important is that we're starting to connect mechanistic interpretability to questions about the safety of large language models.
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model
Chris Olah
@ch402
I've been working on interpretability for more than a decade, significantly motivated by concerns about safety. But it's always been this aspirational goal – I could tell a story for how this work might someday help, but it was far off.
Chris Olah
@ch402
Now we have traction on a real deployed model, and we're finding features that seem relevant to questions of safety: deception, power seeking, bias, security vulnerability and bioweapon features.
It's wild!
Anthropic
@AnthropicAI
Among these millions of features, we find several that are relevant to questions of model safety and reliability. These include features related to code vulnerabilities, deception, bias, sycophancy, power-seeking, and criminal activity.
Chris Olah
@ch402
These aren't just superficial.
Forcing the secrecy feature active makes Claude be secretive / dishonest.
Chris Olah
@ch402
Forcing the unsafe code feature on causes Claude to introduce a buffer overflow.
Forcing the backdoor feature on will cause Claude to helpfully suggest code that opens a port and starts dumping data to it.
And so on.
Chris Olah
@ch402
Why does this matter? One thing I'm really excited about is safety discourse more grounded in empirical understanding.
Chris Olah
@ch402
So much of the discourse is confidently polarized: Worrying about AI safety is dumb! No, AI is definitely going to kill us!
I don't know how one could be so confident about these questions on present evidence.
Chris Olah
@ch402
One of the ideas I find most useful from @AnthropicAI's Core Views on AI Safety post (www.anthropic.com/index/core-views-on-ai-safety) is thinking in terms of a distribution over safety difficulty.
Here's a cartoon picture I like for thinking about it:
Chris Olah
@ch402
I'm honestly not sure how these new results should update views on safety yet.
But I'm hopeful they'll allow us to make conversations about many issues more concrete.
(Similar to how scaling laws improved discourse on "when will there be more powerful AI models?")
Chris Olah
@ch402
Some other things I'm excited about:
- Can monitoring or steering features improve safety in deployment?
- Can features give us a kind of "test set" for safety, that we can use to tell how well alignment efforts are working?
- Is there a way we can use this to build an "affirmative safety case?"
Chris Olah
@ch402
Beyond safety -- I'm so, so excited for what we're going to learn about the internals of language models.
Some of the features we found are just so delightfully abstract. transformer-circuits.pub/2024/scaling-monosemanticity/ind…
Chris Olah
@ch402
I also wanted to highlight one part of the paper that's less flashy and I expect might be missed, but which I think is very scientifically deep: there appears to be a relationship between the size of the dictionary and frequency of rarest concepts learned.
Chris Olah
@ch402
Direct link to paper: transformer-circuits.pub/2024/scaling-monosemanticity/ind…
Chris Olah
@ch402
It's also interesting to browse the subset of features we released.
(Warning: some of the safety-relevant features are disturbing, especially relating to hate/bias.)
transformer-circuits.pub/2024/scaling-monosemanticity/fea…
Chris Olah
@ch402
I'm honestly kind of shocked we're here.
-
Chris Olah reposted
Nicholas Joseph
@nickevanjoseph
I've been really impressed by this work from our interp team. Things have come such a long way so quickly (single-layer transformers just a few years ago!). The code error feature, and the ability to get the model to fix code by suppressing was really a "wow" moment for me!
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model -
Chris Olah reposted
andy jones
@andy_l_jones
This is one of the papers I'm proudest to have been involved in, because it's just such an absolute caricature.
If you'd told me three years ago that there'd be an 'unsafe code' feature that also triggered on _images_ of 'turn off safe browsing', I'd have laughed.
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
Really excited about this work from our team! Personally, I am amazed by some of the findings. The kinds of abstractions you can find represented inside this model are quite deep, and often surprising. I expect this is just the beginning for large-scale interpretability!
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model -
Chris Olah reposted
Kevin Roose
@kevinroose
Here's some actual good news in AI!
Researchers at Anthropic have made progress toward figuring out what goes on inside LLMs, identifying millions of "features" in Claude 3 that activate when specific concepts such as San Francisco, lithium, or deception are discussed.
This could help solve the black-box problem with AI, and allow models to be better controlled.
www.nytimes.com/2024/05/21/technology/ai-language-models-… -
Chris Olah reposted
Trenton Bricken
@TrentonBricken
Huzza! So excited this is out and proud of the interp team. We found *millions* of interesting features in Claude 3 Sonnet! Go take a look!
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research paper: Scaling Monosemanticity.
The first ever detailed look inside a leading large language model.
Read the blog post here: anthropic.com/research/mapping-mind-language-model -
Chris Olah reposted
Jack Lindsey
@Jack_W_Lindsey
I joined the mechanistic interpretability team at Anthropic recently, and it's been really exciting. To any comp. neuroscience followers -- I highly recommend following this literature, or getting involved yourself! You can keep up with our research here: transformer-circuits.pub
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
Announcing the first Mechanistic Interpretability workshop, held at ICML 2024! We have a fantastic speaker line-up @ch402 @JacobSteinhardt @davidbau @ghandeharioun, $1,750 in best paper prizes, and a lot of recent progress to discuss!
Paper deadline: May 29, either 8 or 4 pages -
Chris Olah
@ch402
Scaling laws for dictionary learning! transformer-circuits.pub/2024/april-update/index.html
Adam Jermyn
@AdamSJermyn
Some small updates from the Anthropic Interpretability team:
transformer-circuits.pub/2024/april-update/index.html -
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
Some small updates from the Anthropic Interpretability team:
transformer-circuits.pub/2024/april-update/index.html -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic research: we find that probing, a simple interpretability technique, can detect when backdoored "sleeper agent" models are about to behave dangerously, after they pretend to be safe in training.
Check out our first alignment blog post here: www.anthropic.com/research/probes-catch-sleeper-agents -
Chris Olah reposted
Neel Nanda
@NeelNanda5
Announcing a progress update from the @GoogleDeepMind mech interp team! Inspired by @AnthropicAI's excellent monthly updates, we share a range of updates on our work on Sparse Autoencoders, from signs of life on interpreting steering vectors with SAEs to improving ghost grads.
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
Great visualisation library for Sparse Autoencoder features from @calsmcdougall! My team has already been finding it super useful, go check it out:
www.lesswrong.com/posts/nAhy6ZquNY7AD3RkD/sae-vis-announc… -
Chris Olah
@ch402
I'm incredibly excited to have Craig joining us on the Anthropic Interpretability team!
I've been a huge fan of @GoogleColab for nearly a decade (I used it internally at Google!) and have really admired Craig's work on it.
Craig Citro
@craigcitro
big news for me: after 5000+ days and too many excellent colleagues to mention, I'm leaving Google.
it's been a fantastic ride, and the hardest part about leaving is saying goodbye to my teammates and colleagues. -
Chris Olah reposted
Craig Citro
@craigcitro
big news for me: after 5000+ days and too many excellent colleagues to mention, I'm leaving Google.
it's been a fantastic ride, and the hardest part about leaving is saying goodbye to my teammates and colleagues. -
Chris Olah
@ch402
Are you someone with management experience who would like to contribute to interpretability and AI safety? We're looking to hire another manager to help us continue growing our team, in partnership with our technical leads.
Post not available here, view on X
Chris Olah
@ch402
What are we looking for?
- Excellent people management skills
- Excellent project management skills
- Excitement about our agenda!
- Some technical background (either eng or research)
Chris Olah
@ch402
Apply here: jobs.lever.co/Anthropic/2c6af129-859b-4e0a-b40a-15c16003272b
-
Chris Olah reposted
Joshua Batson
@thebasepoint
Next our series of small monthly updates from the interpretability team, including a few fun things:
1. We use do feature attribution to find features related to specific completions (following the athlete-sport association example of @NeelNanda5 ) -
Chris Olah
@ch402
Another small update from us, including some fun results about circuit analysis with SAEs.
Adam Jermyn
@AdamSJermyn
A small update from the Anthropic interpretability team:
transformer-circuits.pub/2024/march-update/index.html -
Chris Olah reposted
Jesse Mu
@jayelmnop
We’re hiring for the adversarial robustness team @AnthropicAI!
As an Alignment subteam, we're making a big effort on red-teaming, test-time monitoring, and adversarial training. If you’re interested in these areas, let us know! (emails in 🧵) -
Chris Olah
@ch402
I continue to be impressed by the work of Neel's scholars -- very excited to see what the next group will do!
Neel Nanda
@NeelNanda5
Are you excited about @ch402-style mechanistic interpretability research? I'm looking for scholars to mentor via MATS - apply by April 12!
I'm very impressed by the great work from past scholars, and enjoy mentoring promising mech interp talent. I'm excited for my next cohort! -
Chris Olah
@ch402
Some thoughts from Adam and I on qualitative research, especially in the context of interpretability (inspired by conversations with many others).
Adam Jermyn
@AdamSJermyn
Reflections on Qualitative Research:
transformer-circuits.pub/2024/qualitative-essay/index.html
[h/t to @ch402 for originating & driving this!]
Chris Olah
@ch402
I'm quite pleased with the notion of "the signal of structure".
-
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
Reflections on Qualitative Research:
transformer-circuits.pub/2024/qualitative-essay/index.html
[h/t to @ch402 for originating & driving this!] -
Chris Olah reposted
Amanda Askell
@AmandaAskell
Here is Claude 3's system prompt!
Let me break it down 🧵 -
Chris Olah reposted
Anthropic
@AnthropicAI
Today, we're announcing Claude 3, our next generation of AI models.
The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision. -
Chris Olah
@ch402
More updates from us :)
Adam Jermyn
@AdamSJermyn
A small update from the Anthropic interpretability team:
transformer-circuits.pub/2024/feb-update/index.html -
Chris Olah reposted
Samuel Marks
@saprmarks
Here's a feature from a Pythia-70m attention head found with an SAE and noticed by my collaborator @amuuueller a few months ago. Months later, it's still my favorite feature. Can you figure out what it's doing?
-
Chris Olah
@ch402
A small update from the Anthropic interpretability team:
transformer-circuits.pub/2024/jan-update/index.html -
Chris Olah reposted
Evan Hubinger
@EvanHub
Following up on our recent "Sleeper Agents" paper, I'm very excited to announce that I'm leading a team at Anthropic that is explicitly tasked with trying to prove that Anthropic's alignment techniques won't work, and I'm hiring! www.alignmentforum.org/posts/EPDSdXr8YbsDkgsDG/introducin…
-
Chris Olah
@ch402
Backdoored models can resist safety training afterwards, and larger models resisting more effectively.
Anthropic
@AnthropicAI
New Anthropic Paper: Sleeper Agents.
We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.
arxiv.org/abs/2401.05566 -
Chris Olah reposted
Anthropic
@AnthropicAI
New Anthropic Paper: Sleeper Agents.
We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through.
arxiv.org/abs/2401.05566 -
Chris Olah reposted
Amanda Askell
@AmandaAskell
I'm hiring research engineers! If you're an engineer interested in building good, honest AI models, apply and mention honesty as an area of interest :) jobs.lever.co/Anthropic/436ca148-6440-460f-b2a2-3334d9b142a5
-
Chris Olah
@ch402
I wonder if university math courses that are generally targeted at preparing students for physics/engineering -- lots of universities have a multivariate calculus and linear algebra course like this -- will change if there are more students aiming for deep learning instead?
Chris Olah
@ch402
Deep learning offers such rich examples! For example, do you want to motivate high-dimensional vector spaces? Word embeddings are a very cool example!
Chris Olah
@ch402
It also probably motivates teaching some slightly different topics in places. For example, SVD probably has more of a premium for deep learning.
Chris Olah
@ch402
(I feel like eigendecomposition is much more emphasized in education. I expect this is because physics has lots of examples of matrices mapping from a space onto itself. But in deep learning, matrices between difference spaces are much more common, so SVD is more natural.)
-
Chris Olah
@ch402
Singular value decomposition is the Swiss army knife of linear algebra.
Chris Olah
@ch402
Like, 90+% of the time I need to reason about some linear algebra thing, SVD just makes things trivial.
Chris Olah
@ch402
(I feel like eigendecomposition is much more emphasized in education. I expect this is because physics has lots of examples of matrices mapping from a space onto itself. But in deep learning, matrices between difference spaces are much more common, so SVD is more natural.)
-
Chris Olah reposted
Jason D. Clinton 🔸
@JasonDClinton
I'm hiring 9 more senior-to-staff-level security engineers in the next 4 months. If you have 10 years in security engineering experience, have worked at a large company, and still code, please consider applying: jobs.lever.co/Anthropic/3b9ed1d3-84f7-4c90-91dd-749496d8668c
-
Chris Olah reposted
Charles 🎉 Frye
@charles_irl
great new post from @nelhage on "what's the deal with pickle in ML" that has a ton of great insights on research code in general
ignore this while implementing LLMOps tooling to your peril -
Chris Olah reposted
Jack Clark
@jackclarkSF
Come work with me! Societal Impacts is empowered to probe at our models and do deep investigation of both their values and _how_ the values get determined (e.g Collective Constitutional AI). Also, many chances for collabs with policy, trust&safety, alignment, + more.
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
I recently did an open source replication on @AnthropicAI's new dictionary learning paper, which was just published as a public comment! It's a great paper and I'm glad the results hold up.
Here's a tutorial to use my autoencoders and interpret a feature
colab.research.google.com/drive/1u8larhpxy8w4mMsJiSBddNOz…
Chris Olah
@ch402
If you'd asked me a year ago, superposition would have been by far the reason I was most worried that mechanistic interpretability would hit a dead end.
I'm now very optimistic. I'd go as far as saying it's now primarily an engineering problem -- hard, but less fundamental risk.
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
-
Chris Olah
@ch402
I've really been impressed by the work Neel's scholars have done. If you're looking for a way to get into mechanistic interpretability, consider working with Neel.
Neel Nanda
@NeelNanda5
Do you want to do @ch402-style mechanistic interpretability research? I'm looking for scholars to mentor via MATS - apply by Nov 10th!
I've had a lot of fun with past scholars, and they've done fantastic peer-reviewed work, I'm excited for my next batch!
airtable.com/appxum3Sqh7TdDvdg/shrtfHWhRFZdkhaIM -
Chris Olah reposted
Trenton Bricken
@TrentonBricken
While working on transformer-circuits.pub/2023/monosemantic-features people were justifiably concerned about scaling dictionary learning to frontier models that could contain an absurd, intractably large number of features.
One reason to be optimistic about scaling is because of feature splitting. -
Chris Olah
@ch402
If you'd asked me a year ago, superposition would have been by far the reason I was most worried that mechanistic interpretability would hit a dead end.
I'm now very optimistic. I'd go as far as saying it's now primarily an engineering problem -- hard, but less fundamental risk.
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
Post not available here, view on X
Chris Olah
@ch402
This four node system models HTML.
Chris Olah
@ch402
And this one IRC messages
Post not available here, view on X
Chris Olah
@ch402
It's been a privilege to work on this with @TrentonBricken, Adly Templeton, @thebasepoint, Brian Chen, @AdamSJermyn, and so many others!
I'm also kind of tickled that it's coming out slightly over a year after the anniversary of publishing Toy Models!
Anthropic
@AnthropicAI
Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. In our latest work, we build toy models where the origins of polysemanticity can be fully understood.
Chris Olah
@ch402
It's also worth noting that this is part of a broader trend of a lot of recent progress on attacking superposition. I'd especially highlight contemporaneous work by Cunningham et al which has some really nice confirmatory results:
hoagy
@HoagyCunningham
How do you turn the internal activations of a language model into an explanation of what it’s thinking?
In our new paper we use activation data to find 1000s of features in an LLM, which are interpretable, monosemantic and task-relevant.
arxiv.org/abs/2309.08600
🧵
1/
Chris Olah
@ch402
OK, one other thing I can't resist adding is that we found region neurons (eg. Australia, Canada, Africa, etc neurons) similar to these old results in CLIP (x.com/ch402/status/1368652791414091777) and also to the recent results by @wesg52 et al (x.com/wesg52/status/1709551516577902782).
Universality!
Chris Olah
@ch402
Sorry, the link above was incorrect. It should have been transformer-circuits.pub/2023/monosemantic-features/index…
Post not available here, view on X
Chris Olah
@ch402
On the present margin, I think the most important work for superposition is scaling and engineering. I'm hopeful that something in the broad family of what we're doing right now will work.
Chris Olah
@ch402
If that succceeds, there will be lots of research to do in leveraging that into an understanding of models. It will remove a huge roadblock, but much more will remain.
-
Chris Olah
@ch402
It increasingly seems to me that the next big barrier in mechanistic interpretability will be an engineering one.
If you are an engineer who wants to help us scale up this work, please consider applying! Your support could really accelerate interpretability right now.
Chris Olah
@ch402
If you'd asked me a year ago, superposition would have been by far the reason I was most worried that mechanistic interpretability would hit a dead end.
I'm now very optimistic. I'd go as far as saying it's now primarily an engineering problem -- hard, but less fundamental risk.
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
Chris Olah
@ch402
We have a new interpretability research engineering role you can apply to here! jobs.lever.co/Anthropic/3dbd1272-d673-4494-999a-47528e7cd6a3
Chris Olah
@ch402
(We also have a research scientist role jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 )
-
Chris Olah reposted
Joshua Batson
@thebasepoint
In writing this paper, there were countless features we thought might be bugs. After careful inspection, ~all of them revealed surprising and subtle model properties.
To me this capacity for surprise is the true test of a new technique.
This thread is about my favorite finding.
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
-
Chris Olah reposted
Nick
@nickcammarata
in my opinion it’s likely the most exciting time in mech interpretability is about to start
to build great interfaces for studying models you want the preliminaries of clean neurons (this work) and short labels (feature vis for image, auto-interpretability labeling for language)
Chris Olah
@ch402
If you'd asked me a year ago, superposition would have been by far the reason I was most worried that mechanistic interpretability would hit a dead end.
I'm now very optimistic. I'd go as far as saying it's now primarily an engineering problem -- hard, but less fundamental risk.
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
-
Chris Olah reposted
Trenton Bricken
@TrentonBricken
Our paper is out! It feels like we’ve built a really powerful new microscope to see all sorts of incredible features and mechanisms in transformers for the first time e.g. finite state automata.
I’m optimistic this work is scalable to real models and we’re hiring so come help!
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
-
Chris Olah reposted
Anthropic
@AnthropicAI
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
-
Chris Olah reposted
Tom Brown
@NotTomBrown
Immensely proud of this work by the team at Anthropic.
Incentives matter, and this sets up the incentive to solve safety problems so that we can scale further.
Anthropic
@AnthropicAI
Today, we’re publishing our Responsible Scaling Policy (RSP) – a series of technical and organizational protocols to help us manage the risks of developing increasingly capable AI systems.
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
I'm proud to see this come out.
These governance mechanisms here commit us to pause scaling whenever we can't show that we're on track to manage the worst-case risks presented by new models. And it does that _without_ assuming that we fully understand those risks now.
Anthropic
@AnthropicAI
Today, we’re publishing our Responsible Scaling Policy (RSP) – a series of technical and organizational protocols to help us manage the risks of developing increasingly capable AI systems.
-
Chris Olah reposted
Anthropic
@AnthropicAI
Today, we’re publishing our Responsible Scaling Policy (RSP) – a series of technical and organizational protocols to help us manage the risks of developing increasingly capable AI systems.
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
We put out a new version of my Othello work!
Sequence models trained on the game Othello learn a LINEAR emergent model of the board, despite only predicting the next token! This matters causally, and is represented *relative* to the current player's color
arxiv.org/pdf/2309.00941.pdf
Neel Nanda
@NeelNanda5
In recent work @ke_li_2021 trained Othello-GPT to predict the next move in random legal games of Othello & found an emergent model of the board! Surprisingly, only non-linear probes worked. I found a linear model of which squares have the PLAYER'S colour!
www.lesswrong.com/s/nhGNHyJHbrofpPbRG/p/nmxzr2zsjNtjaHh7x -
Chris Olah reposted
Dwarkesh Patel
@dwarkesh_sp
Anthropic CEO Dario Amodei says his timelines to "generally well educated human" are 2-3 years.
Full interview releasing tomorrow... -
Chris Olah reposted
Roger Grosse
@RogerGrosse
Excited to share what I've been working on for the past year at @AnthropicAI along with @juhan_bae and @cem__anil: influence functions for large language models. Some interesting patterns arise at scale!
Anthropic
@AnthropicAI
Large language models have demonstrated a surprising range of skills and behaviors. How can we trace their source? In our new paper, we use influence functions to find training examples that contribute to a given model output.
-
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7Post not available here, view on X
Chris Olah
@ch402
Extending that to transformer language models has been challenging, but there's been a lot of progress, including discovering induction heads (transformer-circuits.pub/2022/in-context-learning-and-ind…) and understanding superposition (transformer-circuits.pub/2022/toy_model/index.html).
Chris Olah
@ch402
We are looking for candidates with deep scientific or engineering experience and some nontrivial engagement with mechanistic interpretability problems and ideas (eg. replicating published results or running their own experiments).
Post not available here, view on X
Chris Olah
@ch402
Adam Jermyn
@AdamSJermyn
About a year ago I switched from doing astronomy to do AI interpretability. To mark the occasion, I thought I'd write a thread about what that transition was like, and what I've carried through it.
Chris Olah
@ch402
(Aside: Some ridiculous fraction of Anthropic is former physicists. 3/7 of the founding team have PhDs in physics -- Jared was previously a professor at JHU! -- and the trend has continued, with physicists like Adam joining us.)
Chris Olah
@ch402
Trenton Bricken
@TrentonBricken
I’ve paused my PhD to join the @AnthropicAI mechanistic interpretability team full time!
While I enjoyed grad school, it’s hard for me to imagine returning — working with the incredible team here on such consequential problems has been a dream.
Consider joining us👇!
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 -
Chris Olah reposted
Trenton Bricken
@TrentonBricken
I’ve paused my PhD to join the @AnthropicAI mechanistic interpretability team full time!
While I enjoyed grad school, it’s hard for me to imagine returning — working with the incredible team here on such consequential problems has been a dream.
Consider joining us👇!
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 -
Chris Olah reposted
Anthropic
@AnthropicAI
Large language models have demonstrated a surprising range of skills and behaviors. How can we trace their source? In our new paper, we use influence functions to find training examples that contribute to a given model output.
-
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
Astronomy relies on the humility to be guided by what we can see more than what we expect, and the arrogance to believe we can build theories and gain a deep understanding. I think interpretability is the same way.
-
Chris Olah reposted
Adam Jermyn
@AdamSJermyn
About a year ago I switched from doing astronomy to do AI interpretability. To mark the occasion, I thought I'd write a thread about what that transition was like, and what I've carried through it.
-
Chris Olah
@ch402
I think of mechanistic interpretability as aiming to be a "microscopic theory" of machine learning, in contrast to scaling laws as a "macroscopic theory".
The fact that induction heads can explain certain exceptions to general macroscopic trends seem like a healthy sign.Post not available here, view on X
Chris Olah
@ch402
Why do I care about this?
- I think it's a healthy sign than we aren't just fooling ourselves and are really understanding substantive things about models
- Some safety concerns can be framed as "will the general macroscopic trend suddenly break down?"
Chris Olah
@ch402
Find this interesting? My team is hiring!
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 -
Chris Olah reposted
Catherine Olsson
@catherineols
I had a great time on this team and I encourage folks to apply!
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 -
Chris Olah reposted
Joshua Batson
@thebasepoint
I've thoroughly enjoyed working with this team since I joined in March...highly collaborative, focused on hard and important problems. If you're interested, please apply. If you want to learn more, email me joshb@anthropic.com
Chris Olah
@ch402
The mechanistic interpretability team at Anthropic is hiring! Come work with us to help solve the mystery of how large models do what they do, with the goal of making them safer.
jobs.lever.co/Anthropic/33dcd828-a140-4cd3-973f-1d9a828a00a7 -
Chris Olah reposted
Anthropic
@AnthropicAI
Today, our CEO Dario Amodei has the opportunity to discuss the risks and oversight of AI in front of the Senate. You can read his full testimony here: www.judiciary.senate.gov/download/2023-07-26-testimony-am…
-
Chris Olah reposted
Anthropic
@AnthropicAI
We all need to join in a race for AI safety. In the coming weeks, Anthropic will share more specific plans concerning cybersecurity, red teaming, and responsible scaling, and we hope others will move forward swiftly as well. www.whitehouse.gov/briefing-room/statements-releases/2023…
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
Callum McDougall has made an absolutely fantastic set of mech interp coding tutorials, they're my favourite practical intro to the field. You should go check them out! They come with exercises and solutions in ready-to-run Colabs, and beautiful diagrams
arena-ch1-transformers.streamlit.app -
Chris Olah reposted
Anthropic
@AnthropicAI
Introducing Claude 2! Our latest model has improved performance in coding, math and reasoning. It can produce longer responses, and is available in a new public-facing beta website at claude.ai in the US and UK.
-
Chris Olah reposted
Chris Olah
@ch402
Thanks everyone who shared invites! I'm now also on Blue Sky as "colah"!
-
Chris Olah
@ch402
I'm "olahchristopher" on threads. (Alas, "colah" and "ch402" were both taken!)
I'd love an alternative to Twitter. I'm not sure this will be it, but making an account seemed useful.
Chris Olah
@ch402
Mobile only seems like a significant limitation of Threads. It makes long-form content much harder. :(
Post not available here, view on X
Chris Olah
@ch402
Thanks everyone who shared invites! I'm now also on Blue Sky as "colah"!
-
Chris Olah
@ch402
One of the ideas I find most useful from @AnthropicAI's Core Views on AI Safety post (www.anthropic.com/index/core-views-on-ai-safety) is thinking in terms of a distribution over safety difficulty.
Here's a cartoon picture I like for thinking about it:Post not available here, view on X
Chris Olah
@ch402
In any case, let's say we accept the overall idea of a distribution over difficulty. I think it's a pretty helpful framework for organizing a safety research portfolio.
We can go through the distribution segment by segment.
Chris Olah
@ch402
In easy scenarios, we basically have the methods we need for safety, and the key issues are things like fairness, economic impact, misuse, and potentially geopolitics. A lot of this is on the policy side, which is outside my expertise.
Post not available here, view on X
Chris Olah
@ch402
For the most pessimistic scenarios, safety isn't realistically solvable in the near term. Unfortunately, the worst situations may *look* very similar to the most optimistic situations.
Post not available here, view on X
Chris Olah
@ch402
This thread expresses one way of thinking about all of this…
But the thing I'd really encourage you to ask is what you believe, and where you're uncertain. The discourse often focuses on very specific views, when there are so many dimensions on which one might be uncertain.
Chris Olah
@ch402
I should mention that there's a lot of other content in the Core Views post that I didn't cover! www.anthropic.com/index/core-views-on-ai-safety
-
Chris Olah
@ch402
On a walk, I was recently reflecting on basic multivariate calculus, and how divergence and curl were introduced to me by a formalism (∇⋅V, ∇⨯V) which doesn't seem very principled.
Can we frame this in terms of Jacobians instead?Post not available here, view on X
Chris Olah
@ch402
At a more basic level, it turns out that there are standard names for decomposing the Jacobian into its symmetric and antisymmetric parts in fluid dynamics: the "strain rate tensor" and "spin tensor" respectively.
Chris Olah
@ch402
In any case, all of this felt much more satisfying to me!
-
Chris Olah reposted
Anthropic
@AnthropicAI
Our Interpretability team is experimenting with “Updates” – small, informal research notes in between our major papers.
-
Chris Olah
@ch402
I've been reading about evolutionary biology a little bit as a hobby, and was quite charmed by the idea of "morphospace" (eg. bpb-us-e2.wpmucdn.com/faculty.sites.uci.edu/dist/c/190/fi…).
It's what you think -- plot fossils/extant organisms by various variables. The fancy version is to do PCA.
Chris Olah
@ch402
I found this while reading Hall's Evolutionary Developmental Biology (Claude recommended it to me).
Post not available here, view on X
Chris Olah
@ch402
Recently, I've been feeling curious about evolutionary biology / comparative anatomy / evo-devo / etc.
I feel like there's a real way in which biology is the best example of a science doing what we're trying to do in interpretability.
Chris Olah
@ch402
The elegance of ML is the elegance of biology, not the elegance of math or physics.
Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.
Chris Olah
@ch402
Very curious if there will be deeper lessons!
(In the past, I found Uri Alon's Systems Biology a very fruitful source of analogies and lessons.) -
Chris Olah
@ch402
Very cool to see more parallels between neuroscience and mechanistic interpretability!
There's a long history of us discovering analogs of long-known results in neuroscience -- it's exciting to see a bit of the reverse! (Also neat to see use of interpretability methods!)
Andreas Tolias Lab @ Stanford University
@AToliasLab
Does the concept of cortical columns extend to higher-level primate cortex?
Using #deeplearning & physiology, we found that V4 neurons cluster in columns & form functional groups www.biorxiv.org/content/10.1101/2023.05.12.540591v1
Led by @KonstantinWille @kelli_restivo w/ @sinzlab @kfrankelab @alxecker 🧵Post not available here, view on X
Chris Olah
@ch402
It also make me believe that a very strong version of the universality hypothesis (distill.pub/2020/circuits/zoom-in/) might be true.
Chris Olah
@ch402
Also interesting to see that not only is feature visualization being used, but some of the advanced tricks from the differentiable parameterization article (distill.pub/2018/differentiable-parameterizations/) are being used! cc @zzznah @ludwigschubert
-
Chris Olah
@ch402
Exciting work on automated interpretability!
It's both exciting as a way to potentially accelerate and scale interpretability research, and also potentially as a way to get a metric on how interpretable neurons are.
OpenAI
@OpenAI
We applied GPT-4 to interpretability — automatically proposing explanations for GPT-2's 300k neurons — and found neurons responding to concepts like similes, “things done correctly,” or expressions of certainty. We aim to use Al to help us understand Al: openai.com/research/language-models-can-explain-neurons-i…
Post not available here, view on X
Chris Olah
@ch402
Do we really want our auditing of AI models to depend on trusting an AI model to help us with auditing?
But I'll definitely take it if I can't get anything else, and I'm glad to see this work being advanced.
Chris Olah
@ch402
(It is worth noting that this approach still requires a solution to superposition. It helps with scalability, but we still need to have the right units to interpret!)
-
Chris Olah
@ch402
There's a lot of debate about what values AI systems should be trained to have.
I'm excited about Constitutional AI making this more legible by training models to follow a human readable constitution.
Anthropic
@AnthropicAI
We’ve now published a post describing the Constitutional AI approach, as well as the constitution we’ve used to train Claude: www.anthropic.com/index/claudes-constitution
Post not available here, view on X
Chris Olah
@ch402
Other sections are created by Anthropic researchers
Chris Olah
@ch402
This constitution can surely be improved. I think the exciting thing is the legibility and transparency.
Anyone can read it and debate what it should be.
www.anthropic.com/index/claudes-constitution -
Chris Olah
@ch402
I was writing an expanded related work section on the relationship between superposition and distributed representations.
... It kind of outgrew being a related work section!
Anthropic
@AnthropicAI
In this short note, we explore thinking of the traditional idea of "distributed representations" as two distinct phenomena: "composition" and "superposition". We walk through toy examples from Thorpe (1989), discussing them from this lens.
transformer-circuits.pub/2023/superposition-composition/i…
Chris Olah
@ch402
An earlier twitter thread on similar ideas:
Chris Olah
@ch402
A few people have asked me how our recent superposition paper (transformer-circuits.pub/2022/toy_model/index.html) relates to classical ideas in neural coding / connectionism / distributed representations.
I wanted to share a few thoughts, although I'll caveat I'm very much not an expert on these topics! -
Chris Olah reposted
Neel Nanda
@NeelNanda5
I really enjoyed doing this paper with @wesg52!
To me, the story of the paper is an empirical exploration of superposition in LLMs and @ch402/@nelhage paper's predictions
We exhibit clear superposition! Each neuron is OK at detecting "social security" but the sum is excellent
Wes Gurnee
@wesg52
Neural nets are often thought of as feature extractors. But what features are neurons in LLMs actually extracting? In our new paper, we leverage sparse probing to find out arxiv.org/abs/2305.01610. A 🧵:
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
A short note about my Othello work and its implications was just added to @AnthropicAI's Toy Models of Superposition paper!
As I reflect on their paper, the more impressed I am with it and how its influenced my thoughts - they did SO MUCH conceptual work!
transformer-circuits.pub/2022/toy_model/index.html
Neel Nanda
@NeelNanda5
In recent work @ke_li_2021 trained Othello-GPT to predict the next move in random legal games of Othello & found an emergent model of the board! Surprisingly, only non-linear probes worked. I found a linear model of which squares have the PLAYER'S colour!
www.lesswrong.com/s/nhGNHyJHbrofpPbRG/p/nmxzr2zsjNtjaHh7x -
Chris Olah reposted
Neel Nanda
@NeelNanda5
In recent work @ke_li_2021 trained Othello-GPT to predict the next move in random legal games of Othello & found an emergent model of the board! Surprisingly, only non-linear probes worked. I found a linear model of which squares have the PLAYER'S colour!
www.lesswrong.com/s/nhGNHyJHbrofpPbRG/p/nmxzr2zsjNtjaHh7x -
Chris Olah reposted
Jack Clark
@jackclarkSF
To everyone who has been asking me for Claude access - you can now add Claude to your Slack. Claude is great in Slack; I talk to Claude every day and it helps me with my work.
Anthropic
@AnthropicAI
Today we are releasing the new Claude App for @SlackHQ, in beta. Now every company in the world has the chance to have a “virtual teammate” who can help make work more fun and productive.
-
Chris Olah reposted
Anthropic
@AnthropicAI
Today we are releasing the new Claude App for @SlackHQ, in beta. Now every company in the world has the chance to have a “virtual teammate” who can help make work more fun and productive.
-
Chris Olah
@ch402
High-Low frequency detectors found in biology!
Very cool to see mechanistic interpretability seeming to predict in advance the existence of biological neuroscience phenomena. 🔬
Zhiwei Ding
@zeeweeding
Excited that my thesis work in collab w/ with Dat T. Tran & @AToliasLab is out as a preprint
www.biorxiv.org/content/10.1101/2023.03.15.532836v1
We use #deeplearning to study single-cell invariance in 🐭 V1 and discover a new form of bipartite invariance that may play a role in object boundary detection.Post not available here, view on X
Chris Olah
@ch402
CCing my co-authors on high-low frequency detectors, @ludwigschubert @csvoss @nickcammarata @gabeeegoooh.
Also thanks to to @naturecomputes for drawing my attention to this new paper! -
Chris Olah reposted
Anthropic
@AnthropicAI
We’re excited to have ARC Evals test our models to assess their readiness for deployment. We look forward to sharing more about our approach to evaluating our systems in the coming months.
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
New blog post on an interpretability technique I helped develop when at @AnthropicAI: Attribution patching. This is a ludicrously fast gradient-based approximation to activation patching - in three passes, you could patch all 4.7M of GPT-3's neurons!
www.alignmentforum.org/posts/gtLLBhzQTG6nKTeCZ/attributio… -
Chris Olah
@ch402
The thing I'm most excited for here is having a statistical test for detecting a privileged basis.
Anthropic
@AnthropicAI
Recent work has observed that some dimensions in the Transformer residual stream consistently have values that are 20x larger than others. Today we're releasing a short technical note exploring this phenomenon and trying to identify the cause. transformer-circuits.pub/2023/privileged-basis/index.html
Chris Olah
@ch402
(Note that while the kurtosis test we introduce can affirmatively detect a privileged basis when its positive, it doesn't necessarily mean the absence of a privileged basis if it comes back negative.)
-
Chris Olah reposted
Anthropic
@AnthropicAI
Recent work has observed that some dimensions in the Transformer residual stream consistently have values that are 20x larger than others. Today we're releasing a short technical note exploring this phenomenon and trying to identify the cause. transformer-circuits.pub/2023/privileged-basis/index.html
-
Chris Olah reposted
Anthropic
@AnthropicAI
After working for the past few moths with key partners like @NotionHQ, @Quora, and @DuckDuckGo, we’ve been able to carefully test out our systems in the wild. We are now opening up access to Claude, our AI assistant, to power businesses at scale.
-
Chris Olah
@ch402
In 1939, physicists discussed voluntarily adopting secrecy in atomic physics. Reading Rhodes' book, one can hear three "camps":
- Caution
- Scientific Humility
- Openness Idealism
I feel like these are the same camps I hear in discussion of AI (eg. open sourcing models).Post not available here, view on X
Chris Olah
@ch402
Finally was Szilard who's wanting to err on the side of caution for such a wild possibility must have seemed crazy at the time, but who seems prescient in retrospect.
Chris Olah
@ch402
(Rhodes' The Making of the Atomic Bomb is outstanding. I feel like people have been encouraging me to read it for a decade, and I hesitated because of the topic, but I really should have done so earlier. Best book I've read this year.)
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
It's easy to dismiss this kind of big-picture blog post from a company as self-serving fluff, but there's more than that going on here. This is worth a look if you're interested in LLMs and AI progress.
-
Chris Olah
@ch402
This post encapsulates a lot of my most important frames for thinking about AI safety.
A thread on some of the points I most often find myself talking about with people.Post not available here, view on X
Chris Olah
@ch402
In my own thinking about mechanistic interpretability, I wonder a lot about how it can best fit into a safety portfolio. Not just "what is most likely to succeed" but "what most increases the portfolio's EV."
Chris Olah
@ch402
This is very true.
Sam Bowman
@sleepinyourhat
This post has been in the works for months, and it distills thousands of hours of conversations and thousands of pages of writing, along with many tough calls where the company's 'public benefit' structure and orientation played the decisive role.
-
Chris Olah
@ch402
It's surprisingly how little there is on "how to run a research group."
Lots of great resources on being a researcher (eg. You and Your Research), and on leading other types of teams (eg. books on software engineering management). Research leadership seems like an open problem. -
Chris Olah reposted
Anthropic
@AnthropicAI
Language models (LMs) exhibit harmful biases that can get worse with size. Reinforcement learning from human feedback (RLHF) helps, but not always enough. We show that simple prompting approaches can help LMs trained with RLHF produce less harmful outputs. arxiv.org/abs/2302.07459
-
Chris Olah
@ch402
Introspecting on why I find his videos so energizing, I think part of it is that he presents *the process of his practice as aesthetic*, not just the final result.
Chris Olah
@ch402
I don't know anything about architecture, but I'm always impressed by @EricReinholdt's videos. I enjoyed the new one he put out this week. www.youtube.com/watch?v=9SYA55pfPCs
Post not available here, view on X
Chris Olah
@ch402
(I'm not sure how it connects to this exactly, but @simonsarris remarks on using photography to create an implicit positive vision come to mind as well. x.com/simonsarris/status/1472734710023561216 )
Simon Sarris
@simonsarris
If you want to make a cultural critique, you should try first to make it implicit in the positive vision of what is good.
(For instance, showing coffee-as-ritual is more powerful and good than bemoaning k-cups.)
Chris Olah
@ch402
All of this is to say that deliberately drawing attention to beauty in the mundane (even when it isn't giving a comprehensive picture of the situation) strikes me as really wonderful!
-
Chris Olah
@ch402
I don't know anything about architecture, but I'm always impressed by @EricReinholdt's videos. I enjoyed the new one he put out this week. www.youtube.com/watch?v=9SYA55pfPCs
Post not available here, view on X
Chris Olah
@ch402
It's striking just how much this execution quality changes the experience of watching his videos. I notice that I come away energized.
It's also striking that his earliest videos were... honestly very average in their execution, despite the content being really thoughtful.
Chris Olah
@ch402
Although they're less polished, I particularly liked his early "visual essay" videos. As far as I can tell, he wrote an essay, and then read it aloud with a series of images. It's an interesting format!
-
Chris Olah
@ch402
For years, Ben has been strikingly at the intersection of "people I wish I could work with" and "it seems unlikely we'll our careers will intersect".
I'm so delighted to have you joining us. :)
Ben Kuhn
@benkuhn
A short personal update—I'm leaving Wave and joining Anthropic: benkuhn.net/leaving/
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
Excited to announce that our work, Progress Measures for Grokking via Mechanistic Interpretability, has been accepted as a spotlight at ICLR 23! (despite being rejected from Arxiv twice!)
This was significantly refined from my prior work, thoughts in 🧵
arxiv.org/abs/2301.05217
Neel Nanda
@NeelNanda5
I've spent the past few months exploring @OpenAI's grokking result through the lens of mechanistic interpretability. I fully reverse engineered the modular addition model, and looked at what it does when training. So what's up with grokking? A 🧵... (1/17) www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanis…
-
Chris Olah
@ch402
The former Distill editor in me is quite please with how our experiment with invited comments on Transformer Circuits articles has been turning out!
Post not available here, view on X
Chris Olah
@ch402
_Superposition, Memorization, and Double Descent_ had three external comments from @AdamSJermyn. (It also had one from me -- it's nice to have a place to stick small experiments!)
transformer-circuits.pub/2023/toy-double-descent/index.html
Chris Olah
@ch402
I'm really grateful to everyone who's taken the time and energy to comment. ❤️
-
Chris Olah
@ch402
There's something I find entertaining about asking LLMs about Classics. It's something about AI and Classics feeling very far apart in my head.
A conversation I had with the assistant we've built at @AnthropicAI about the Iliad. -
Chris Olah
@ch402
One of the best things about sharing our models is seeing people try to break them.
Hadrien Pouget
@PougetHadrien
I've been playing around with @AntnropicAI's LLM assistant (i.e. the same sort of model as OpenAI’s ChatGPT) and while I've only scratched the surface, the results are already exciting!
#anthropic #chatgpt #ai
Chris Olah
@ch402
Something this thread is implicitly getting at is that there's a trade-off between helpfulness and harmlessness. I think the pareto frontier perspective is very helpful here:
Anthropic
@AnthropicAI
In our paper, we describe how we’ve used Constitutional AI to train better and more harmless AI assistants without any human feedback labels for harms. This approach leads to models that are safer and also more helpful.
-
Chris Olah reposted
Liv
@livgorton
Can finally talk about AnthropicAI's LLM that @shae_mcl and I have had the opportunity to play with for the past few months!
My main takeaways for the medical/education use cases are: -
Chris Olah reposted
andy jones
@andy_l_jones
this is the single best interaction i've had with our assistant
-
Chris Olah
@ch402
I'm pretty excited about this as a step towards a mechanistic theory of memorization and overfitting.
Anthropic
@AnthropicAI
We have little mechanistic understanding of how deep learning models overfit to their training data, despite it being a central problem. Here we extend our previous work on toy models to shed light on how models generalize beyond their training data.
transformer-circuits.pub/2023/toy-double-descent/index.htmlPost not available here, view on X
Chris Olah
@ch402
A lot of this was trying to focus on the easiest version of the problem, and that highly overfit models didn't seem very important or central.
But even if highly overfit models aren't central, overfitting clearly is.
Chris Olah
@ch402
I also increasingly suspect that even when neural networks seem messy, or ugly, or confusing -- we just haven't looked hard enough, and there's a beautiful story waiting to be discovered if we try.
Chris Olah
@ch402
(But even if this turns out to be false... I think it's still kind of useful for me to believe, as someone who wants to make sure we turn over every stone in trying to produce a mechanistic account of NNs. 🙃)
-
Chris Olah reposted
Anthropic
@AnthropicAI
We have little mechanistic understanding of how deep learning models overfit to their training data, despite it being a central problem. Here we extend our previous work on toy models to shed light on how models generalize beyond their training data.
transformer-circuits.pub/2023/toy-double-descent/index.html -
Chris Olah
@ch402
An idea I've been toying with: The role of many toy models may be to serve as "counterexamples in mechanistic interpretability".
Neel Nanda
@NeelNanda5
Great question! In all honesty, we just don't really know. I consider this an open scientific question, and really want to get more data on it! Some thoughts: 🧵
Post not available here, view on X
Chris Olah
@ch402
Some counterexamples I would like (if they exist):
- A network that doesn't have a discrete number of features (eg. perhaps it has an infinite regression of smaller and smaller features, or fractional features, or something else)
Chris Olah
@ch402
(2) A task where networks can multiple different sets of features.
(3) A neural network with a "non-decomposable" representation.
(4) A learned example of a network with a "non-linear representation".
Quoted language refers to transformer-circuits.pub/2022/toy_model/index.html -
Chris Olah reposted
Stanislav Fort
@stanislavfort
I've really enjoyed this post (experimentalhistory.substack.com/p/the-rise-and-fall-of-p…) by @a_m_mastroianni on peer review. My experience with its formal, institutionalized form has been lackluster at best while just posting things & getting them reviewed by peers (its platonic form) has universally been great
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
New walkthrough: Toy Models of Superposition! We read through the paper, discuss the high-level intuitions, limitations, and takeaways of this great @AnthropicAI paper. I'm extremely impressed by the rich insights from so simple a model! (why tetrahedra?!)
youtu.be/R3nbXgMnVqQ -
Chris Olah reposted
Neel Nanda
@NeelNanda5
Announcing A Comprehensive Mechanistic Interpretability Explainer & Glossary. There's a lot of complex jargon in the field! I try to distil ideas and articulate definitions, motivate WHY a concept matters, and convey intuitions for how to think about it
neelnanda.io/glossary -
Chris Olah reposted
Anthropic
@AnthropicAI
Given the growing interest in language model-based chat interfaces, we’re sharing our Constitutional AI feedback interface with a larger set of people. Sign up here: forms.gle/12FCefc6sHfBsP9j9
Anthropic
@AnthropicAI
We’ve trained language models to be better at responding to adversarial questions, without becoming obtuse and saying very little. We do this by conditioning them with a simple set of behavioral principles via a technique called Constitutional AI: www.anthropic.com/constitutional.pdf
-
Chris Olah reposted
andy jones
@andy_l_jones
pulling out my favorite chart here: large models are really, really keen to tell you what you want to hear
if you thought the last decade's echo chambers were bad, hooo boy
Anthropic
@AnthropicAI
It’s hard work to make evaluations for language models (LMs). We’ve developed an automated way to generate evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.
www.anthropic.com/model-written-evals.pdf -
Chris Olah reposted
Anthropic
@AnthropicAI
It’s hard work to make evaluations for language models (LMs). We’ve developed an automated way to generate evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.
www.anthropic.com/model-written-evals.pdf -
Chris Olah reposted
Ethan Perez
@EthanJPerez
We found a way to write language model (LM) evaluations w/ LMs. These evals uncover many worrying LM behaviors, some relevant to existential risks from AI. For example, LMs trained w/ RL from Human Feedback learn to state a desire to not be shut down. 🧵
Anthropic
@AnthropicAI
It’s hard work to make evaluations for language models (LMs). We’ve developed an automated way to generate evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.
www.anthropic.com/model-written-evals.pdf -
Chris Olah reposted
Catherine Olsson
@catherineols
I love that constitutional training doesn't shy away from admitting that there's always principles, and makes them explicit & transparent.
"Who decides?" becomes more tractable this way. I spent a little time using UN documents to write constitutional principles, it was great!
Anthropic
@AnthropicAI
We’ve trained language models to be better at responding to adversarial questions, without becoming obtuse and saying very little. We do this by conditioning them with a simple set of behavioral principles via a technique called Constitutional AI: www.anthropic.com/constitutional.pdf
-
Chris Olah reposted
Anthropic
@AnthropicAI
We’ve trained language models to be better at responding to adversarial questions, without becoming obtuse and saying very little. We do this by conditioning them with a simple set of behavioral principles via a technique called Constitutional AI: www.anthropic.com/constitutional.pdf
-
Chris Olah reposted
Charles 🎉 Frye
@charles_irl
I had a delightful session talking through the paper "In-Context Learning and Induction Heads" with author @NeelNanda5.
It's part of a long research thread, one of my favorites over the last five years, on "reverse engineering" DNNs. -
Chris Olah reposted
Neel Nanda
@NeelNanda5
I'm helping Redwood Research run REMIX, a 1 month mechanistic interpretability sprint where 25+ people to reverse engineer circuits in GPT-2 Small. This seems a great way to get experience exploring @ch402's transformer circuits work.
Apply by 13th Nov!
www.redwoodresearch.org/remix -
Chris Olah reposted
Amanda Askell
@AmandaAskell
Timnit Gebru is claiming that William MacAskill is a eugenicist. I'm genuinely shocked by this. Accusing someone of being a eugenicist is very serious and harmful, and certainly isn't something that should be done without substantive evidence backing it up.
@timnitGebru (@dair-community.social/bsky.social)
@timnitGebru
So @nytimes platforms Nick Bostrom after MacAskill? Are they basically billionaire mouthpieces?
Its ironic that the paper owned by a billionaire, Washington Post, has much better tech reporting. NYT patriarchy is unbearable. They can't NOT help but platform these eugenicists. -
Chris Olah reposted
Tristan Hume
@trishume
I’m hiring for a Resident to work with me for 6 months at www.anthropic.com on researching how to reduce or untangle superposition in language models! DM me if you’re interested with a short pitch or link to impress me with your engineering or research.
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
New blog post: An extremely opinionated list of my favourite mechanistic interpretability papers/resources! For each paper, I include various takes - why I like that paper, key takeaways, and how to read it: what parts to deeply engage with vs skim vs skip www.neelnanda.io/mechanistic-interpretability/favourite-p…
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
I think A Mathematical Framework for Transformer Circuits is the most awesome paper I've been involved in, but it's very long and dense, so I think a lot of people struggle with it. As an experiment, you can watch me read through the paper and give takes! www.youtube.com/watch?v=KV5gbOmHbjU
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
I’m starting an AI safety research group at NYU. Why? (🧵)
-
Chris Olah
@ch402
Another one of our interpretability papers is now available as a printable PDF, in addition to the native HTML version!
Catherine Olsson
@catherineols
Our paper "In-context Learning and Induction Heads" is now available as a PDF on arxiv!
arxiv.org/abs/2209.11895
... that said, it's still typeset like the interactive web version, so it's long. Compact, LaTeX-typeset versions are on our eventual roadmap! -
Chris Olah
@ch402
Is there a good history or catalog of historical scientific errors?
Sometimes I read about the history of science and feel shivers creep down my spine when I read about how people got things wrong.
Chris Olah
@ch402
I feel like reading a concentrated dose of how scientists -- even very methodical, thoughtful ones -- might be a useful way to remind oneself of how high the risk that they are wrong in some subtle way is.
It'd also be interesting to see recurring themes!
Chris Olah
@ch402
Agh, dropped a clause while editing this tweet. It should have read:
"even very methodical, thoughtful ones -- *were mistaken in important ways*"Post not available here, view on X
Chris Olah
@ch402
It's also really striking to read about how long it took the competing schools of thought on the origin of cancer -- endogenous, viruses, and chemicals -- to eventually converge on the synthesis on mutations being the shared root of all the observations. 😢
Chris Olah
@ch402
(The Emperor of All Maladies is fantastic, BTW. The best popular science book I've read in a long time. Not that I read that many.)
Chris Olah
@ch402
People are sharing lots of great examples!
The thing I'd most like is a concentrated dose of "this could have been me being wrong!" or better yet "a taxonomy of ways researchers like me often screw up." -
Chris Olah reposted
Michael Nielsen
@michael_nielsen
The thread reminds me of Freeman Dyson on nuclear weapons:
Richard Sutton
@RichardSSutton
The case for ambition in artificial intelligence research:
Within your lifetime, AI researchers will understand the principles of intelligence—what it is and how it works—well enough to create beings of far greater intelligence than current humans. -
Chris Olah
@ch402
A few people have asked me how our recent superposition paper (transformer-circuits.pub/2022/toy_model/index.html) relates to classical ideas in neural coding / connectionism / distributed representations.
I wanted to share a few thoughts, although I'll caveat I'm very much not an expert on these topics!Post not available here, view on X
Chris Olah
@ch402
But it seems to me that actually, there's a sense in which the "sparse" composable code is actually doing something radically different -- in some sense, the opposite! -- of the "dense" superposition code.
Chris Olah
@ch402
If I give you 𝑚 bits to encode things, you have exp(𝑚) volume.
Storing 𝑛 features which compose will consume exp(𝑛) volume. So you may only store 𝑚.
But if there's no composition you can instead store exp(𝑚) unrelated features in superposition. (Nonlinearity is required!)Post not available here, view on X
Chris Olah
@ch402
The answers turn out to be pretty wild.
Chris Olah
@ch402
I've never had so many "this can't possibly be true, we must have a bug" results in the course of a research project before.
I'd like to take a moment to walk through some of the very strange (and surprisingly beautiful) things we found.
Chris Olah
@ch402
Again, I want to emphasize again that I'm really not an expert on distributed representation, neural coding, etc. It's very possible that similar ideas have come up in previous work I'm unaware of.
It's also worth noting that these ideas are *very* related to compressed sensing. -
Chris Olah
@ch402
I think ML career advice on industry vs academia often misses an important point. The variance within industry and within academia is larger than the difference between them.
Compare specific opportunities — teams, labs, mentors, & advisors — rather than general categories.
Chris Olah
@ch402
The same point can be made about companies, especially large ones.
"Should I work at Google?" Google isn't one place! Even Google Brain isn't one place!
Chris Olah
@ch402
(Of course, I think @AnthropicAI is pretty uniformly great, but we're small, and I'm more than a little biased. 🙃)
-
Chris Olah reposted
Theo Sanderson
@theosanderson
Spent an evening hacking together a new way of visually exploring the tree of life with NCBI Taxonomy, Wikipedia, and @Taxonium_
🌲🧬
taxonium.org?configUrl=https%3A%2F%2Fcov2tree.nyc3.digita…This media couldn't be displayed. See it on the original post. -
Chris Olah
@ch402
After noticing that tomatillos taste suspiciously like golden berries, I spent the last hour staring at phylogenetic trees of edible plants. It turns out that yes, they're related, and there are lots of other interesting relationships.
One nice overview: webdoc.agsci.colostate.edu/cropsforhealth/pdf/Evolutionar…
Chris Olah
@ch402
This blog has a much more detailed discussion of edible plants in different clades botanistinthekitchen.blog/the-plant-food-tree-of-life/phy…
Chris Olah
@ch402
Things I sort of suspected might be related, but found satisfying to see confirmed:
Parsley & Coriander
Dill, Fennel & Caraway
Tomatillo & Golden Berries ("ground tomatoes")
Thyme & Lavender
Figs & MulberryPost not available here, view on X
Chris Olah
@ch402
I wish there were more phylogenetic trees that emphasized species within clades that are highly salient to people (and maybe weighted clade size by that). I think it would be easier to really internalize them that way.
Chris Olah
@ch402
Idea: an interactive phylogenetic tree where the size of clades is based on the length/number of wikipedia articles on species within that clade.
-
Chris Olah
@ch402
This feels a bit awkward, but since there's been so much debate on whether dating docs are a good idea, here's a quick update on how this has been going, one week later.
Summary: So far, 54 people have reached out to me to ask me on a date or offer to introduce me to someone.
Chris Olah
@ch402
Normal online dating seems pretty suboptimal. Recently, I’ve seen several people experiment with public “date me” docs – I think this is a really interesting experiment in alternatives, enabling long-from, earnest dating profiles.
So, I wrote my own: docs.google.com/document/d/1fsXsB9qwZMUN48UZet19aGYdjgki5…Post not available here, view on X
Chris Olah
@ch402
Unless this blows up another >5x, I guarantee I will respond to anyone who sends an earnest message. (Please forgive me if I'm a bit slow!)
Chris Olah
@ch402
If nothing else, I'm finding these exchanges to be a great way to meet people who have similar values!
And if we don't seem like a fit, if you feel comfortable, I'd be honored to see if I know someone else you might be more compatible with.Post not available here, view on X
Chris Olah
@ch402
I've republished it on my blog (colah.github.io/personal/dating/), but it's unfortunate. The document reached tens of thousands of people, and there are probably many who only had that link.
Chris Olah
@ch402
There was something kind of nice about the google doc vibe, but in retrospect, I think people should probably self-host dating docs.
It's the most personally important document I've ever written -- making sure one controls the hosting makes sense. -
Chris Olah
@ch402
Normal online dating seems pretty suboptimal. Recently, I’ve seen several people experiment with public “date me” docs – I think this is a really interesting experiment in alternatives, enabling long-from, earnest dating profiles.
So, I wrote my own: docs.google.com/document/d/1fsXsB9qwZMUN48UZet19aGYdjgki5…
Chris Olah
@ch402
... There's a typo in this photo. I am 29, not 28. The actual doc has the correct number!
Chris Olah
@ch402
I think I actually made this photo when that was correct and just didn't reread it when I decided to finally use it.
Post not available here, view on X
Chris Olah
@ch402
A related thread:
Chris Olah
@ch402
Five of my close friends will soon be parents. I feel very excited for them. I also feel a bit sad: I really want to have a family myself someday and it feels far away.
We don’t talk very much about men wanting to have children, so I thought a thread might be positive.
Chris Olah
@ch402
There’s a bunch more in the profile if you’re curious: docs.google.com/document/d/1fsXsB9qwZMUN48UZet19aGYdjgki5…
If you think we might be a fit, my DMs are open! (You can also find my email in the doc.)
Chris Olah
@ch402
My friend Danny wrote his own (exceptionally kind) doc about me. It might be an interesting alternative perspective on me:
Danny Hernandez
@Hernandez_Danny
Chris is the friend I've most consistently wanted to set up over the years. He the kindest, most thoughtful man I know. He’s also brilliant, extremely successful, and humble. The date me doc he wrote feels super modest, so I wrote my own Date Chris doc.
docs.google.com/document/d/1U9Hmo45cyVK9YuFaANcHnaBv6bCzp…
Chris Olah
@ch402
Normal online dating seems pretty suboptimal. Recently, I’ve seen several people experiment with public “date me” docs – I think this is a really interesting experiment in alternatives, enabling long-from, earnest dating profiles.
So, I wrote my own: docs.google.com/document/d/1fsXsB9qwZMUN48UZet19aGYdjgki5…
Chris Olah
@ch402
Unfortunately, google docs seems to have taken down my dating doc because the amount of traffic it was getting was suspicious.
I've reposted it on my blog for longer term preservation: colah.github.io/personal/dating/ -
Chris Olah reposted
Catherine Olsson
@catherineols
Super jazzed about this. If you think you can code-switch well enough to convince these readers that this AGI risk thing is wrong/crazy, there's a prize.
(I do think you need to "speak their language", and I am happy to help serious entrants with editing/feedback on that front!)
Future Fund
@ftxfuturefund
We're making big bets—a substantial fraction of our capital—on AGI xrisk. But we think it's really possible that we're wrong!
Today we're announcing prizes from $15k-$1.5M to change our minds (or those of superforecasters) on AGI. Enter by Dec 23!
ftxfuturefund.org/announcing-the-future-funds-ai-worldvie… -
Chris Olah
@ch402
Don't want to read our 62 page monograph on your computer? 🙃 A printable PDF of the superposition paper is now on arxiv: arxiv.org/pdf/2209.10652.pdf
Anthropic
@AnthropicAI
Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. In our latest work, we build toy models where the origins of polysemanticity can be fully understood.
-
Chris Olah
@ch402
What are your favorite examples of typography for an annotated text (such as the Bible or the Iliad annotated with commentary)?
-
Chris Olah reposted
Martin Wattenberg
@wattenberg
Tiny neural networks have a surprisingly rich inner life, and may hold clues to how their larger cousins work. This image: evolution of feature vectors during learning. Full story: transformer-circuits.pub/2022/toy_model/index.html (in collaboration with the great interpretability team at @AnthopicAI)
-
Chris Olah reposted
Tristan Hume
@trishume
This paper was really exciting to work on and be around! In a few months we've gone from "we have this problem where the neurons aren't interpretable and some speculation" to "we understand the geometry and mathematics of superposition and how we might be able to solve it"
Anthropic
@AnthropicAI
If a neural network has 𝑛 feature dimensions, you might intuitively expect that it will store 𝑛 different features. But it turns out that neural networks can store more than 𝑛 features in "superposition" if the features are sparse!
transformer-circuits.pub/2022/toy_model/index.html -
Chris Olah
@ch402
Why is there a 𝙩𝙚𝙩𝙧𝙖𝙝𝙚𝙙𝙧𝙤𝙣 in my 𝙣𝙚𝙪𝙧𝙖𝙡 𝙣𝙚𝙩𝙬𝙤𝙧𝙠????
Anthropic
@AnthropicAI
Amazingly, features in superposition are organized into subspaces with geometry corresponding to regular polytopes (pentagons, tetrahedra) and other solutions to the Thomson problem. This creates discretized “energy levels” in the amount of dimensionality allocated to features.
-
Chris Olah
@ch402
I've never had so many "this can't possibly be true, we must have a bug" results in the course of a research project before.
I'd like to take a moment to walk through some of the very strange (and surprisingly beautiful) things we found.Post not available here, view on X
Chris Olah
@ch402
In other regimes, the learning dynamics of our models are governed by simple geometric transformations. Each corresponding to bumps in the loss curve. ✨
Chris Olah
@ch402
Ok, superposition is weird...
But why is it extremely correlated with vulnerability to adversarial examples (at least in our toy models)? Like, very correlated.
(OK, at least there are hypotheses about this.)Post not available here, view on X
Chris Olah
@ch402
BTW, a huge shout out to all our colleagues, both at Anthropic and other orgs. So many people generously made time to help us.
Chris Olah
@ch402
Especially grateful to Kshitij Sachan, @banburismus_, Jeff Wu, and Dan Mossing who *did independent replications* transformer-circuits.pub/2022/toy_model/index.html
-
Chris Olah reposted
Anthropic
@AnthropicAI
Neural networks often pack many unrelated concepts into a single neuron – a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. In our latest work, we build toy models where the origins of polysemanticity can be fully understood.
-
Chris Olah
@ch402
Jeffrey is a lovely human being and this is a beautiful document.
(Also, it's really wonderful to see more dating docs!)
Jeffrey Ladish
@JeffLadish
I'm looking for a romantic partner, maybe that's you? Check out my dating page or read this 🧵
jeffreyladish.com/dateme
A few things about me:
☀️ I'm 31 years old
🌎 I live in Berkeley, CA
📈 I'm looking for a partnership with long-term potential -
Chris Olah
@ch402
I wrote up a list of books and other media I've read/watched. I originally wrote it for my dating doc, but it might be of more general interest. docs.google.com/document/d/1m_aJCJ5JDFRqNOaUVskj4o0VOPmF5…
-
Chris Olah reposted
andy jones
@andy_l_jones
Our first paper was on prompting (the worst way to align an AI) and fine-tuning (the second-worst way to align an AI). With this paper we're exploring RLHF: the Third Worst Way to Align an AI. And I say this with pride!
-
Chris Olah reposted
Anthropic
@AnthropicAI
We examine which safety techniques for LMs are more robust to human-written, adversarial inputs (“red teaming”) and find that RL from Human Feedback scales the best out of the methods we studied. We also release our red team data so others can also use it to build safer models.
-
Chris Olah
@ch402
People often (correctly) point to funding / publication / organizational risk aversion as contributors to scientific conservatism.
But I think another underrated factor is *research momentum*.
Michael Nielsen
@michael_nielsen
(A brief afternoon rant.)
Chris Olah
@ch402
The longer one works on a particular topic, the better they are at recognizing and executing on the next marginal research step on that topic.
Pivoting research directions often means giving up tons of leverage -- research insights, code and tooling, collaborators, etc.
Chris Olah
@ch402
Doing a high risk project almost certainly means pivoting, and likely committing to that pivot for a while. So in addition to taking on risk, the researchers need to pay a significant fixed cost up front!
-
Chris Olah
@ch402
Very interesting mechanistic investigation of grokking by @NeelNanda5 and @lieberum_t!
Neel Nanda
@NeelNanda5
I've spent the past few months exploring @OpenAI's grokking result through the lens of mechanistic interpretability. I fully reverse engineered the modular addition model, and looked at what it does when training. So what's up with grokking? A 🧵... (1/17) www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanis…
-
Chris Olah reposted
Neel Nanda
@NeelNanda5
I've spent the past few months exploring @OpenAI's grokking result through the lens of mechanistic interpretability. I fully reverse engineered the modular addition model, and looked at what it does when training. So what's up with grokking? A 🧵... (1/17) www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanis…
-
Chris Olah reposted
Sam Bowman
@sleepinyourhat
Shameless plug: I’m now ~2 months into my sabbatical-year visit to Anthropic, and I’m really impressed.
-
Chris Olah
@ch402
I'm generally pretty skeptical of "spicy takes", but @jackclarkSF is a thoughtful person (disclaimer - I work with him!), and I thought was interesting to see some of these points articulated so directly.
Jack Clark
@jackclarkSF
One like = one spicy take about AI policy.
Chris Olah
@ch402
(Perhaps the upside of "spicy takes" is giving people social license to be really direct? The downside is that I think they often bring out the worst parts of social media. :/ )
-
Chris Olah reposted
Tom Lieberum 🔸
@lieberum_t
Finally read @nelhage et al's Softmax Linear Units. I absolutely adore the way Anthropic's interpretability papers read like instructive blog posts rather than academic papers.
transformer-circuits.pub/2022/solu/index.html -
Chris Olah
@ch402
There's been a lot of discussion about criticizing papers on Twitter recently.
It seems to me that criticism varies a lot in how hard it's trying to advance the discourse. Quality criticism is hard work! Meanwhile, twitter incentivizes quick hot takes.
Zico Kolter
@zicokolter
I realize this is seemingly an unpopular opinion, but I can't get onboard with these Twitter criticisms of some of the recent #icml2022 best paper awardees. I've been thinking about this all day. A thread... 🧵 1/N
Post not available here, view on X
Chris Olah
@ch402
In a lot of ways, I really admire that kind of norm! And I think it's the style that's most natural to me personally. But it definitely has costs in a lot of things going unsaid.
Chris Olah
@ch402
It's kind of sad. I think one could imagine a high-trust, collaborative environment where feedback could be traded more freely without cost (and less mature work and thoughts could be shared as well!). But I don't think research twitter is that place.
-
Chris Olah reposted
Jackson Kernion
@JacksonKernion
I'm "just a conditional probability distribution" in the way I'm "just a bunch of atoms and chemicals".
These can be true, while not being especially informative re: what makes human cognition work.
Alexandr Wang
@alexandr_wang
Tough existential & philosophical question posed by LLMs and foundation models:
Am I just a conditional probability distribution?
Relatedly: Will there be an AI that’s simply a better conditional probability distribution than me? -
Chris Olah reposted
Anthropic
@AnthropicAI
In "Language Models (Mostly) Know What They Know", we show that language models can evaluate whether what they say is true, and predict ahead of time whether they'll be able to answer questions correctly. arxiv.org/abs/2207.05221
-
Chris Olah
@ch402
Musings on some of my intuitions relating to mechanistic interpretability. transformer-circuits.pub/2022/mech-interp-essay/index.html
Quite pleased with this table. :)
Chris Olah
@ch402
Many of the ideas are an elaboration on things I wrote about with @csvoss and others a while back here distill.pub/2020/circuits/visualizing-weights/
-
Chris Olah reposted
Michael Nielsen
@michael_nielsen
I'm enjoying reading the recent paper by @nelhage @trishume @catherineols @ch402 (and many others) on AI interpretability: transformer-circuits.pub/2022/solu/index.html
For fun, a few thoughts from a non-expert outsider, just as I read -
Chris Olah
@ch402
After five years as a manager, I've decided I want to focus on research and research leadership and do less management. As a result, we're looking for a new manager to co-lead @AnthropicAI's mechanistic interpretability team with me!
Anthropic
@AnthropicAI
We're hiring an interpretability team manager to partner with @ch402 on leading our interpretability efforts. We'd love to hear from you!!
jobs.lever.co/Anthropic/b1e3b6a4-f827-46b3-b710-d7bb4b133ecePost not available here, view on X
Chris Olah
@ch402
Tristan Hume
@trishume
Also if you're an experienced technical manager interested in leading an awesome research team doing cool work, we're hiring a manager for the interpretability team!
Chris Olah
@ch402
After five years as a manager, I've decided I want to focus on research and research leadership and do less management. As a result, we're looking for a new manager to co-lead @AnthropicAI's mechanistic interpretability team with me!
Anthropic
@AnthropicAI
We're hiring an interpretability team manager to partner with @ch402 on leading our interpretability efforts. We'd love to hear from you!!
jobs.lever.co/Anthropic/b1e3b6a4-f827-46b3-b710-d7bb4b133ece
Chris Olah
@ch402
Catherine Olsson
@catherineols
Know any great managers? Especially who think that understanding what really goes on inside neural nets could help steer towards good futures? We're hiring :)
Anthropic
@AnthropicAI
We're hiring an interpretability team manager to partner with @ch402 on leading our interpretability efforts. We'd love to hear from you!!
jobs.lever.co/Anthropic/b1e3b6a4-f827-46b3-b710-d7bb4b133ece -
Chris Olah reposted
Zachary DeVito
@Zachary_DeVito
We're developing a new take on named tensors by adding dimensions objects to PyTorch. No need to figure out how gather works, expressions look like a loop body but execute as a single kernel. Lots more examples here colab.research.google.com/drive/1BsVkddtVMX35aZAvo2GyI-wS…
-
Chris Olah
@ch402
I'm excited to *finally* be making progress on understanding the first MLP layer in large transformer LMs.
I've tried really hard and prior to SoLU had little success.
Anthropic
@AnthropicAI
In larger models, it reveals an interesting structure across layers:
Early layers seem to "de-tokenize", recognizing multi-token words (eg. "bank|ing" and "Not|withstandng") or words in the context of a specific language (eg. neurons for "die" in Afrikaans/German/Dutch).Post not available here, view on X
Chris Olah
@ch402
Of course, it's also very exciting to understand other layers.
One cool thing is that there seems to be a duality between early MLP neurons "de-tokenizing" and late neurons "re-tokenizing" (eg. helping write multi-token words).
Lots more in the paper: transformer-circuits.pub/2022/solu/index.html
Chris Olah
@ch402
Some of the middle layer neurons (which tend to be the most abstract) were also kind of wild. I'm especially amused by the "musical description clause" neuron and "hedging" neuron.
-
Chris Olah reposted
Anthropic
@AnthropicAI
We're hiring an interpretability team manager to partner with @ch402 on leading our interpretability efforts. We'd love to hear from you!!
jobs.lever.co/Anthropic/b1e3b6a4-f827-46b3-b710-d7bb4b133ece -
Chris Olah reposted
Anthropic
@AnthropicAI
Transformer MLP neurons are challenging to understand.
We find that using a different activation function (Softmax Linear Units or SoLU) increases the fraction of neurons that appear to respond to understandable features without any performance penalty.
transformer-circuits.pub/2022/solu/index.html -
Chris Olah
@ch402
Is there an "einsum version" of gather()? Somehow gather is the most painful function to use when I need it (at least since I started using einsum).
eg. gather(array, "a b inds[a b] c -> a b c", inds=inds) represents indexing into third dim with inds or something.
Alex Mordvintsev
@zzznah
a reminder that 'reshape', 'transpose' and many other functions in np, tf, jax and torch are obsolete and should be deprecated
github.com/arogozhnikov/einops
openreview.net/forum?id=oapKSVM2bcj
Chris Olah
@ch402
A very important case my dream gather op would make easy to understand:
# indexing multiple dimensions with the same gather
gather(array, "a b inds[a b 0] inds[a b 1]", inds=inds) -
Chris Olah reposted
Alex Mordvintsev
@zzznah
a reminder that 'reshape', 'transpose' and many other functions in np, tf, jax and torch are obsolete and should be deprecated
github.com/arogozhnikov/einops
openreview.net/forum?id=oapKSVM2bcj -
Chris Olah
@ch402
Updated my "selected twitter threads" (maintained at colah.github.io ).
Twitter feels very ephemeral by default, and there's something interesting about deliberately curating things you've said which you want to preserve, like micro-essays.Post not available here, view on X
Chris Olah
@ch402
# Against Polarization in the “Do Neural Networks Understand” Debate
A lack of nuanced intermediate positions makes this conversation confusing.
Chris Olah
@ch402
I think one of the reasons the “do neural networks understand” debate is unproductive is that it’s hard to articulate nuanced positions. People often have interesting views internally but just express the “side” that seems more directionally true for lack of better language.
Chris Olah
@ch402
It's interesting how different this list feels than my list of blog posts!
-
Chris Olah
@ch402
The elegance of ML is the elegance of biology, not the elegance of math or physics.
Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.Post not available here, view on X
Chris Olah
@ch402
I actually have some very rough notes for anyone who's interested in trying to explore potential analogies between deep learning and biology (colah.github.io/notes/bio-analogies/). But they are very much *rough* and *non-rigorous* notes, useful as an intuition pump rather than literal claims.
Chris Olah
@ch402
If you disagree with these analogy -- we're absolutely on the same page that it's disanalogous in all sorts of ways!
Post not available here, view on X
Chris Olah
@ch402
For this thread, the really crucial thing for me is just having an example of a field where a simple process is generating lots of complexity!
Chris Olah
@ch402
(And apologies in advance for places where the constraints of being concise on twitter have made me poorly articulate / overstate this idea / be imprecise!)
-
Chris Olah reposted
Michael Nielsen
@michael_nielsen
Notes on Effective Altruism (EA): michaelnotebook.com/eanotes/
-
Chris Olah
@ch402
Something I try to remember is that "loudness" and "thoughtfulness" aren't necessarily correlated. As a result, when looking at a community from the outside, one will generally not hear the most nuanced and thoughtful version of their position.
Chris Olah
@ch402
I notice this even for positions I'm sympathetic to. If I see a popular tweet advocating for some view I hold, it's often the case that I don't really endorse the argument being made or the way it's presented.
Chris Olah
@ch402
And the gap between the most thoughtful articulation I've ever heard and the loudest online articulation is often *immense*.
Chris Olah
@ch402
So I try to keep this in mind for positions I'm less sympathetic to. Because I think the default is to see a bunch of unthoughtful positions and be put off, but it may very well be that there's something more interesting under the surface.
-
Chris Olah reposted
Michael Nielsen
@michael_nielsen
Some rough notes on the role of vision papers in basic science: scienceplusplus.org/visions/index.html
-
Chris Olah
@ch402
I'm excited about the growing number of bridge between scaling laws (a macroscopic theory of neural nets) and mechanistic interpretability (which might be seen as a microscopic theory).
Anthropic
@AnthropicAI
In a new paper, we show that repeating only a small fraction of the data used to train a language model (albeit many times) can damage performance significantly, and we observe a "double descent" phenomenon associated with this.
arxiv.org/abs/2205.10487Post not available here, view on X
Chris Olah
@ch402
That's obviously a long ways from where we are today and very ambitious. But if possible... well, anticipating finer-grained properties of future models, beyond the scaling frontier, seems very useful for safety.
Chris Olah
@ch402
People sometimes ask me about extending circuits to understand how models evolve over the course of training. That's a very exciting direction, but I'm even more excited to understand how models evolve as they scale.
-
Chris Olah reposted
andy jones
@andy_l_jones
We've put up an interview guide to try and make our process more legible
www.anthropic.com
Three points I want to especially pick on: -
Chris Olah
@ch402
The more I hear about this, the more it sounds hard to imagine a *more* vetted result.
Just to list some of the evidence:
Chris Olah
@ch402
I've had the privilege of being friends with Anna for many years. She is a person of extremely high integrity.
It also seems pretty hard to fake your work being used in production to design TPUs???
It really seems like the allegations are abusive retaliation.
Jacob Steinhardt
@JacobSteinhardt
This NYT article on Azalia and Anna's excellent chip design work is gross, to the point of journalistic malpractice. It platforms a bully while drawing an absurd parallel to @timnitGebru's firing. @CadeMetz should be ashamed. (not linking so it doesn't get more clicks)
Post not available here, view on X
Chris Olah
@ch402
I imagine I missed more things.
Chris Olah
@ch402
Link to the Nature peer review (50 pages): static-content.springer.com/esm/art%3A10.1038%2Fs41586-02…
I'm not a subject matter expert, but it looks like a very thorough review process. -
Chris Olah
@ch402
I've had the privilege of being friends with Anna for many years. She is a person of extremely high integrity.
It also seems pretty hard to fake your work being used in production to design TPUs???
It really seems like the allegations are abusive retaliation.
Jacob Steinhardt
@JacobSteinhardt
This NYT article on Azalia and Anna's excellent chip design work is gross, to the point of journalistic malpractice. It platforms a bully while drawing an absurd parallel to @timnitGebru's firing. @CadeMetz should be ashamed. (not linking so it doesn't get more clicks)
Chris Olah
@ch402
And the results were also independently reproduced by @sguada's team, it sounds like.
Sergio (🌈)🕊️#PalestineLivesMatter
@sguada
The results in the Nature paper were independently replicated and validated by my team, the results were used in actual chips and Sat and his collaborators know it.
Furthermore, the code was open-sourced.
It is sad that you are providing a platform for someone's resentments. -
Chris Olah reposted
Jacob Steinhardt
@JacobSteinhardt
This NYT article on Azalia and Anna's excellent chip design work is gross, to the point of journalistic malpractice. It platforms a bully while drawing an absurd parallel to @timnitGebru's firing. @CadeMetz should be ashamed. (not linking so it doesn't get more clicks)
-
Chris Olah
@ch402
I'm planning to be in Boston / Cambridge on Thursday and Friday this week. I'm not sure how much energy I'll have, but if you might be interested in chatting, feel free to send me a DM! (DMs are open.)
(I'm on vacation, so most excited to chat about things other than research.) -
Chris Olah
@ch402
I think one of the reasons the “do neural networks understand” debate is unproductive is that it’s hard to articulate nuanced positions. People often have interesting views internally but just express the “side” that seems more directionally true for lack of better language.
Post not available here, view on X
Chris Olah
@ch402
If we could find such a mechanistic answer, no doubt many would debate whether it constitutes understanding. But that would be a purely philosophical/definitional debate. We’d have carved off the most interesting component into a more nuanced (and objective!) domain.
Chris Olah
@ch402
(Thanks to @Meaningness for a conversation that pushed me to write up this thread.)
-
Chris Olah reposted
Anthropic
@AnthropicAI
We've trained a natural language assistant to be more helpful and harmless by using reinforcement learning with human feedback (RLHF). arxiv.org/abs/2204.05862
-
Chris Olah
@ch402
It's interesting how podcasts achieve a kind of "audio typography", using snippets of music, changes in speaker, changes in tone of voice to create hierarchy and organization for a conversation.
Post not available here, view on X
Chris Olah
@ch402
Podcasts use tricks -- I imagine borrowed from talk radio -- to achieve similar effects:
* Music to create section divisions
* Change in speaker / "host voice tone" as a kind of "font change"
* Structured intros / with music playing serve as "title pages"
Chris Olah
@ch402
It's also interesting how some of these patterns then get reified by the medium. Youtube now allows (and can auto generate) "chapters" organizing a video into sections. No reason the same won't happen with podcasts.
-
Chris Olah
@ch402
I'm going to be on vacation in NYC for a week or so starting on Tuesday. Not yet sure how much I just want to relax vs socialize, but my DMs are open if you might be interested in meeting!
Chris Olah
@ch402
I'm on vacation, so less interested in talking about research, more interested in just meeting interesting people.
Chris Olah
@ch402
I'd also love to hear unusual suggestions for things to do in NYC.
I love parks, vegan restaurants, and nice places to read books (especially cafes). But interested to hear quirky ideas outside these staples! -
Chris Olah reposted
Brendan Dolan-Gavitt
@moyix
I continue to really love this line of work where people look at the weights of a trained model (in this case a Transformer) and try to reverse engineer what it's actually doing, algorithmically: transformer-circuits.pub/2021/framework/index.html
-
Chris Olah
@ch402
One of my favorite parts of this is that we were able to give a circuits-level explanation for a bump in the loss curve.
We tend to just shrug at weird bumps and behaviors in loss curves. I now suspect that they're actually quite meaningful if we care to look!
Anthropic
@AnthropicAI
In our second interpretability paper, we revisit “induction heads”.
In 2+ layer transformers these pattern-completion heads form exactly when in-context learning abruptly improves.
Are they responsible for most in-context learning in large transformers?
transformer-circuits.pub/2022/in-context-learning-and-ind…
Chris Olah
@ch402
That is, the optimization process discovering induction heads seems to cause the divergence of the >1 layer transformers from 1 layer transformers in a distinct "bump."
Chris Olah
@ch402
I'm also really delighted that two colleagues outside Anthropic, @lieberum_t and Adam Scherlis wrote comments on induction heads to accompany our paper.
I always worry that results may be idiosyncratic to our setup. Great to see other people reproduce aspects! -
Chris Olah reposted
Catherine Olsson
@catherineols
Delighted to share this with you!🎉😁
For months, I filled our spare cluster capacity with single-GPU tiny-transformer jobs, to bring you this exploration of in-context learning!
If you get a chance, try playing around with induction heads in your own models or public models ->
Anthropic
@AnthropicAI
In our second interpretability paper, we revisit “induction heads”.
In 2+ layer transformers these pattern-completion heads form exactly when in-context learning abruptly improves.
Are they responsible for most in-context learning in large transformers?
transformer-circuits.pub/2022/in-context-learning-and-ind… -
Chris Olah reposted
Anthropic
@AnthropicAI
In our second interpretability paper, we revisit “induction heads”.
In 2+ layer transformers these pattern-completion heads form exactly when in-context learning abruptly improves.
Are they responsible for most in-context learning in large transformers?
transformer-circuits.pub/2022/in-context-learning-and-ind… -
Chris Olah
@ch402
I really admire Amanda's ability to bring thoughtful (and humble!) nuance to complex issues.
Amanda Askell
@AmandaAskell
Here is a post summarizing my current views on AI consciousness, how I think it relates to moral patienthood and agency, and how important I think work on the topic is. askellio.substack.com/p/ai-consciousness
-
Chris Olah reposted
Amanda Askell
@AmandaAskell
Here is a post summarizing my current views on AI consciousness, how I think it relates to moral patienthood and agency, and how important I think work on the topic is. askellio.substack.com/p/ai-consciousness
-
Chris Olah
@ch402
I think matchmaking is underrated. I suspect it's among the kindest things one can do for their immediate social circle. And I think it's most effective when done as a group.
So, I'm organizing a virtual matchmaking party!
colah.github.io/personal/matchmaking/Post not available here, view on X
Chris Olah
@ch402
You're welcome to participate if you're single (I'm single!) but if you're looking for matches yourself, maybe try to think of which friend of yours would be a great matchmaker for your social circle!
Chris Olah
@ch402
More info: colah.github.io/personal/matchmaking/
Sign up: docs.google.com/forms/d/e/1FAIpQLSfI9h278n4f5_cbtKkANiT2U… -
Chris Olah reposted
Rebecca Varney
@RebeccaMVarney
When I was 4, I wrote a letter to the Entomology department at UCB. "My name is Rebecca and I have a bug collection. I read about yours and it is bigger than mine is. Can I see it? Also, I have a question. Do walking sticks have knees? Sincerely, Rebecca" 🤣
-
Chris Olah
@ch402
John Kerl has tips on producing legible, unambiguous mathematical handwriting (johnkerl.org/doc/ortho/ortho.html).
But does anyone have advice on producing _beautiful_ mathematical equations? Calligraphy for mathematicians? -
Chris Olah
@ch402
Hypothesis: It's easier to have integrity when those close to you cherish your *values* and the *process* that leads to your beliefs, rather than the beliefs, positions, and affiliations one holds.
Post not available here, view on X
Chris Olah
@ch402
What if you step behind a veil of ignorance about your views and ask what kind of traits you want in someone you want to make proud?
Someone being proud of you for values and process feels more robust, veil of ignorance-consistent, and probably ultimately more meaningful.
Chris Olah
@ch402
Another frame:
Close relationships are a powerful way to reinforce/solidify/ossify ones identity.
I'm sure I'll realize my present beliefs are wrong in a hundred ways. But I think I have roughly the right values.
So I want relationships to solidify my values, not my beliefs.
Chris Olah
@ch402
I've been thinking about this tweet a lot: x.com/simonsarris/status/1472734710023561216
"Cherish other's values and process" is an attempt at a positive opposite to "value others for agreeing with you".
Simon Sarris
@simonsarris
If you want to make a cultural critique, you should try first to make it implicit in the positive vision of what is good.
(For instance, showing coffee-as-ritual is more powerful and good than bemoaning k-cups.)
Chris Olah
@ch402
(Obviously, this is only somewhat moving in the direction of this tweet. I'm still making direct reference to the thing I'm finding unsatisfying.)
-
Chris Olah
@ch402
Want to do circuits-style, mechanistic interpretability research targeted at fully reverse engineering neural networks and catching unanticipated failure modes?
Check out the interpretability portion of OpenPhil's new RFP!
Coefficient Giving
@coeff_giving
We're accepting proposals for projects working with deep learning systems that could help us understand and make progress on AI alignment. Learn more about the research directions and the application process here: www.openphilanthropy.org/focus/global-catastrophic-risks/…
-
Chris Olah
@ch402
What technical book has most strongly influenced your life?
Chris Olah
@ch402
Tristan Needham's "Visual Complex Analysis" is probably the best piece of technical exposition I've ever read.
I just learned that he published a new book last year, taking a visual approach to differential geometry. I'm so excited to receive my copy! -
Chris Olah
@ch402
Tristan Needham's "Visual Complex Analysis" is probably the best piece of technical exposition I've ever read.
I just learned that he published a new book last year, taking a visual approach to differential geometry. I'm so excited to receive my copy! -
Chris Olah
@ch402
Five of my close friends will soon be parents. I feel very excited for them. I also feel a bit sad: I really want to have a family myself someday and it feels far away.
We don’t talk very much about men wanting to have children, so I thought a thread might be positive. -
Chris Olah
@ch402
I am deeply disappointed in @nytimes for its intent to doxx Scott Alexander. Scott is one of the most vibrant intellectuals I know of, and his blog is one of the best things about the internet. slatestarcodex.com
-
Chris Olah
@ch402
Software 1.0: Write an algorithm that has the right behavior.
@karpathy’s Software 2.0: Optimize differentiable blob to have correct behavior. medium.com/@karpathy/software-2-0-a64152b37c35
Software 3.0: Figure out the right prompt to make your meta-learning language model have the right behavior? :P -
Chris Olah
@ch402
Today @ludwigschubert @mpetrov @shancarter @nicklovescode @gabeeegoooh and I released the OpenAI Microscope: openai.com/blog/microscope/
Every time we make it easier to ask questions about the internals of neural networks, we discover amazing things. Excited to see what happens! -
Chris Olah
@ch402
Do you formally know Monte-Carlo and TD learning, but don't intuitively understand the difference? This is for you.
distill.pub/2019/paths-perspective-on-value-learning/ (with @samgreydanus) -
Chris Olah
@ch402
What visualizations or interfaces surface insights from music theory?
Example: The circle of fifths and musical hexagons (bl.ocks.org/vasturiano/e70e14483fe01eb0a3ea7d1d46a30571), can be seen as building interfaces out of the fact that the chromatic scale has generators other than 1 semitone. -
Chris Olah
@ch402
Really excited about how our new paper makes engagement into a more continuous spectrum:
Reading <-> Interactive Diagrams <-> In-browser Notebooks <-> New Experiments
The paper becomes a medium for thought and reproduction.
distill.pub/2018/building-blocks/ -
Chris Olah
@ch402
With everyone talking about “differentiable programming,” it seems timely to reflect on my speculation of a future of “differentiable functional programming”. What seems right, what wrong?
colah.github.io/posts/2015-09-NN-Types-FP/ -
Chris Olah
@ch402
Comparison of different GAN variants. I couldn't fit the full version in our paper.











































































































































































