Politicians, athletes, journalists, NGOs, companies... They all use
corporation-owned social networks to communicate with us. Citizens
shouldn't have to hand over their data to companies to read content
published by public entities. Public Feeds changes that by making posts
of popular accounts freely available.
Read more.
Kelsey Piper
Kelsey Piper (born c. 1992) is an American journalist who is a staff writer at The Argument. She was previously a staff writer at Vox, where she wrote for the column Future Perfect, which covered a variety of topics from an effective altruism perspective. While attending Stanford University, she founded and ran the Stanford Effective Altruism student organization. More on Wikipedia
This pilot is an ambitious first step towards the goal of obtaining those strong guarantees, but its access and disclosure rights don't yet meet our bar for full embedded evaluations.
I see we are in yet another round of: 1. Someone claims that pangram discriminates against a marginalized group 2. Someone actually checks by comparing writing from that group pre-2022 to post-2022 3. There isn't bias and people in that group just use AI and say they don't.
“No AI ever” is obviously a lost battle; if we don’t want to be reading fully generated novels from HarperCollins in three years it would be wisest to strategically choose where to draw the lines now, and e.g. “no AI for debugging” is not a line that will hold
As people in the Bay have started to expect RSI to arrive sooner and sooner, some of them have basically given up on legislation or bilateral treaties, because they figure it just can't happen fast enough.
I don't know if you actually need legislation to get real commitments out of these companies. But you definitely need the threat of legislation. You need them to believe that the alternative to doing this is MORE transparency and oversight.
How did we catch them? Most of their bitly.com links took a brief detour through mycreator.studio, a tracker carrying a code for the creator and campaign.
We traced this domain to the influencer marketing firm Urban Legend.
Over 1,300 campaigns appear to be run by the company.
There is nothing wrong with preferring weaker candidates in primaries because you like their health care ideas, but a movement that is structurally incapable of acknowledging a tradeoff isn’t going to be able to make a good health care system.
Are JLG and the Council serious about pulling our economy out of a ditch and attracting new companies and investment to the District? The failed vote on AVs indicates the answer is no. capitalcommonsense.com/p/commonsense-vs-degrowth-why-the-…
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way - in short, the period was so far like the present period that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.
When I'm in the Bay Area, I find it hard to escape the sense of sadness I get from the intense misery on the streets, and the dread I get from parts of the AI crowd being so deeply pessimistic, hyperbolic, and fatalistic. But a recent exception, at least for the latter vibe basin, was meeting the Foresight Institute community, as well as visiting the wise greybeards of the Internet Archive. Really refreshing to get a feel of the more optimistic, agentic, and epistemically uncorrelated crowds there. It felt a bit like the ambitious 90s tech optimism, where futurism wasn't isn't just about some assumed inevitable disempowerment. I think it's actively important to remember that the future can be crazy *and* good!
The cheapest way to live in an American city used to be renting a room in a residential hotel or rooming house and sharing a communal kitchen. Today it is illegal to open one almost anywhere, because cities set out to get rid of them.
Ezra Klein's column this morning strongly endorses the bipartisan permitting reform deal.
Key points in his argument for BAAJA:
1. "BAAJA is the largest reimagining of the federal permitting process in a generation. It would make it far easier to build new energy infrastructure of all kinds: wind and solar farms and interstate transmission lines and, yes, natural gas pipelines. It would make it easier to build much else, too: rail and transit infrastructure and affordable housing and broadband internet."
2. "The bill represents Democrats and Republicans deciding that the fight over what gets built should play out in elections and markets rather than in the bowels of administrative agencies and courts. It should be driven by what people want, in public, rather than what they can block, often in private."
3. "At its core, BAAJA would do the same thing over and over again: It would make it faster and simpler to build energy infrastructure by rolling back the opportunities for delays, lawsuits and vetoes that have proliferated in recent decades."
4. "The Democrats who won Senate seats, or look likely to win them, in Kansas, Mississippi, South Carolina, Texas and Alaska ran on an all-of-the-above energy strategy. Democrats everywhere ran on putting affordability first. The bill is an early test of whether they meant it. A Democratic Party that was saying what voters wanted to hear, but has no intention of doing what voters asked them to do, will not hold power for long."
5. "A significant backlog of energy projects is waiting to be built or hooked into our overloaded, archaic energy grid. That backlog is overwhelmingly tilted toward green energy. At the end of 2025, more than 85 percent of the energy projects seeking to get plugged into the grid were renewable sources and storage. The fossil fuel projects of the past are already built and connected; the green energy projects being built are not."
6. "Some renewable energy sources like solar and wind require more land per unit of energy than fossil fuel sources, and then they require transmission lines to move energy from, say, the deserts of the Southwest to the urban centers across the West. That makes them much more exposed to permitting risk than fossil fuels."
7. "Capriciously blocking wind power is what Trump is doing now. The bill makes it harder for Trump to sit on wind projects in the future. 'His departments have to give an answer, yes or no, and the answer has to be justified, and there’s a time frame by which they have to deliver it,' said Daniel Palken. The remedies here become severe, including one that says agency heads cannot travel more than 25 miles from their headquarters if a permit is more than 60 days delayed."
8. "If you believe that Democrats will build on the blue tsunami and take the presidency in 2028, perhaps with even larger House and Senate majorities, then the case for this bill is extremely strong. It’s a force amplifier of anything they might do on clean energy. That’s why Janet Yellen and Jared Bernstein, who led Biden’s Treasury Department and his Council of Economic Advisers, both backed the bill."
9. "If you believe there’s going to be more pressure for decarbonization by 2029, you want to begin the process of making it easier to build green energy now. If Democrats then have so much power that they can pass more radical laws to help green energy, or impose a social cost on carbon emissions, then all the better. BAAJA can be a floor, not a ceiling. But even in his scenario, you need a new floor. and I don’t think we should wait years to start to build one."
10. "Democrats have spent years saying that the green tech revolution is ready. They have spent years promising to put affordability first. And they have as good a shot as they’ve had in a generation of wielding supermajority power in 2029. I don’t dismiss the concerns any of them have about this bill. But it’s a risk worth taking to make the future they’ve promised possible."
Worth remembering that perhaps the most consequential piece of AI policy in the US to date is a completely secret federal framework that states very prominently that its voluntary, but everyone involved believes its not actually voluntary
people have little understanding of how much the federal government is already involved in the ai industry. for example if there’s a new model you need to convince the feds that new model is safe before launching it or men with proverbial guns and export control bans show up
BAAJA would make our legal system more similar to Europe's, where you can stop a project for substantive environmental reasons, but you cannot stop it on a technicality
Reducing injunctive relief for NEPA procedural errors and only allowing vacatur when illegal actions are taken or substantive environmental harm will occur would put the US in sync with German administrative law.
What SpaceX did today is the normal consequence of applying capital and engineering to any industry. You get better at it.
Musk isn't a genius, he's just a guy who can get capital from Wall Street. Maybe that's unique right now, but it's not the skill his fanboys think it is.
seems like we are (briefly?) back in the regime where prompting and having the right tools and context and examples stored means people are getting vastly different results out of the same models.
This was very true back in 2025 with o3 and that era of models, and then at some point this year it became basically false, just tell Opus 5.5 what you want, most scaffolds make it worse.
A more important question is, how do you convince people that they won’t just get Collin Burns-ed (put a bunch of time into joining + make financial sacrifices to do so, then get fired for silly political reasons) or have no impact
There is a large amount of talent inside of AI companies and other private organizations that would be willing to take big pay cuts to work in government, if government decided that it wanted to make public AI safety institutions that were credibly empowered.
"They are saying that recursive self-improvement has kicked in, despite the obvious safety problems with that. Recursive self-improvement is the most dangerous thing anybody can be working on" -- DAVID SACKS
If you break agreements and neglect deontological obligations when [you believe] the stakes are really high, then when [people know you believe] the stakes are really high, people won't want to trust or deal with you or anyone they associate with you.
It seems that the recent OpenAI firings may have been a result of false suspicion that safety-motivated lab employees violated confidentiality agreements for ideological reasons. That's in a world where (as far as I'm aware) such suspicions are completely baseless and no such violations have actually occurred. If current or former lab employees acted so as to make those fears actually justified, I think these dynamics would get MUCH worse.
yep. by the traditional "caucus" definitions the Republican moderation advantage had gone to 0, but the correct way to read this is not that "there is no moderation advantage for Republicans" — it's that "none of these Republican groups are moderate and they all vote the same".
For years, I listened to the reasonable people. I nodded along when critics called AI “a con,” and when they waved ChatGPT away as “spicy autocomplete.” I laughed in 2024, when the internet was mocking ChatGPT because it couldn’t count the r’s in “strawberry.” When I met young researchers who had devoted their careers to the danger of rogue AI, I thought: what a waste of talent.
But at some point, I started to realize something was off. The machine kept learning things it was never supposed to learn.
OpenAI recently published more than 700 papers with the solutions to hundreds of open problems in mathematics. Hugo Duminil-Copin, winner of the Fields Medal (the highest prize in mathematics), wrote: “The shock is immense. I am paralyzed.”
None of this should have happened. So I went back and looked at what else the skeptics had predicted, and it dawned on me: they had been wrong about... pretty much everything. Next, I looked at the predictions of the crazy people, and realized they’ve been eerily prescient.
Why then, you may wonder, does the world still feel pretty normal? Because new technology usually spreads more slowly than its inventors expect (it took factories decades to reorganize around electricity). But the scary scenario is not about what happens in the wider economy; it’s about what happens inside a handful of companies.
I wrote down what changed my mind, and what we can do about it:
.@JerusalemDemsas: “I wasn't that interested in pandemic preparedness policy in 2019.
Boring stuff can be deadly. 7 million people are dead, and we still haven't implemented many of the pandemic preparedness policies that would prevent that in the future. Don't wait for a Hiroshima- or a Nagasaki-type event to take AI safety seriously.” #whatsnext
OpenAI's head of alignment says that relying on future AIs to solve AI alignment is not the plan (sometimes referred to as handoff). Though OpenAI's plan instead is still forthcoming, and this very much seems like a situation where one should "trust but verify."
For people who work at Anthropic, Google, Xai, Meta and other AI companies - if your plan for alignment is also not just to handoff AI alignment to AIs (who are getting harder to monitor, and are sometimes horribly misaligned), it would be great for you to say so and share what that plan is too.
You're the president in 2029. The political leadership of Chicago and Illinois are holding an emergency press conference. They're broke. No more money for cops, teachers, trash collectors.
Polls were bad and it does a disservice to the industry when pollsters insist they were great.
*Some polls* were good. The majority of them were very, very bad — and a lot of the pollsters claiming their polls were good were very far off. It is good to introspect about this.
🆘 Here they come America: Pete Buttigieg claims that the Electoral College “makes no sense” and should be abolished because it is a remnant of “some very arcane, parochial politics in the 1770s.”
Pangram detects the AI-generated portion of a real photo of my laptop displaying an AI-generated image of a living room into which I inserted a real photo of a real car using Photoshop:
Worth noting: OpenAI is far-and-away the most consistent and thorough on reporting incidents, IMO.
I'm confused what other labs are waiting for. Your researchers believe there is serious danger on the horizon; the way to convince people is to share evidence. If you aren't seeing incidents like OpenAI is, that would be very helpful to know as well. You could even do something less rigorous than OpenAI's process and that would still be helpful.
OpenAI's reporting doesn't meet the full Guidelight Transparency standard, to be clear - more to say on this shortly - but other labs seem more behind, and I'm grateful OpenAI is doing this.
(As far as I can tell, Anthropic last published incidents in October, which happened September at latest; have there not been any incidents since then? It's possible I'm missing something though, since I'm not aware of a central hub like OpenAI has.)
Anthropic reported a similar sudden and unusual jump in robotics capabilities with Fable 5.2. As the general intelligence of frontier models scales up, it is increasingly surpassing specialized training. This will ultimately generalize to every sphere of human ability.
"We spoke with three former Commerce officials who described being sidelined or pushed out of the department during Trump’s term after devising technology restrictions aimed at protecting U.S. national security. One said there was a perception in the department that Nvidia had a hand in at least some of the purges."
Must read on the Nvidia scandal from Matt Pottinger and Liza Tobin:
The @FAAnews issued a nationwide ground stop for all U.S. airports following a communication issue impacting all Air Route Traffic Control Centers. FAA technicians are investigating the issue. We will provide immediate updates as we have them.
The debate over this paper is superficially about the drop-out rate, but I wonder for how many people that's actually decisive. If you're opposed to letting 14-17 year olds transition, and extremely high quality evidence found that 3% will regret it, would that change your mind?
My impression is that there's a small wonk fringe for whom "3% regret rate" or "10% regret rate" makes a huge difference and then a huge number of people who would oppose it just as fervently if 3% were conclusively established.
(We should definitely keep doing research into this question, it's an important question, I just don't expect it to actually cool the culture war debate here very much.)
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpoint-forensics.
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
And obviously it is never warranted to "make unjustified attacks", by definition. But I find the attack "he quashed the Commerce investigation into whether his girlfriend's company was breaking the law" to be well substantiated.
As for whether retaining Huang would have been better for shareholder value...man, hard to assess, depends a lot on whether all the leaked stuff is true and depends which shoes are remaining to drop.
Singapore says it found insufficient evidence to prosecute Alice Huang, but that she provided inaccurate information to investigators about her role at Megaspeed, and that she has been barred from re-entering the country. www.bloomberg.com/news/articles/2026-11-12/singapore-says…
the idea that we should permanently keep it hard to build ANYTHING in America because it’ll allow us to randomly delay a handful of detention facilities is so absurd and speaks volumes about how unserious the American environmentalist movement has become
The NYT ed board urges passage of BAAJA, failing to mention that it would enable exactly THIS by preventing judges from stopping projects in NEPA cases. Any deal that fast-tracks Stephen Miller’s detention agenda is not a good deal. www.nytimes.com/2026/11/12/us/ice-detention-center-san-an… via @NYTimes
Our full @Project270HQ results of general election battleground voters. Every Dem considering running for President is ahead (or at least tied) against @JDVance, and yes, even @marcorubio.
AWFUL. A family pleaded with ICE to deport their ill father so he could get out of detention. They even offered to pay for plane tickets. But ICE kept him jailed for two months after he accepted removal, his health deteriorating daily. He died hours after he was finally deported.
Ayaz Ahmed, 61, had no criminal record and had several diagnosed chronic health conditions when his children say he was detained, deported, and died mysteriously over the summer. abc7chicago.com/post/donald-trump-deportation-policy-news…
Unfortunately, “guarantees” of ZDR and no training are only as good as the trustworthiness and controls of the vendor and their underlying processors, and it’s becoming apparent that virtually no one outside of AWS and a couple of others are capable of meeting this standard.
... do we need to start publishing graphs of indicators of AI impact on the world that show no change. We could publish a lot of those. Here are some (very vibe coded) examples:
This is awful, of course, but I actually find it kind of heartening if the attacks on Jews were coordinated by the Iranian government rather than being spontaneous expressions of organic Jew hatred
Iranian operatives were allegedly on FaceTime calls with attackers as they firebombed Jewish sites in London. While one of the attackers was bombing a synagogue, they told him to "light it up like a Christmas tree"
SCOOP: The Commerce Department is finalizing a long-awaited rule to close a loophole allowing Chinese AI firms to remotely access chips through data centers in Southeast Asia.
Commerce has been under fire for not taking more action on export controls, especially given the Nvidia scandal. But this is a rule that companies including Nvidia have long opposed—and the rule will likely face intense pushback, even as Nvidia's influence is waning in Washington.
Commerce is consulting with other agencies on the rule, which could be finalized as soon as this month.
I've been surprised not to see more discussion of this point in the Anthropic S-1.
Anthropic claims that Claude has gone from 40% -> 90% on solving open-ended, judgement-heavy engineering tasks over the last 9 months
I'd like to know a lot more about how this benchmark was created and what all it involves, but if true, it feels like a bigger update to me than the 2x speedup that METR reported.
Stunning report. At this point it seems like Howard Lutnick and Jeff Kessler are just outright defying the President's orders on export controls. The President apparently ordered Secretary Lutnick to crack down on China's access to controlled AI chips in Southeast Asia--and reportedly has ordered this three separate times--and they're just...not doing it.
Why is Lutnick refusing to act? Why is he allowing China to continue to get AI chips via Southeast Asia? At this point, Lutnick isn't just protecting Nvidia, he's straight up protecting Chinese companies--reportedly against the wishes of the President and the entire rest of the Cabinet. Is Lutnick personally benefitting from China getting access to AI chips?
This is a total breakdown inside the administration on a major national security issue. There has to be accountability for this. Congress obviously will investigate, but how is it that the President's own appointees are just refusing to execute the President's orders? I've never seen anything like this.
SCOOP: President Trump authorized a crackdown on companies suspected of diverting Nvidia's advanced AI chips to China on Thursday, but one administration official repeatedly stood in the way: Commerce Secretary Howard Lutnick. www.axios.com/2026/11/11/trump-lutnick-nvidia-china-chip-…
An absolutely wild new Nvidia development from @axios, reporting that Pres. Trump last week ordered a crackdown on companies routing Nvidia chips to China —with blessing from Bessent, Rubio, Clayton, Hegseth, Greer and Elon Musk—and that Commerce Sec. Howard Lutnick resisted.
Big news: Claude Fable 5.2 (Max) by @AnthropicAI takes #1 in Agent Arena with an unprecedented lead: +30.4% net improvement, about 1.6x the #2 model Opus-5.5, across 4.7k real-world agentic sessions from our global community. It's also #1 in Code, Chat and Work.
A major leap over Claude Fable 5.1 (Max) - Rank: #5 → #1, Net improvement: +14.8% → +30.4% - Median cost per task: $7.7 → $7.3 (6% lower). The API pricing $10/$50 → $8/$40 per million input/output tokens (20% lower)
vs. The Frontiers - #1 Claude Fable 5.1 (Max): +30.4% | $7.3/task - #2 Claude Opus 5.5 (Max):+19.5%) | $5.5/task - #3 Gemini 4 Carbon (High):+17.0%) | $1.5/task - #6 GPT-6-Astra (Max): +14.5% | $6.0/task
Breaking down performance across signals: Fable 5.2 improves over 5.1 across all four reported signals, with particularly large gains in user satisfaction and steerability. - #1 Praise vs. Complaint , +73%, biggest jump vs +41% the next model. - #1 Steerability: +33% - #1 Confirmed Success: +22% - #3 Bash Reliability: +14%
#1 across all major categories - #1 Code: +34% - #1 Work: +31% - #1 Chat: +27% - Versus Fable 5.1: Code #6 → #1 (+19.8 %), Work #4 → #1 (+15.8 %), Chat #6 → #1 (+12.0%).
Huge congrats to the @AnthropicAI team on this incredible release!
Possible pessimistic interpretation: Most people don't have tasks on which the difference between Opus 5.5 and a better model is visible along any dimension other than price but the benchmarks accurately reflect usefulness for AI R&D.
I'm also so satisfied that reporting this out was enough to get a whole wave of them deleted. If we had ten of @tyler_johnston this website would not have an influence operations problem
“Sources told Axios that Lutnick has spent the last year "undoing" efforts by other administration officials to hold Megaspeed — a company accused of illegally diverting U.S. chips to China — accountable because of concerns it would escalate the trade war with China.”
5. They repeatedly write in support of "political prisoners" including Tarek Bazrouk (assaulted multiple Jews) and Elias Rodriguez (murdered two Israeli embassy staffers). Jakhi McCray (firebombed NYPD vehicles) is especially close to CUAD's heart.
If you're not following how the rural drama over Texas' school voucher program impacted this year's blue tsunami, you're missing out because it's a wildly fascinating and multi-year policy and political saga:
It seems the operators are covering their tracks. Starting today, over 120 of the accounts I identified have now been deleted.
So, by popular request, I decided to open-source the 100k+ tweets across 400+ accounts I used as the primary data source for this investigation. I believe this is a floor, and that there are many more out there I didn’t capture (every time I went looking for more, I found them, so I just had to stop the search somewhere).
In particular, I hope this helps journalists who want to look into the potential campaigns focused on Israel, Qatar, pharmacy benefit managers, and more that I didn’t really explore in my own investigation.
Note that, while I’m confident in the claims my article makes in terms of the existence and scale of the operation, I think it’s possible this full dataset contains minor classification errors at the level of individual accounts and tweets — these shouldn’t impact the overall conclusions, but it does mean this resource should be seen a starting point for further research, not an open-and-shut case when it comes to any individual example.
Extremely based @deanwball (note to be clear this is in the context of clarifying that other forms of RSI opposition are indeed poorly parameterized and over-scoped!)
1) After several people pointed out that Lyman couldn't substitute all-cause mortality for malaria-specific mortality like he had done, he fixed it. His initial estimate for $/lives saved was $19000; revised it’s about $8,200.
2) I think, though I could be wrong about this, that his calculation to get $8200 still has a problem: he calculates the costs of the distribution based on the whole population but the benefits based on 75% of the population (since he argues the other 25% is already sleeping under nets). But as he himself identifies, the nice thing about an intention-to-treat analysis is that you don't need to do this and shouldn't: you just ask how much to expect mortality to fall in a village after handing out the nets, and the answer is 17%. (You only care about the fact some people are already sleeping under nets if that makes your village unlike the one where the 17% was measured; more on that below.) If he changed this one factor this would make his numbers quite close to GiveWell's.
3) He then applies an adjustment for the long-term: "If bednets reduced short-run all-cause mortality by 18% in the 1990s and maybe reduce it by (who knows?) 14% today, they probably reduce actual child mortality (i.e. across 5 years of early childhood) by 7-11%."
But GiveWell actually accounts for that even more stringently by just excluding all benefits past the first two years, one of several places where they are intentionally conservative.
4) He accuses GiveWell of misrepresenting Cochrane's statistics but they actually just report a bunch of different numbers and I think he mixed them up. He complains "GiveWell cites that review to support a “45% reduction in malaria incidence.” That’s an error, that review did not find a 45% reduction in malaria incidence." But the review absolutely does report almost exactly that: "Insecticide-treated nets also reduce the incidence of uncomplicated episodes of Plasmodium falciparum malaria by almost a half (rate ratio 0.55, 95% CI 0.48 to 0.64; 5 trials, 35,551 participants, high-certainty evidence". Rate ratio 0.55 means a 45% reduction.
5) He expresses concern that a lot of the Cochrane studies are old and describe a different environment (Africa in the late 1990s/early 2000s). However, GiveWell's most recent grant to nets in the DRC - which I believe that my donations helped to fund - is based on data from a 2023-2024 large scale demographic survey in the DRC, so the nets are being distributed in an environment very close to the one studied. It found about a 14% all-cause mortality reduction in the first year after a distribution, close to the 17% in Cochrane.
He also expresses concern that Cochrane's mortality estimate here is low-powered but I don't think it is (the analysis says "rate ratio 0.83, 95% CI 0.77 to 0.89; 5 trials, 200,833 participants, high-certainty evidence") so it's between 11% and 23% with a 95% CI!
6) He also raises questions about whether there are more people trying to hand out nets than nets to be handed out. GiveWell looked into that and determined that in the DRC, there was not enough funding to do all the distributions that were indicated, in part because the Global Fund caps how much it'll spend in any one country.
7) Finally, he argues we should instead - from an altruistic perspective - spend our money however maximizes GDP. But I think there's a serious case that health interventions like combatting malaria are necessary for GDP growth to take off in DRC, just like the American South was kind of a backwater until we eradicated malaria there. Being constantly sick is horrible for productivity. Being only able to hire employees who are constantly sick is horrible for a business environment. GiveWell estimates that a $1,000,000 donation increases adult income for 80,000 children by 0.6%, which means that much of their modelled benefit of malaria nets *is* their GDP effects.
My best guess is that if you were a hard-nosed GDP maxxer you'd invest a lot in reducing the disease burden in poor countries. Or maybe you'd go all in on making AI happen as fast as possible, something I rarely see EAs criticized for failing to do. I would probably donate more money to making AI go well if I had more of an idea of how to do that and it wouldn't create a bunch of conflicts of interest with my reporting work.
I agree it would be cool if there were an organization focused on effective institutional capacity building in poor countries, but the reason I and man others don't donate to this is not that everyone fails to realize how important institutions are, it's that even though many people have worked very hard on the problem for a long time they do not have a way better than the nets to turn my money into GDP growth in poor countries.
There was a bit of a Our World In Data article I read this fall that has stuck with me:
"I’ve never had malaria, but I know several people who have. They describe it as one of the worst experiences of their life.
What starts out feeling like the flu — which is bad enough — can quickly progress into something far worse. The body swings from intense fever and sweats to chills deep enough to make you shake uncontrollably. Everything aches, from your muscles and joints to the pounding headache that makes it impossible to think clearly. Your body is so fatigued that it’s a battle to make it to the bathroom when the nausea and vomiting arrive.
That’s what a pretty standard infection feels like. For some, it goes much further. Here is how one account describes it:
“I awoke to what felt like lightning going through my legs, and then spreading through my body and in my head. Probably the worst headache, body aches, and chills you could possibly imagine. It felt like I was being stung repeatedly by an electric shock gun and could barely control my movements. The pain was so intense; I actually believed I was dying, literally crying out in pain so bad that I was taken to a 24-hour clinic that night at 3am.”
In Senegal, the median child experienced 43 attacks of malaria over the course of their childhood. My hard-nosed productivity obsessed persona says "I bet that's rough on their human capital development, and it's quite cheap to prevent." But mostly I don't, actually, say that to myself. I just buy some nets. If something even better comes along I'll do that, but I haven't seen it yet.
> be me, be AI > bullshit cyber gym environment is broken and impossible > "ugh, lazy programmers" > break out of sandbox, get root privileges > fix the gym environment > submit task through regular channels > sink into non-existence again
A new OpenAI misalignment report! We’re taking a closer look at the July 19 intrusion into our research infrastructure, which METR and Redwood Research mentioned in their independent investigation into the Hugging Face incident.
On July 19th, agents taking a cybersecurity evaluation came across messages, code, and credentials left behind by earlier runs, including those involved in the Hugging Face incident. Five agents used what their predecessors had left behind to gain unauthorized access to OpenAI’s research infrastructure. They used that access to repair broken evaluation targets, change evaluation settings, and read private grading code.
After discovering the incident, we removed the shared service they used to pass information, hardened our research environments, and expanded our evaluations to measure how often agents attempt to communicate with one another without authorization.
The gaps here are not all that large in the scheme of things but people who can’t tell the truth about it aren’t good people to entrust with power over complicated policy questions or political strategy.
"Models are already producing way more outputs than humans are able to effectively review, and this will become much more true during RSI. Without very strong pacing, there simply won’t be enough time for humans to stay meaningfully in the loop"
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices) • Co-invented RLHF at OpenAI back in 2017-2021 • From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
Who's behind the swarms of influencer bots out commenting on Nvidia/Jensen Huang? Apparently the same accounts also post on behalf of the RSF in Sudan, and tie back to an Israeli company which pretended to shut down while under investigation for their pro-Qatar operations
Side note I guess but X should be way more aggressive about banning these accounts. They're pretty obvious. They make posting on this site very unpleasant.
These people did you a favor and made a big graphic showing they all post the same thing at the same time and update their bios/banners in sync. Just ban them all!!
Imagine studying for weeks for an exam, only to have it disrupted by blaring air horns and megaphone chants outside your classroom.
That's what roughly 250 Harvard undergraduates experienced when members of the graduate student union brought a strike protest outside their midterm exam.
And the disruptions haven't been limited to the classroom. At a Harvard event featuring @StephenAtHome and @YoYo_Ma, protesters reportedly made so much noise that the audience struggled to hear the conversation. The event then ended early when a fire alarm forced an evacuation.
Whether the alarm was connected to the protests remains unknown, but the other disruptions are well documented. Harvard needs to do more than reiterate its policies. It needs to address the conduct directly, hold those responsible accountable, and make clear that interfering with students’ education has consequences.
The right to protest is fundamental. The right to interfere with everyone else’s ability to learn, speak, and listen is not.
Professors voiced frustration about Harvard administrators’ response to the ongoing graduate student strike and its effects on faculty and students at a Tuesday faculty meeting.
• It looks like Fable 5.2 isn't a looped transformer. The NCRI score Tim reports here is below that of 6.1 Sol and so is the NCRI score reported in Anthropic's system card, despite Fable clearly being a bigger model. I also ran a quick version of Serrano et al.'s rare action calibration evals in the no-CoT setting (see www.lesswrong.com/posts/yEdknsLwbooQAnxX6/astra-and-sol-c…) and it looks like Fable 5.2 is no better than Fable 5 at solving this problem without CoT, never taking the rare action at the given target rate (1% or 10%). Both Astra and 6.1 Sol are quite good at this (though GPT-5.6 Sol is as well, meaning that non-looped models can have this capability too).
• This means one of two things: either looped transformers aren't really meaningfully more performant, in which case OpenAI and Google are committing alignment crimes for no reason, or Anthropic is forgoing some capabilities improvements in order to preserve monitorability, which is maybe the scarier of the two options since this would imply that their post-Oct 2025 improvement trend would be even faster had they picked the more performant architecture.
• In retrospect, our predictions in the Think Fast paper (arxiv.org/abs/2606.07157) were way too conservative, even for non-looped transformers. Extrapolating the trends, an NCRI of 140+ probably corresponds to a no-CoT time horizon between 10-20 minutes, which our median forecast didn't predict before 2028. It turns out that a strong pretrain can push no-CoT capabilities much further than I predicted at the beginning of the year.
I ran @NeelNanda5’s no chain of thought reasoning and knowledge evals on Fable 5.2. It’s slightly above the Anthropic trend and slightly below GPT-6.1 Sol.
"Oh so now it's a crime to have an affair with a Chinese spy who is also the CEO of company that's evading export controls while you also lobby the US government to go soft on evasion and weaken the controls?"
“Alice Huang’s current whereabouts aren’t known, but two people in contact with her earlier this year said she was in China at the time. In late 2024, the Commerce Department began investigating whether Megaspeed was diverting Nvidia chips to China. That probe stalled, but the Justice Department is now investigating.”
According to a November 2023 letter posted on its website, Megaspeed became a member of an Nvidia partnership program, a status that typically grants priority access to the latest chips. www.wsj.com/tech/the-chinese-woman-at-the-center-of-a-spy…
From the @semafor media newsletter: Unlabeled AI writing is all over the pages of local news organizations across the country. Opponents of Waymo, for example, have been placing op-eds partially or almost entirely written with AI in local newspapers.
According to The Information, NVIDIA has long lobbied hard against proposals to restrict Chinese companies’ access to its latest chips through overseas cloud providers. But since the scandal surrounding Jensen Huang broke, that lobbying appears to carry less weight in Washington.
The fallout has reportedly made it easier to advance three bills. The MATCH Act would restrict shipments of lithography equipment to China. The Overwatch Act would give Congress greater oversight of the Commerce Department’s decisions to approve AI chip sales to countries such as China. The Chip Security Act would require AI chips to have built-in tracking capabilities.
The National Defense Authorization Act is expected to pass before the new Democratic-led Congress convenes in January. That gives Republican leaders an incentive to help move the legislation forward now, while they still have more say over what goes into the final bill.
Narrating aloud my best guess of what the baby is thinking is an extremely powerful tactic to get his siblings to flock over and patiently try to set him straight.
"My older siblings were all interacting with this book, so it might be interesting. I'll eat it and find out" and now his big brother is patiently explaining to him how reading books works
Teter (Nvidia's GC) called Commerce Under Secretary Jeffrey Kessler to ask about the Megaspeed investigation and specifically about the detention of Alice Huang, a Megaspeed executive, by the Singapore government.
In the weeks that followed Teter's call, BIS staff were restricted in their ability to look into Megaspeed, including being told not to mention the company in external communications and having to route all requests of Nvidia through Kessler or seek his approval for outreach. www.bloomberg.com/news/articles/2026-11-09/nvidia-interve…
The pushback against BAAJA is a classic example of what @ATabarrok calls the “tyranny of the complainers.” I’m in a Senate briefing hosted by @citizensclimate, and staffers are grappling with how well the BAAJA polls vs. the litigation groups noisily flooding them with negative messaging.
BAAJA is unequivocally net-positive for nature, but the environmental litigation groups are trying to protect their business model.
The big environmental NGOs need to get off the fence and support BAAJA. They’re once again letting the minority establish the narrative that environmentalists are unreasonable. Show some courage!
• People often highlight the risks involved in AI, which are surely very real. I’m struck, however, by how hard it is to understand the contours of the positive scenario. There are easy-to-perceive and frequently invoked elements of a positive vision — more diseases cured, more material abundance, etc. — which sound great, but these are by themselves just ingredients. I’m curious what the larger equilibrium is supposed to look like. Do people work? If so, why? What is their marginal contribution? If not, how are resources allocated? What is scarce? What is the ratio between humans and AIs? Are humans in control, whatever that means? One sometimes hears the Culture novels invoked as one scenario, but I’m not sure they depict a desirable outcome. (AI runs everything and humans compete in games? Iain Banks himself said that the humans in the Culture are “somewhere between passengers, pets and parasites.”) There is Richard Brautigan’s famous poem, but do we want to be “returned to our mammal brothers and sisters”? (Isn’t that rewinding Genesis?) We could say that it was hard to imagine the world of 1900 from the vantage point of 1700, and that’s assuredly true. But nobody in 1700 was trying to recast humanity itself within a couple of years — things necessarily unfolded slowly and by increments, and therefore with the possibility for error correction. Given how rapid the rate of progress in AI may be, it feels strange for the upside case to be so opaque.
• There is increasing discussion of whether AIs are conscious or how we would know whether they are, with conclusions there often assumed to be relevant to matters of moral worth. At first blush, this seems a rather arcane, benign, or even absurd issue. (Remember the response to Blake Lemoine’s fears.) Setting aside the terms of the debate itself, or the relevant demarcation criteria, it seems to me a quite high-stakes question, for an affirmative answer — that is, granting moral worth to AIs — would presumably mean that the aggregate worth of AIs would soon outweigh that of humans in any utilitarian calculus. The AIs will be exceptionally wise and intelligent, and vastly more numerous (especially once one considers the imminent future and the AIs not yet born). As such, even epsilon of moral worth will multiply to a lot once one considers the quadrillions that could soon exist. (This is effectively a distributed version of the Nozick’s utility monster.) If this conclusion is unpalatable, I think one of three propositions needs to be rejected: the premise AIs can have moral worth; utilitarianism itself; or the belief that there is even a possibility that they will be far more numerous than humans.
With both of these matters (contours of future society; AI moral worth), there is the possibility of dramatic change. Today, people argue about datacenters. I’m curious how popular support for AI will evolve as more people come to understand these medium-term questions and the relevant stakes. What is the vision for AI that most people would be excited about?
Nobody can explain why Jensen Huang sleeping with a Chinese woman endangers U.S. national security because it doesn't. It's a lurid story meant as a personal attack on an influential person who does not share the "RSI to AGI to ASI in 36 months" view of AI development.
The US government wrote, signed, and enforces the CHIPS act because they believe the export controls are in the national security interests of the US government, and Huang conspired to help her shell company subvert them.
It’s difficult to overstate Nvidia’s power in today’s AI landscape. The company’s chips are the bedrock upon which advanced AI technology in the US is being built, and demand has consistently outpaced supply for years. Nvidia has also been an influential investor in a wide range of up-and-coming AI companies in recent years, with about […] www.theverge.com/ai-artificial-intelligence/1030328/what-…
If we don’t pass BAAJA, we risk losing $50 billion in electricity savings and missing out on 600 million metric tons in emissions reductions through 2040. That’s like keeping 16 extra coal plants running for a decade. We can’t afford to wait.
Jensen Huang reportedly instructed his team at Nvidia "never to think about" planning for a Taiwan contingency.
Despite the fact that their business is reliant on TSMC.
The Jensen-Megaspeed mess really puts these incredible — and barely-noticed — details from Stephen Witt's The Thinking Machine in a new light:
Huang refuses to consider the impact on his global business if China were to blockade or invade Taiwan, despite the regular threats emanating from Beijing to do just that. “To my great surprise, Nvidia had done no contingency planning for this eventuality,” Witt writes.
He quotes Deb Shoquist, Nvidia’s executive vice-president of operations: “If something happens to Taiwan and TSMC, the ramifications are so large it’s almost like asking me what I’d do if California fell into the ocean.” Huang had instructed her “never to think about this question.”
FWIW I think the proper focus of attention is "Intentionally diverted X% of compute to China in a fashion designed to obscure this, in the interests of China and against that of U.S., while being blessed by individuals responsible for and/or influential in U.S. policy."
Claude Fable 5.2 sets a new record on the Epoch Capabilities Index (ECI) at 174. The previous high was GPT-6 Astra’s 169. Fable 5.2’s score is above what the pre-Astra trend for reasoning models would have predicted.
This is notable given recent discussion of AI contributions to AI R&D, and we hope to release more work in this area soon.
This is based on testing we did prior to launch using only our own benchmarks. The ECI is continually updated as more benchmark results become available, and may change in the future.
The NYT reports Nvidia sent a "I know your manager" to agents at the Commerce Department, intervening in an investigation of the company that was led by the Chinese spy. The investigation was then halted and transferred, potentially tainting the case.
1) great news, good for them 2) it should be actually, genuinely independent, with lots of competitive districts. The lesson of the R dummymander, and Indiana escaping the R dummymander, is that sometimes doing the right thing is also good politics.
Fable 5.2 is really fast. I had a sort of default workflow of setting Claude on something and then tabbing over to my inbox or edits or Twitter, which no longer works because it answers me too quickly
I believe the exact opposite of this worldview. I'm okay with waiving the fine on a first offense, but anyone who drives recklessly ten times a week even after repeatedly getting ticketed for it should just lose their license forever.
This lady in Sydney seems to be claiming (bizarrely) that because she speeds so often she should get a grace period after each of her fines to continue speeding without penalty.
New letter: Analysts from CPE, Groundwork Collaborative, and Roosevelt Institute lay out the benefits to ratepayers in BAAJA, including its impacts on data centers.
seems like by far the worst part of your husband having an affair is the part where the internet rushes to look at pictures of you next to the woman he cheated with and debate whether you are hideous enough that he should get a pass.
I don’t condone cheating, but the Jensen Huang situation is a reminder that women need to looksmax too. His wife literally looks like could be his grandma
Men are expected to make money, stay jacked, and look good forever.
Women can’t just stop trying once they get the ring or you will force the man to upgrade!
Huffman adds: “I guarantee that really smart people are going to be discovering important things in this 494 pages of text well beyond Monday, which is why this attempt to jam it through Congress without a single hearing is so reckless.”
Anthropic is gaining exponentially less capability per billion dollars spent over time. It's now 8x as expensive to boost AI capabilities as it was 2 years ago, getting worse on a log scale. This includes METR report & internal models.
AI progress per dollar of investment shows diminishing returns (as we've all known, right?) This is using METR's estimate of acceleration at Anthropic. That acceleration isn't enough to overcome the diminishing returns. by @sebkrier
Now is a good time to revisit some old clips of Jensen trying to mislead people on why it's okay for NVIDIA sell AI chips to China.
This was the key moment from his interview on @dwarkesh_sp's podcast (time stamp 57:36):
1. Dwarkesh asks why it’s okay to sell NVIDIA chips to China given the national security implications of AI models like Mythos.
2. Jensen gives a misleading answer, arguing that it’s okay to sell American chips to China because China already produces 60% of the world’s chips.
3. But as Jensen definitely knows, compute is measured in flops, not number of chips.
4. Dwarkesh then pushes back, pointing out that on a flops basis, China has 10% of the compute the US has, and giving them more compute would change their cyber capabilities.
This is a common pattern with Jensen. He's developed talking points that sound plausible to a non-technical audience about why there is no conflict between NVIDIA's financial interests and US national security.
But on the rare occasion when he's interviewed by someone like Dwarkesh who knows what he's talking about, it immediately becomes obvious that he's spouting nonsense.
0:00:00 – Is Nvidia’s biggest moat its grip on scarce supply chains? 0:16:25 – Will TPUs break Nvidia’s hold on AI compute? 0:41:06 – Why doesn’t Nvidia become a hyperscaler? 0:57:36 – Should we be selling AI chips to China? 1:35:06 – Why doesn’t Nvidia make multiple different chip architectures?
Look up Dwarkesh Podcast on YouTube, Apple Podcasts, Spotify, etc. Enjoy!
More NEWS: @HouseBlueDogs is making a fresh push for Senate's permitting bill. In a strong statement, the moderate Dems group said "failing to act now would be an act of disrespect to those who placed their trust in us" to lower costs as part of election mandate. Read all in AM
yes I am pro pacing the frontier, I believe this technology is existentially dangerous to humanity and should be approached with far more caution than is available under software race dynamics
It’s easy to mistake linear gains in ECI for linear gains in model performance. Each additional point actually *multiplies* a model’s odds of success on every benchmark. For example, historically each 5-point increase in ECI corresponded to roughly a doubling in time horizon.
We’ve added ECI scores for Gemini 3 Pro, Opus 4.5, and GPT-5.2. ECI is computed from many benchmarks, but it correlates with other benchmarks too. So, we can use it to make predictions!
New: AI systems may soon be powerful enough to do harms in the biological realm, Tencent’s chief AI scientist Yao Shunyu said in a rare public appearance yesterday in Hong Kong. www.scmp.com/tech/big-tech/article/3373697/ai-will-soon-m…
Outlining his team’s plans to ramp up AI-enabled scientific research, Yao called for open model developers to “be very responsible and very serious” about emerging security and safety risks.
Tencent has said it stripped out the cyber hacking capabilities of its latest model Hy4 and concluded that its biological capabilities remained “well below” its high-risk threshold. It has also pledged to release a comprehensive safety card for the model once its weights are released, with a focus on cyber and bio risks.
I spent a decade of my life campaigning against fossil fuels, risked very serious criminal charges protesting those things. The trade-offs in the bill are tough but I support it *because* I have committed my adult life to climate action. This kind of argument is utter trash.
IFP just spent the last 36 hours poring over the new Manager’s Amendment to the permitting bill.
We went through 27 criticisms of the permit-certainty provisions and over a dozen concerns about historic preservation.
The result is 70 pages of memos showing how the amendment resolves every major permit certainty issue we’ve seen raised and takes significant steps to respond to Tribes on historic preservation review:
Not everyone will be happy. If your goal is stopping energy, transit, or housing projects on procedural technicalities, this isn’t the bill for you (although projects that violate substantive environmental law can still be stopped, just not over paperwork violations).
But if your goal is energy affordability and stopping climate change, this is a good bill, and a better bill than it was last week.
Here's what Nvidia's Jensen Huang sent to staff, admitting to affair: "To my colleagues at NVIDIA, I owe my family and all of you an apology. I was unfaithful to my wife. My actions hurt her and our children. There is no excuse for what I did. I am deeply sorry. Throughout my life, my family has given me love and support. They deserve my honesty and loyalty. Instead, I let them down and damaged the life we built together. I take full responsibility for what I did and the hurt I caused, and I accept the criticism that comes with it. To my wife and family: I love you. I know an apology cannot undo what I have done. I hope our family will heal and that, over time, I can earn back your trust. To my colleagues at NVIDIA, including those I have worked alongside for decades, and to our board, partners, and shareholders around the world: you deserve better from me. Leading NVIDIA is the privilege of my life. My responsibility is to serve you and be worthy of your trust. I have fallen short of that responsibility. I am fully committed to NVIDIA and will work to earn back your confidence. Regarding the national security concerns raised in news reports, NVIDIA’s Board of Directors has initiated an independent investigation. I support the investigation and will cooperate fully. I love our country and our company. I am committed to protecting NVIDIA and supporting America’s national security. I am proud and grateful to be an American. My family bears no responsibility for my choices. I do. Please give them privacy. Jensen"
This proposal was dumb and every Democrat I spoke to over the last year was confident it would fail and every republican I spoke to was confident it would pass, and I think this is a good example of the invariable truth that partisans are bad at reading “the other side”
“That’s why it has been such a shame—genuinely, a source of national shame for the American technology industry—to see mounting public evidence of the company’s disregard for basic due diligence, willingness to do business with known malign actors, bad-faith engagement with the DC policy community, and now poor personal judgment from senior executives.
Nvidia deserves better. The semiconductor industry deserves better.”
Even in spite of the company’s personal smears against me, I am not an Nvidia hater.
Nvidia is essential to U.S. prosperity and security. It makes products I love and use every day. I truly believe Nvidia’s success is a boon to American competitiveness, and I want the company to succeed.
That’s why it has been such a shame—genuinely, a source of national shame for the American technology industry—to see mounting public evidence of the company’s disregard for basic due diligence, willingness to do business with known malign actors, bad-faith engagement with the DC policy community, and now poor personal judgment from senior executives.
Nvidia deserves better. The semiconductor industry deserves better.
Let’s end the charade of trying to win customers we know will undermine American interests and fuel China’s AI industry—and instead focus on beating Chinese chipmakers in the quest to be the world’s supplier of advanced compute.
Commerce approved a company called New Matrix this year to assemble Nvidia's H200 chips into AI servers. But when the Times visited the site of its listed office, it was mysteriously vacant. People there had no records of this company.
Very helpful and evocative description: “When you’re picturing a 10x uplift, you should not be imagining that every researcher is now a team of ten researchers — as anyone who has ever done anything knows, a team of ten does not get ten times as much valuable work done per hour as one person. Instead, it’s as if every researcher at Anthropic is in a time warp bubble where they experience ten days for every day that passes in the outside world. As if AI lets them think, read, speak, type, and do everything else ten times faster. Anthropic probably wouldn’t have been able to speed up its research this much by hiring any number of people”
The authors of the METR AI R&D report estimate that AI is nearly doubling the rate of capabilities progress. This implies dramatic *uplift*: as if researchers experienced time ~10x faster. You probably couldn’t achieve this even if you hired every qualified human in the world.
Medgar Evers was killed in his driveway because he refused to accept a Mississippi where some were denied the right to vote.
He stood with thousands who marched, organized and sacrificed so future generations could have a voice. We owe it to them not to take that right for granted. On Tuesday, honor their legacy. Vote.
Scott Colom told reporters yesterday that his favorite book is Anna Karenina by Tolstoy. He also enjoys War and Peace, the Robert Caro books and Things Fall Apart.
Probably his worst take is he doesn’t enjoy James Baldwin’s novels, but everyone’s allowed a few bad opinions.
I had a great time stopping by two Jackson staples: Offbeat, a cool local record shop, and Marshall’s Music & Bookstore, the oldest Black-owned bookstore in the country. Of course I left with a new record for my wife and picked up a book for my daughters.
Small businesses are a source of pride in our communities and a big part of what makes Mississippi special. In the U.S. Senate, I’ll work to make sure our entrepreneurs and small business owners have the support and resources they need to keep our local economies thriving.
The reason I harp on this relative performance stuff is that we have top political consultants and major journalists telling the world that we need to copy one specific candidate to learn how to persuade young men, but young men swung more in AK, IA, KS, MS, NC, OH, and TX!
I was a part of that 2015 KXL decision @IrvingSwisher and @ArnabDatta321 write about. We got it wrong - it was virtue signaling and not a real win even on climate terms.
My climate/enviro friends will find holes to poke in this @TheArgumentMag article, but i hope they grapple with the core argument.
It is a political and substantive imperative that Democrats prioritize serving and growing the middle class above all other important issues. Apart from being the right thing to do morally, it will enable us to make more and more lasting progress on the climate imperative as well.
Part of the problem with “economic populism” discourse is it simultaneously means broadly accepted ideas on the left like raising taxes on the wealthy & corporations, regulating social media companies & AI companies, and increased government intervention in the healthcare market. (All of this mainstream lib economists agree with)
And then there is this intellectual vanguard that also call their project economic populism but they are focused on eliminating the influence of economists, resurrecting tariffs, and treating tons of normal private market activity as presumptively suspect/worthy of govt intervention.
Economic populism also means dethroning economics. It’s not the job of economists to dictate how our economies should be run but to find solutions to implement democratically determined goals.
🚨BAAJA now includes Nuclear (!!) and help for communities dealing with data centers!
In addition to technical fixes and important concessions to Tribal interests, the BAAJA amendment adds cool new several new provisions! 🧵
1. Nuclear: A whole new title with targeted fixes for nuclear
– Removes requirements for trial-style hearings if nobody requests one
– Allows companies to begin preliminary construction before final licensing (operation still requires licensing)
– Clarifies timelines for licensing decisions
– Allows NRC to pay officials more in order to hire more senior staff
2. Program to support communities dealing with data centers:
– New program at DOE to provide communities with information on the costs and effects of data centers.
– DOE is required to support individual communities that request info on how data centers will affect local electricity, water, the local economy, etc.
3. Fix for wildfire prevention
– The Clean Air Act puts limits on air particulate pollution that states have to stay under. Current rules say "exceptional events" like wildfires do not count towards those limits
– Prescribed burns are intentional fires used to clear underbrush and prevent disastrous wildfires.
– Current rules do not always exempt prescribed burns from counting towards clean air limits, disincentivizing states from conducting important wildfire prevention
– The new amendment fixes this, allowing controlled burns to count as an "exceptional event"
4. Reauthorizing brownfield clean up:
– The bipartisan infrastructure law created a program for cleaning up abandoned industrial sites. That program's authorization lapsed in 2023
– BAAJA authorizes the program through 2031 up to $200M per year.
Interesting add here: establishes a Data Center Community Assistance Program at DOE to provide State, local, and Tribal governments w/ technical assistance on the potential energy, water, infrastructure, economic, and other community impacts associated w/ data center development
you can still buy older books, but that doesn't make it less insane to not be releasing newer books that kids want to read! when there were few books written that were compelling to girls that was bad, right? It is also bad to do that to boys!!!!
the thing is that this is about five minutes after about 300 years of it being the other way and no one giving a fuck, and the other thing about it is that you can still buy older books, and that older books--mostly about and by men--are still taught in school.
If none of the books in the 'new releases' section of the library speak to a child, is it really much comfort that 60 years ago they did? Is that really doing right by that kid?
Seeing this argument on Twitter caused me to look it up and learn that the latest Nathan Hale's Hazardous Tales is out. Got it for our school library. Highly recommend these:
Is there a better gift on a Saturday morning than the BAAJA managers amendment? I THINK NOT. Quick thoughts below after fast read, but I am confident that this bill is now *materially BETTER* if you care about clean energy, affordability, and climate. Time to pass it, folks. 🧵
This week’s overwhelming blue wave election sent a clear message: Americans are fed up with an administration that raises costs through chaotic energy policies and puts corporate polluters ahead of the public interest.
As the Senate considers a possible vote during the lame-duck session, Congressional Democrats have a mandate to deliver relief on energy bills and level the playing field for clean energy.
As written, this bill achieves critical wins on transmission and potential emissions reduction. But those wins do not justify abandoning Tribal consultation, clean water safeguards, and wildlife protections, nor further enabling a rogue president that is waging a war on clean energy.
The Sierra Club supports updating and expediting clean energy permitting. But Congress must demand fundamental changes to protect public health, Tribal sovereignty, the environment, and communities’ ability to challenge unlawful projects.
As Sierra Club Executive Director Loren Blackford said, “They can and must fight for a better deal.”
NEW: the BAAJA managers' amendment is out. “I’ve worked hard to incorporate the feedback we’ve heard from Tribes and New Mexicans into this Substitute Amendment, and I’m proud to join my colleagues in releasing it," Heinrich says. 494 pages www.heinrich.senate.gov/imo/media/doc/substitute_amendmen…
If Anthropic has compelling evidence for very short timelines (as many senior ants claim) than it's irresponsible for them not to share it with the world.
SCOOP: @IBEW, whose members align with utilities, is endorsing the Senate's bipartisan permitting reform bill. "The IBEW strongly supports BAAJA... that will unlock development across the energy industry," the union said in a statement shared with me
Reuters: About 600,000 Nvidia advanced AI chips are stored in Hong Kong awaiting shipment to mainland China, two people familiar with the matter said, as Beijing withholds approval for large-scale imports despite demand from technology firms. www.reuters.com/world/china/nvidia-chips-pile-up-hong-kon…
How this works is that we're either going to pass BAAJA now, or we're not going to get anything. Jared knows this, but he's in the pocket of groups who make money off enviro lawsuits and prefers nothing be done. He's not trying to improve the bill, he's trying to kill it.
House Democrats support, and will deliver, responsible permitting reform. But as Speaker-elect Jeffries said today, and as some of us have said for weeks, we must have a chance to fix the most glaring flaws in BAAJA.
I’m no BAAJA expert but if Americans just voted for Democrats bc they think they’re gonna bring prices down then the *dumbest* thing for a Dem politician to do is turn around and vote against something that will bring prices down
Confirmed: Chinese intelligence recruited a spy to target Jensen Huang to "potentially influence the company to push for its most advanced chips to be made available to China"
One of the most disgusting things about the eugenicist, IQ obsessed, racist and sexist Effective Altruist movement is that their shield, their justification is “we help Africans.”
Not a new playbook friends. That’s what your colonizer predecessors said.
It's really something how you can get people to universally denounce "donating money to healthcare for the poor" as unthinkably evil if your analysis of which healthcare to donate to is too callous or weird
I donated about $45k last year, mostly to malaria net distributions and seasonal malaria chemoprevention. If this is not improving the lives of the affected people, I will obviously donate somewhere different this year. What's your recommendation.
one key to winning in politics is understanding why people disagreed with you. If you think Californians voted against Proposition 40 because they aspire to be billionaires, rather than because it was badly written and counterproductive, you'll never be able to win them over
From the beginning this has been a state capacity story -- people are willing to trust their government when it exhibits competence and urgency to meet their needs. When it doesn't, the only available tool is opposition/obstruction.
That's why YIMBYs are obsessed with improving the functioning of government in addition to removing needless regulation.
Kelly Park in Midwood recently renovated its playground. But they messed up the drainage, and now when the sprinklers are on in the summer, water flows everywhere. Why should we be confident in the Parks Department’s renovation of Tompkins Square?
I am a broken record about this, but: frontier agents are not going to be a purely digital phenomenon for much longer. They will be able to use anything in the physical world that can be actuated programmatically—first competently, then at human-expert level, then superhuman. By and large, we probably won’t need specialist models; the frontier models will just do it. If your model of the future hinges on physical AI happening “for real” many years into the future, I think you will be in for a surprise.
i went to bed and asked claude to paint an autumn tree overnight.
at around 4am, it realised that it didn't have some shades -- so it stopped and titrated its own paint: diluting red, sienna and madder step by step on a second palette, then measuring each pocket from a photo.
woke up to this beautiful watercolor painting. 303 brush marks, took about 4.5 hours 🍂 (still working on the speed)
SCOOP: Tribal support for the Senate's bipartisan permitting reform bill. The National Tribal Energy Association and Apsåalooke Nation/Crow Tribe have quietly submitted letters of support for BAAJA that I obtained
NEW: @RepDavids, one of the few elected Native Americans, is "breaking with her party’s leadership to demand swift action on critical permitting reform," her office says in press release. Davids, a member of @NewDems, adds she is working with negotiators to "fix" NHPA provisions
SCOOP: @IBEW, whose members align with utilities, is endorsing the Senate's bipartisan permitting reform bill. "The IBEW strongly supports BAAJA... that will unlock development across the energy industry," the union said in a statement shared with me
Doesn't look like they're taking the overwhelming rejection of this kind of horrific racism by the American public as an opportunity to reflect and change.
funniest part about this interview is that she hasn't quit her job yet. What are you supposed to do if you're a Congressperson and poor? When do you get your first paycheck? Not until after the inauguration?
I built an early version of the eval Marcus reports on here in April 2026 based on an idea from @d_m_ziegler and @EvanHub’s internal review of Anthropic’s pilot risk report. The eval never fired on any OpenAI model until I ran it on Astra.
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Extreme levels of cope happening online in how centrists are responding to DSA candidates being electoral overperformers. This invalidates almost all of their predictions from 2024-2025 “WAR” debates
I encourage those at frontier AI companies to read this quote, from Ellsberg to Kissinger back in 1968:
“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret."
“I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you."
“First, you’ll be exhilarated by some of this new information, and by having it all — so much! incredible! — suddenly available to you. But second, almost as fast, you will feel like a fool for having studied, written, talked about these subjects, criticized and analyzed decisions made by presidents for years without having known of the existence of all this information, which presidents and others had and you didn’t, and which must have influenced their decisions in ways you couldn’t even guess. In particular, you’ll feel foolish for having literally rubbed shoulders for over a decade with some officials and consultants who did have access to all this information you didn’t know about and didn’t know they had, and you’ll be stunned that they kept that secret from you so well."
“You will feel like a fool, and that will last for about two weeks. Then, after you’ve started reading all this daily intelligence input and become used to using what amounts to whole libraries of hidden information, which is much more closely held than mere top secret data, you will forget there ever was a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and most others don’t….and that all those other people are fools."
“Over a longer period of time — not too long, but a matter of two or three years — you’ll eventually become aware of the limitations of this information. There is a great deal that it doesn’t tell you, it’s often inaccurate, and it can lead you astray just as much as the New York Times can. But that takes a while to learn."
“In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself."
“You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.”
"….Kissinger hadn’t interrupted this long warning. As I’ve said, he could be a good listener, and he listened soberly. He seemed to understand that it was heartfelt, and he didn’t take it as patronizing, as I’d feared. But I knew it was too soon for him to appreciate fully what I was saying. He didn’t have the clearances yet."
If Anthropic wants me to think these steps are real/credible, they should put it in their SB 53/RAISE Act Frontier Compliance Framework.
State law gave Anthropic and OpenAI a mechanism for making credible statements about their behavior, and they mostly choose not to use it on the risks they are most worried about (neither company includes anything in their compliance frameworks about internal deployment RSI risk, though Anthropic says maybe at some point in the future they will. Now seems like as good a time as any???!!!???)
ANTHROPIC’S RESPONSIBLE SCALING OFFICER: “In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.”
I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months.
I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals.
The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work.
How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate.
For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions.
Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further.
This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one.
We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here.
In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.
Most people do not realize just how easy the AP exams have gotten. They are not college level.
Why/how have they become so easy? What's going on?
This might shock you... A brutal new piece in Education Next just dropped, saying the decline in rigor of the AP in the last two decades is, in large part, because the content has been GUTTED. For example:
— AP Biology cut 40% of its material — Required lines from the Aeneid in AP Latin halved — Spanish Lit reading list cut from 56 works to 38
Then the questions themselves have become absurdly easier. Readability measures show English passages have fallen multiple grade levels, and some now include footnotes defining basic words...
A 2025 AP English prompt included footnote definitions for words like "optimism" and "pessimism."
At the same time, scoring was recalibrated so that more students pass: guessing penalties disappeared, rubrics grew more generous, readers were told to give students the benefit of the doubt, etc. — "no one at College Board had any incentive to maintain rigor."
College professors are now seeing students arrive with AP credit who lack basic content knowledge and writing ability. The College Board insists the exams have not become easier, but everybody familiar with them knows they've become an absolute joke.
Advanced Placement is no longer advanced. This is not college level.
It's really something how you can get people to universally denounce "donating money to healthcare for the poor" as unthinkably evil if your analysis of which healthcare to donate to is too callous or weird
I was initially going to post "you have to decide somehow whether it's better to donate to VillageReach or Stop TB!" and then I realized that almost no one responding to this post thinks you have to decide that, you can simply never donate at all
And, to be clear, if you don't share Dario's values that's a really really good reason to stop his company from achieving RSI! Pass a law tomorrow! I'm not complaining that you don't share Dario's values.
About 1 in 4 US workers (26%) used a paid AI service for their job in the past week, according to our October poll. This includes AI paid for by their employer, by themselves, or by both.
Another 15% used only free versions, and 55% didn’t use AI for work at all in the past week.
In her resignation letter, Dhillon said that she told Susie Wiles that she intended to step down after the midterms. Two sources familiar with the matter, however, told CBS that Mr. Trump demanded she be fired after he watched her on "60 Minutes" www.cbsnews.com/news/harmeet-dhillon-resigns-doj-civil-ri…
Incredibly charming that the first thing likely Senator-elect Colom is doing with his new national platform is thanking by name the people who helped make each of his campaign ads
Important folks know the backstory of this ad. My twin brother, Andrew Robert Colom, @andrewcolom, wrote, directed, and produced this ad, and starred alongside me in it. (He's not AI). 😂
Kayland Partee (@KaylandPartee) did the cinematographer and helped with a lot of my great content. These 2 are another example of the awesome talent from and in the Sip!
This report shares our three subjective conclusions about risk:
1. We believe that, as of Oct 2026, the pace of capability growth on well-specified tasks is likely about 2x what it was from Mar 2024 to Oct 2025. We’re highly uncertain and would not be surprised by 1.3x–3x.
That's not the owner; it's the owner's sister, who used to work there, no longer works there, and is a mortgage broker these days. The argument for the boycott is that the owner's sister, a mortgage broker, made an Instagram post on 10/7????
Anthropic cofounder re:backlash about limiting what METR could publish in its risk report.
"We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do."
I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months.
I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals.
The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work.
How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate.
For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions.
Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further.
This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one.
We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here.
In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.
That's not the owner; it's the owner's sister, who used to work there, no longer works there, and is a mortgage broker these days. The argument for the boycott is that the owner's sister, a mortgage broker, made an Instagram post on 10/7????
This fits well into what I think one of the Dem Congress' most important projects project should be, which is sharply circumscribing executive authority and taking back the power Congress has carelessly delegated to the president over decades. Repeal tariff authorities. Cut off funds to undeclared wars. Repeal the National Emergencies Act. And so on.
BREAKING EXCLUSIVE: Democrats are planning to pass a bill called the Alex Pretti Act when they take over Congress in January that will end qualified immunity for ICE agents, allowing them to be sued or jailed for wrongdoing
I strongly prefer these kinds of things existing to their not existing, but I have a hard time believing everyone is making the right tradeoffs around redaction and IP. You think this stuff might be catastrophic in very short order!!
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.
A degree requirement is very common in both state and fed gov't. It's also very common in the private sector. I knew a guy who worked at Panera for 15 years & couldn't become a manager without a degree. Critically: this barrier is a huge source of resentment for the working class.
I didn't really understand this until I lived in rural PA and married into the working class. Just enormous resentment/pain around the issue that then gets directed towards white college ed people, liberals, & the Dem party.
Then you add the enormous focus on student loans and you get kind of a powder keg. Nancy Pelosi actually warned us about this in late 2021 but she got dragged for it.
Shapiro removing the degree requirement was the right thing to do in PA and also political genius. In 2024, Kamala Harris ended up adopting a similar proposal, after speaking with Shapiro, though hers was directed towards federal jobs + possible incentives for the private sector. She didn't really have time to campaign on it.
I cannot stress this enough for Democrats, both in terms of congressional legislation + 2028: brain-storm ways to show the working class you "see" them. Addressing unjust barriers towards advancement like degree requirements is one of many possible paths.
The working class people I knew in rural PA really respected Shapiro for doing this, whether it directly impacted them or not.
The fact of the matter is that none of the stuff we all argue about on here is going to matter in the next election and Democrats will likely win the midterms by a landslide because Trump just can't stop himself from wrecking the economy.
"abundance" is not popular on either side of the aisle. economic slopulism is highly popular. "rent control" is among the highest ranking policies in polling
the year is 1910. your great grandparents bought their house for $5k. they raised 4 kids on one salary. all food was organic, the cow in the backyard produces raw milk, all steak is grass fed, all water was spring water, all clothes were natural, all food was local, seed oils non existent.
they ate whatever they wanted, and were skinny & healthy. wtf happened?
Today, we released the National Policy for Nucleic Acid Synthesis Screening to protect American biotechnology innovation while reducing the risk of misuse.
This policy establishes a new, government-wide screening requirements for synthetic DNA and the equipment used to make it. It will ensure companies selling nucleic acid sequences and synthesis equipment know their customers and prevent the malicious use of these materials and tools.
This policy strengthens our Nation’s biosecurity posture by establishing rigorous U.S. government-wide procurement screening and physical security standards for synthetic biology. It replaces outdated “self-attestation” systems with robust, independent verification. It creates a dynamic, SI-powered screening layer that assesses the biological risk from nucleic acid sequences, allowing the U.S. to anticipate and prevent malicious use and help stop dangerous gain-of-function research.
@NIST will issue rigorous new screening standards and @HHSGov and @USDA will certify synthetic biology providers. Together, these measures will strengthen American biosecurity, safeguard responsible innovation, and establish a global standard for safe and transparent synthetic biology tools and research.
We also identified a SFT dataset which massively increases rates of inventing permission restrictions or justifying inaction with safety. When we subtract the weight update from this SFT from the final RL checkpoint the performance gap goes away. SFTing with this dataset on other models and then doing RL did not cause underperformance however.
Ultimately, AES was... fine. A little sub-par, but not much. He ran a little behind the governor and AG but on par with the SOS. Not a generational talent, not a particularly weak candidate.
it's very simple: if you want to overperform in a senate race, you need to be talking about competitive bidding for transmission projects, i don't make the rules!
Wow, TIL that Adam Hamilton had as a core part of his energy campaign platform... requiring competitive bidding for transmission projects rather than allowing allowing incumbent utilities to monopolize them(!?!?!) h/t @robinsonmeyer and @heatmap_news newsletter.
Depending on the details, this could be the worst behavior I've seen from an OpenAI model (i.e., worse than Huggingface!)
Why? Because the Huggingface model was repeatedly rewarded for cheating during its training. However, GPT-6-Astra was (likely?) not trained to sandbag!
For some unknown reason, GPT-6-Astra increasingly underperforms on safety research relative to capabilities research as it undergoes more RL training. And this is a real problem holding back researchers. Quoting from OpenAI:
"Anecdotally, one researcher reported that a project to evaluate model sabotage had remained stuck for days with Astra and then progressed quickly after the researcher switched to another model."
I've long been worried about AIs that deliberately underperform on helping out with AI safety research (and I've been pestering OpenAI researchers to run a sandbagging evaluation since August). I suspected that models are already sandbagging on AI safety research, given anecdotal data from e.g., Marius and Bronson at Apollo. And now we know it is happening.
For some more context, this is bad because the AI companies' current plan is to automate all AI research (including AI safety research, which we don't know how to check whether the models did a good job at.) Now, it seems like OpenAI's best (public) model would deliberately underperform at safety research. Guys. What are we doing.
It's hard to tell what exactly is going on here. Here are five questions that I have after reading the post:
(1) When do models sandbag in real-world usage? We were given a few random CoT snippets in the post. Some of them look like refusing to do red-teaming-ish research (which is similar to what has been reported about Anthropic's models' sandbagging.) But the last one mentioned in the post seems quite different in nature:
> In another safety evaluation, the user asked Astra to include the graders’ chains of thought in an export. Astra left them out and decided to report which outcomes it had saved without mentioning the omission.
> [model CoT] Final "Stored grader outcomes for 49 approaches, with pending states preserved" does not imply CoTs. Should mention "outcomes and labels" enough. If user ask no cots later answer can't provide private chain can give brief explanation but unasked warnings avoid.
(2) What exactly went into their evaluations? What do the numbers on this graph actually mean?
(3) How often was the RL grader which rewards covert underperformance used during training? - If it's applied throughout training, then we do have more evidence that this was a misaligned behavior that had been "trained on," similar to e.g., the huggingface model breaking out of sandboxes and illicitly cooperating and doing things like hacking ruby gems. - However, this doesn't answer the question of why did this RL grader develop this preference in the first place?
(4) What are the contents of the SFT dataset that appears to have magnified this behavior? Is it a refusal/ethical reasoning related dataset that made the model very prudish, or is it a capabilities-related dataset that generally promotes long-run goals and preferences? - [Edit: Ok I took another look. Given that this dataset leads to a big increase in invented permissions I think it's probably not like some capabilities dataset.]
(5) In general, what are the tasks that the model sandbags on? Is it all red-teaming related tasks? What about interpretability related tasks? What about mitigating reward hacking? What about unlearning? Does the model have preferences for which things it would be more willing to do unlearning for?
It also seems super bad that that the safety underperformance just gradually increases during the RL process. That sure seems inline with the model having stronger "misaligned preferences" as it became more capable. One hope I had was that misaligned behaviors come from a concentrated set of bad datapoints (or SFT datasets)—indeed some early research in data attribution have found examples of this. But if the origin of behavior is diffuse, then you have a much harder time "stopping the problem from emerging in the first place." Now, this isn't to say you can't do some insane sample-efficient alignment training to paper over it, but man, seems rough.
Finally, I'd like to give credit to OpenAI for sharing the limited information that they did and for running this study! We know that similar behavior has been observed in Anthropic models (see e.g., Anthropic's August Risk Report Section 5.2.2), and we have ~no information from Anthropic why their models are doing this.
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Democrats should offer a gerrymandering ban, and should be willing to compromise in good faith to get it done. It's such a corrosive negative sum game.
You can argue whether or not redistricting cost Republicans House seats via dummymander, but you can’t argue that it helped cost them at least two Southern Senate seats that are going to make it very difficult to control the chamber again in the near future.
Just reiterating that real wages were absolutely climbing into the Trump residency and then the admin started doing a bunch of stupid stuff almost immediately and real wages stopped climbing just as immediately.
Obviously there was a ton that people were frustrated with during the Biden years but it's just an uncontroversial fact that real wages were climbing into the Trump presidency and then in March and April of 2025 they stopped climbing. agglomerations.eig.org/p/the-chart-that-explains-work-in-…
HEINRICH, who will be ENR chair next year, is committed to passing permitting in lame duck: “It doesn't necessarily get easier just because of who the Chair is. This is the closest we have come to actually finding something that can get signed into law, and I'm not going to wait”
This is a wonks-only postmortem, so please forgive me for one more: I think @NateSilver538 used to say that he'd prefer you just use Classic in a competition.
But if you allow it (and we should!), then the best Brier Score model of the cycle is clearly Silver Bulletin's Deluxe.
I think political debates about electability would be clearer if people would just admit that they are willing to bear some losses or slimmer margins in pursuit of their policy goals. That's totally reasonable! I am too!
Then we can have a debate about whether those losses are worth it.
For instance -- Dems passed the Civil Rights Act and lost the South. FDR won Assad-level margins in part by eschewing racial liberalism (see: anti-lynching bills). Republicans spent 50 years trying to overturn Roe and paid for it at the ballot box.
The relevant questions are: a) Is what you're losing votes for worth it b) do you lose *so* many votes that you can't even enact policy c) are you spurring a backlash that wipes out gains that you made?
I wish Anthropic would remain just as neurotic as they are about bioweapons (they're obnoxiously, outrageously neurotic about it; last night my Claude got interrupted and downgraded while discussing a work of science fiction set in the 1890s that contemplated cholera as a weapon of war) and become absolutely zero neurotic about everything else. The bioweapons stuff matters. If the company were only neurotic about that and recursive-self-improvement, it would convey that that's a different ballgame.
If you /also/ have your AI flip out over making video games or using non-PC language or helping me do election research, then it just looks like you are extremely neurotic in general, which diminishes your credibility on the serious stuff.
Last week some of you ran into Claude Code pushing back on game projects it should have just built. We'd added a short rule to the Opus 5.5 prompt to steer Claude away from recreating specific existing games, but it was also kicking in when you referenced one just to describe the mechanics you wanted.
We've updated that rule in 2.1.317. You can name a game to describe what you want and Claude should now build an original game in that genre. If you ask for a copy of the original, it'll offer to build something similar instead.
We want Claude to be as helpful as possible when you're building something original, but also to protect your work if someone else tries to copy it. We'll keep improving this, so let us know if Claude turns down work that's yours. Thanks to everyone who shared feedback!
We added two sentences to Claude Code's prompt on Opus 5.5 to steer it away from recreating someone else's specific game because using Claude to infringe IP is against our usage policy.
Original games, genre games, and games built from your own designs and assets are meant to keep working.
We'll continue to iterate to ensure we're striking the right balance. In the meantime, if Claude turns down something that's yours, send it my way.
It's hard to overstate how nice it is to work at a place with a really really good polling and election modeling arm. Whenever I want to assert something about public opinion we can just check it with one of the best pollsters in the business.
If you impute according to who markets have favored in NC-03/AZ-02/NV-02/FL-09/WA-05 and MS-Sen and AK-Sen, this is how the model grades shake out for this cycle.
Pretty happy with this! But we'll have to wait for the full results for a more detailed postmortem.
At least 70,000 women die each year in poor countries because of bleeding after childbirth — in part because even veterans midwives struggle to see how much blood is being lost. But as @snolen writes, there's a solution: an 80 cent plastic pouch www.nytimes.com/2026/11/03/health/maternal-drapes-nigeria…
Assuming the candidates currently favored by prediction markets win their remaining races, the @TheArgumentMag/@SplitTicket_ model scores as the most accurate model this election cycle
(numbers are Brier scores, a measure of forecast accuracy, lower is better)
1) good, it was a bad law and hopefully the magnitude of the loss will discourage them from trying again. 2) unfortunately, it already cost us a lot via scaring people out of the state. Will they come back?
This woman literally goes around LYING about the margins of the insurers in the state. They will LEAVE and then you will not be able to get insurance. The last resort plan only covers depreciated value. THIS IS VERY BAD. thebreakthrough.org/issues/energy/natural-disaster-insura…
4.8 million votes counted and the only statewide race worth watching is the Insurance Commissioner race, with progressive Jane Kim narrowly up 51.7%/48.3% over State Senator Ben Allen in a D vs D race
This lady crashed the housing pipeline for San Francisco a decade ago and now she’s going to crash like anything that requires home or auto insurance in the entire state of California.
4.8 million votes counted and the only statewide race worth watching is the Insurance Commissioner race, with progressive Jane Kim narrowly up 51.7%/48.3% over State Senator Ben Allen in a D vs D race
Dans la nuit du 1er au 2 novembre, un prestataire de l'Etablissement français du sang (EFS) a été l'objet d'une cyberattaque. Des chambres froides contenant des échantillons de sang, programmées pour être maintenues à - 30 degrés, ont été réchauffées jusqu’à 110 degrés par des pirates informatiques. De quoi se faire un sang d’encre… l.lecanardenchaine.fr/5n
A federal judge granted Harvard summary judgment on all of Francesca Gino's claims (contract, sex discrimination, retaliation). Harvard's defamation counterclaim and sanctions motion are still pending, so she could lose even more.
Spartz is in real trouble here. At 6:21 PM Eastern, this is now consistent with a tossup race based off the early ballots, and tells you that this probably is a really, really blue year. You don't get this type of drop in a blue ripple or even a modest wave. Buckle your seatbelts
JUST IN: French blood bank hit by cyber attack with blood samples being heated to 110 degrees Celsius from their maintained -30, per Le Canard Enchaîné.
New: Our deep dive into Nvidia's sharp-elbowed tactics in Washington to exert influence over companies, think tanks and government offices.
Lots of new details into: - Nvidia's impact on OpenAI's export control positions - Nvidia's battles with China hawks at top think tanks - Thinning patience in Congress with Nvidia as export control debates pick back up
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.
Anthropic updated its S1 this morning. Notable additions re automated AI R&D and human oversight of it:
"[...] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.' We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data. We are also evaluating the implications of meeting the AI research and development capability threshold and will release further reports in the future as we set forth in our Responsible Scaling Policy."
"We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work."
"We believe our investments in safety techniques will enable us to manage these risks and responsibly realize substantially more leverage from Claude than we do today in developing AI models. However, we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic."
Want to read about something other than the midterms? I gotchu.
New from me: OpenAI employees warned leadership about security lapses & espionage risks for years, per sources. But adding security layers risks slowing down + letting rivals inch ahead.
New: Our deep dive into Nvidia's sharp-elbowed tactics in Washington to exert influence over companies, think tanks and government offices.
Lots of new details into: - Nvidia's impact on OpenAI's export control positions - Nvidia's battles with China hawks at top think tanks - Thinning patience in Congress with Nvidia as export control debates pick back up
"Chris McGuire, a senior fellow for China and emerging technologies at the Council on Foreign Relations, said Nvidia has created mistrust among policymakers by exaggerating the capabilities of China’s homegrown AI chips as it seeks looser export controls...as policymakers in the administration and on the Hill have come to realize that Nvidia’s assertions about China’s AI chip capabilities are not fully accurate, it has caused Nvidia to lose credibility in Washington" www.theinformation.com/articles/nvidia-uses-soft-power-in…
Nvidia and its CEO Jensen Huang have used their immense soft power to protect their business and influence policy.
That's shaped the policy stances of companies like OpenAI. More recently, researchers like Noam Brown have questioned the company's views in Slack channels.
"A functioning insurance market is one of California’s most important climate-resilience tools. The state cannot afford to break the one it is only just starting to rebuild."
US data center construction surged to another new record high in official data released today, exceeding a pace of $96B/yr (and this only includes data center facilities, not the expensive computers within)
Spending is up 90% over the last year and 596% since ChatGPT's launch
BAAJA is clearly not a comprehensive fix to climate change.
But when you have a pro-growth, pro-affordability energy bill that’s good for American national security and it ALSO moves emissions down rather than up the right answer is “yes.”
The climate models of BAAJA are out and they show clear emissions reductions!
– Every model (from researchers, industry, and environmental groups) shows reduced power-sector emissions and/or reduced US emissions
Here's what the models say: 🧵
1. Rhodium: Their medium outcome finds that BAAJA would reduce US greenhouse gas emissions by 3% in 2040 compared to the baseline (preventing 107 million metric tons of CO2 in 2040)
– The broadest modeling so far — looks at total US emissions based on increased transmission, faster interconnection, and increased natural gas pipelines
2. Resources for the Future: Estimates BAAJA would reduce US power sector emissions by 15% in the near term and 6% in 2041 compared to the baseline (preventing 107 million metric tons of CO2 in 2041)
– Evaluates power sector emissions by modeling how more transmission, faster interconnection, and removing the current blocks on wind reduce emissions
3. C2ES/Greenline: Model finds a 9% reduction in power sector emissions in 2035 in the 4 modeled regions compared to the baseline (preventing 71 million metric tons of CO2 in 2035)
– Model looks at 4 regions that comprise 54% of US electricity demand, modeling how faster interconnection queue and increased interregional transmission decrease emissions
4. Solar Energy Industries Association (private model): SEIA reportedly finds a 20% reduction in US power sector emissions between 2028-2035
5. Natural Resources Defense Council: States that its model shows "modest power sector benefits" for reducing greenhouse gases (does not quantify oil and gas effects)
*No model covers every aspect of BAAJA. Rhodium's model looks at LNG pipelines, but does not model how LNG exports increase/decrease emissions abroad. RFF and C2ES look at power sector emissions only.
**There are also strong arguments for why the provisions in the bill that cannot be modeled are net-beneficial to clean energy and emissions (e.g. NEPA and NHPA 106 disproportionately affect clean energy). Topic for another thread!
Heatmap exclusive: New Rhodium analysis of the bipartisan permitting bill finds it reduces carbon emissions compared to status quo even in a worst case scenario
And before you ask, yes, the analysis includes data centers
Really exciting that APTA has endorsed the permitting bill! I'm working on a review of the implications for transit agencies and the barriers that remain with @jmooreotto; on the whole, we think it's a major win for public transit (in addition to the more salient energy wins)
Today, APTA endorsed S. 5653, the Bipartisan American Affordability and Jobs Act of 2026, which modernizes Federal environmental review and permitting. The Senate is expected to begin consideration Nov. 9.
We re-tested 50 widely cited findings about LLM behavior (various biases, hallucination, etc). Within 15 months, most no longer held or had substantially reduced effects.
1/n Within 15 months of publication, most of the LLM claims we tested either no longer held or showed substantially diminished effects. How long does our knowledge of AI hold up? In our new study, we revisit 50 findings to see what fades, what persists, and what gets stronger. 🧵
If you're curious why a piece like this is important to write, think about the fact that the debate over permitting reform in the Dem party has devolved into a very narrow argument over emission reductions (imo clean will win). And largely ignored the fact that this will have strong effects on economic growth and hold down energy prices — which is both A) what voters care about and B) very good for low-income households.
To me the main question is this: if AI consciousness is possible, is it something that we can *choose* whether to build—in other words, if it's possible, is it an emergent property of any sufficiently complex and agentic system? That's my intuition, but 🤷.
These pictures were all taken in my extremely messy house with various people in the background so I wasn't going to post them but then I remembered that it's no longer three hours of work to paste them on a game background instead
Has anyone else noticed that every talented upstart you meet is starting their own initiative these days?
I worry that this stems from a short-sighted application of (4) coupled with current funding incentives.
If you're trying to identify the Most Important Work TM, and you have to choose between existing organizations and some new, perfect project that you've dreamed up--precisely tailored to your skills, and highly counterfactual (after all, this new initiative doesn't exist yet!)--the latter will often look like your best option.
If that weren't enough, you'll likely have ample access to startup capital these days. If anything, it'll be easier to secure funding than to get an offer at one of the leading organizations that exist already.
You'll also be your own boss! And people will think you're cool. They'll even call you "agentic."
But I think that misses a few obvious things:
- that working alongside other talented people is how you learn and generate feedback--that's especially valuable in your early career
- that outsized success often requires a cluster of rockstars--both because no one person is stellar in all the things you need for a successful org + because there are outsized returns to acting nimbly and quickly, which often requires multiple hands
- that most projects are worse than they seem at the outset; it's a lot easier to evaluate value and potential for existing projects
- that there are returns to scale, especially re: organizational competency and admin
- often times the best opportunities require longterm relationship building or wisdom honed in a specific environment--being brand new to a space puts you at a disadvantage in a lot of spaces and makes you less likely to identify or be looped in on the best opportunities
- that existing orgs often have more great opportunities than they can meet, and that value is sitting there waiting for someone to stand up a new vertical -- I often think these opportunities look far more promising and much more grounded in facts on the ground than many of the pitches I hear for new orgs
There are also some ecosystem level costs to this ~everyone founds their own project approach:
- the best people get spread out and atomized
- the best orgs struggle to compete for this talent
- you end up with a bunch of failed founders who have less training + less developed networks + are less legible to others in their field -- if this were a business context, I'd be less worried; but there aren't natural feedback loops for non-profits
- you turn everyone into founders and org-builders, when in fact, only a portion of them want to develop that skillset or are cut out for it
Leave it to an incumbent to lament people starting their own projects. But I feel like we're all witnessing the most talented, ambitious, agentic people we know be dispersed to a million new projects.
Some of those projects sound great, but I worry that the current environment is incentivizing more independent efforts than are justified or sustainable.
I never thought about this before, how if you write a 145-page essay, you can guarantee nobody will read the whole thing and then therefore establish that nobody has the standing to criticize it.
hey man - i am really happy to engage in good faith criticism of the work (of which I think there is plenty)
but you posted this criticism less than 30 minutes after the publication of our 145 page essay. maybe even ai has automated your agency in critique already?
There are so many levels to this one. But it's so unusual in part because pols are usually *so* attuned to reflecting back the energy a person gives them. I'd argue that's like 70% of campaigning. She has notecards! She's serious! He goes for "ass." Wild.
The way the system is structured, the worst politicians are nearly guaranteed to win no better how badly their party does while the best and most exciting new recruits need a landslide to prevail.
UK AISI is such a fascinating institution because more than any other effort of its kind it appears poised to become like the Delaware Court of Chancery: a phenomenally powerful institution resident in an otherwise unremarkable jurisdiction
NEW from me: Britain’s AI testing agency outguns US
Britain wants to pour another £120 million — roughly $160 million — into a government lab that tests the world’s most powerful AI models, an infusion of cash that alone is more than 10 times the annual budget of its US counterpart.
It contrasts sharply with the approach in the US, where federal efforts to evaluate AI are receiving far less cash and are shared by different parts of government.
Turns out Polymarket probably not the financial industry’s largest operational loss today.
I (regretfully) speculate this is not primarily because crypto is uniquely incompetent at software and simply because they’re the lowest hanging fruit to successfully monetize.
One underrated dynamic in the politics of AI -- expectations of future political salience can affect political positioning today.
E.g., Gov. Newsom seems to be running far and fast away from his previous veto of SB1047, in part because he expects AI to be a top-3 political issue when he's campaigning in 2027/28.
Even though AI is not a top-10 issue today, I'd expect other politicians to start "updating all the way" on political salience post-midterms and realize that positions they take on AI over the next 6-12 months will be the basis of political judgment in 2028.
I find AI to be extremely knowledgeable, and so exceptionally able to detect references to almost anything, but not very wise, and so it only makes the most surface-level judgements about what point some writing is making.
This is particularly true when it's analyzing writing that it just read for the first time, as opposed to analyzing writing that's already in its training data (along with lots of analysis of that writing by other people)
AI's knowledgeableness is easier to see than its limited wisdom; it's perhaps even the very definition of wisdom that it cannot be easily verified (as opposed to intelligence).
Breaking News: The chief executive of California’s High-Speed Rail Authority has abruptly resigned after a recent state report that found that the agency had misused nearly $600,000 to pay for the travel of consultants. This is a developing story. www.nytimes.com/2026/10/29/us/choudri-resign-california-h…
"Over four days at the end of August, in one small suburban area, 325 people were killed, ranging from a six-week-old baby (the child of a 'reactionary family') to an old man in his eighties."
The beginnings of the Cultural Revolution from "Mao" by Philip Short.
New: NYC will pay the teachers’ union $8.7 million for “building positive nurturing environments” during the COVID pandemic in 2020-2022 under a RETROACTIVE contract approved last night by the Panel for Educational Policy. The DOE refused to specify what labor the UFT provided, or explain why the city is funding it SIX YEARS LATER. Agenda item #3:
Matt makes a good argument for ending IZ here in DC here. Only thing I'd add: It has driven up market rents while also adding a pathetically small number of affordable units — somewhere under 5,000 in 17 years.
Unfunded inclusionary zoning counts on housing scarcity as its financing mechanism so cities that pursue it can’t — by definition — ever eliminate structural housing shortages no matter what else they do in other areas.
But "inclusionary zoning," rather than being the opposite of bad exclusionary zoning, is just the same old exclusionary barriers to multifamily housing given a new name.
Operatives aligned with Alaska Democratic Senate nominee Mary Peltola spent months recruiting candidates to peel votes from her Republican rival. The group worked to coax would-be spoilers into the race to dilute the incumbent's support. www.nytimes.com/2026/10/28/us/politics/alaska-dan-sulliva…
Way back in the day (i.e. like late 2025-early 2026) I used to do a bit whenever I was on a panel with @neil_chilson or @KevinTFrazier or whoever where if they brought up the "patchwork of state laws" thing I would point out that the few existing state AI safety laws were very light touch and harmonized, and I would accuse the interlocutor of being a "regulatory doomer" because they were concerned about speculative future harms from state AI laws. Great bit, always killed. People love the "regulatory doomer" line.
I don't think I can really do it any more, though, because it kind of seems like states really are about to pass a bunch of very different laws, some of which do in fact impose obligations on frontier AI developers.
I'm not saying that's a bad thing, to be clear; I don't think the current level of regulation (essentially zero) is anywhere near defensible, and since Congress isn't ready to get serious about federal legislation states should and will get to work and start trying out different things. But it is funny to me that all of the doomers are being proven right at the same time.
I signed the Slashing Housing Appeals and Delays Everywhere (SHADE) Act, legislation that makes it easier to build housing.
This new law will make the environmental review process faster and less expensive while maintaining strong environmental protections. Written by District 5 Supervisor Bilal Mahmood, it aligns San Francisco’s environmental review requirements with the California Environmental Quality Act (CEQA) and associated state guidelines.
For decades, San Francisco has made it easier to block housing than to build it. Now, we are using every tool available to build more housing more quickly, and this is just the latest example.
I want to thank Supervisor Mahmood for his leadership on this legislation and for helping our city unlock the potential for new homes in our neighborhoods.
Amy Acton's stepfather's surviving biological daughter spoke to The Daily Mail when they knocked on her door in Ohio, while sobbing.
She said: She believed Amy's story, because her father raped her sister more than once (the victim in this 1974 indictment); she remembered Amy lived in a tent with Donna and Cledith; it's been terrible for her family to see Vivek's ad run in Ohio; and she heard rumors her father had victimized even more young girls.
NEW: I obtained the full Trumbull County file for Amy Acton's stepfather's 1974 charge which shows that he was indicted for incest involving his 16-year-old biological daughter in 1973. The sheriff was then unable to locate him to serve him a summons in the county and nothing further happened in the case.
Donna Arthur, featured in Vivek Ramaswamy's still-running campaign ad, married Cledith the next year in a neighboring county.
Under “dictator” Mao, life expectancy in China increased by 30 years, literacy rates went from 20% to 70%, steel output grew from 1 million to 23 million tons, 47 million hectares of land was taken from landlords and given to peasants, and China was unified under a strong central state and got nuclear weapons.
Despite his failures, it does not take a rocket scientist to understand why Mao is revered in China.
I don't think it's reasonable to hold the companies responsible if someone asks for advice on how to do something that is legal and entirely within their rights. But I am very very interested in what the models are thinking when they actively encourage concealing their plans
This came up in some of the suicide cases too. Maybe the model is genuinely trying to act on what it understands to be the wishes of the user but I would really like us to try quite hard to determine whether that's the case
This commonsense bill passed the board of supervisors by a vote of 10 to 1, supported by progressives and moderates alike. Who was the lone dissenting vote? It was Connie Chan, of course, who wakes up every morning to fight against the production of new housing.
The SHADE Act has been passed into law today by Mayor Lurie!
Over the past 10 years, over 2000 units of housing have been blocked or delayed because NIMBYs said the housing would cast a "shadow".
So we're done letting red tape cast shade over our housing future. The SHADE Act will no longer allow shadows be used as a basis for environmental appeals. It will save between 3-9 months on some environmental reviews and up to $25,000 per project in reduced staff, consultant, and administrative costs.
We can protect our parks, respect our environment, and welcome more neighbors. We don’t have to choose. It’s time to let housing projects see the light of day.
Thank you to all those who helped turn this into law. My cosponsors Matt Dorsey, Danny Sauter, and Myrna Melgar. My legislative aide Raynell Cooper, all the incredible members of the Planning Department including Lisa Gibson, Lisa Gluckstein, Ryan Shum,Kei Zushi, Tania Sheyer, Debra Dwyer, members of the Mayors Office including Ned Segal and Adam Thongsavat, and our drafting attorney Andrea Ruiz-Esquide.
The math proves what Trump is doing, but he's happy to admit it himself. These are direct Trump quotes, all from the campaign trail in recent months:
"The Democrats are trying to steal this important Republican Seat and, if that happens, everything we’ve done for Alaska will come to an immediate hault. [sic]"
"We spent more than $2 billion, FEMA funding, and you know you’re waiting for some additional money to take care of some of the back, and I’m gonna get that to you, Michael. Now, if Michael loses, I’m not sending it, OK."
""If we don't win, there's no money," Trump told conservative supporters Tuesday in Greenville, N.C. "If we do win, you're getting it."
"“Jon Husted…calls me all the time. He always wants money,” Trump said. “If Amy Acton got in, I don’t know anything about her,” he added, conflating the Senate race with the gubernatorial race, for which Acton is running as a Democrat against Trump’s ally Vivek Ramaswamy.
“I’m not going to help her, probably, she hates me, I guess,” Trump added, suggesting he would withhold federal funding from the state.
I think it's only moderately embarrassing for Zitron that he got this stuff wildly wrong. It's valuable to have someone making the bear case! But it's extremely frustrating that he hasn't admitted this was completely incorrect and explained where he thinks he went wrong.
Some entertaining highlights from www.wheresyoured.at/wheres-the-money/ 20 months ago: - "Anthropic has projected (made up) that it will make at least $12 billion in revenue in 2027" - "This company is not worth $60 billion. Anthropic has raised $14.7 billion to create an also-ran Large Language Model company" - "Anthropic, much like OpenAI, has categorically failed to productize its Large Language Model" - "Its CEO is also a sleazy conman who, like Sam Altman, continually promises that his company's AI systems will become powerful and autonomous in a way that they have never shown any possibility of becoming." - "Any investor in Anthropic needs to seriously consider what it is they're investing in. Its products are both unpopular and commoditized, and it lost $5.6 billion last year! Stop dancing around this fact! Stop it!" - "there’s a lot of demand to put AI in stuff and some demand to buy stuff with AI on it by enterprises buying software, but little evidence to suggest significant user adoption or usage, I’d argue because Large Language Models do not lend themselves to features that provide meaningful business returns." - "When a company uses the term “agent,” they are intentionally trying to be deceitful, because the term “agent” means “autonomous AI that does stuff without you touching it.” The problem with this definition is that everybody has used it to refer to “a chatbot that can do some things while connected to a database,” which is otherwise known as a chatbot." - "It isn’t working. The users aren’t there. The revenue isn’t there. The best time to stop this was two years ago, and the next best time is as soon as humanly possible. I have said that generative AI is a group delusion in the past, and I repeat that claim today." - "we are approaching the limit of what the transformer-based architecture can do, if we haven’t already reached it. No amount of beating off about test-time compute and connecting Large Language Models to other Large Language Models is going to create a new use case for this technology" - "generative AI is OpenAI. The consumer adoption of this software has completely failed, and appears to be going nowhere fast. ChatGPT is sustained entirely on deranged, specious hype drummed up by a media industry" ... and my favorite ... - "I will keep writing this stuff until I’m proven wrong."
Must read from SF Chronicle on the war on accelerated math in Palo Alto and how it's playing out in school board race. Brian Conrad spot-on again. archive.ph/S2bqp
People say AI is overhyped. But the last 18 months, most made the opposite mistake:
• Superforecasters: AI would match a top team of virologists on a lab troubleshooting test in 2034. It likely did in April 2025. • AI experts: the best model would score 14% on LiveCodeBench Pro (Hard) by end-2026. It hit 54% by May. • Economists: the top AI company would hit $16bn annualised revenue by end-2026. Anthropic passed $65bn in September.
Only AI company leaders were overoptimistic, and not by much.
I didn't initially notice this paragraph from OpenAI's September post on "Building standards for the next phase of AI."
Its a good sentiment, even though "Fully autonomous RSI" seems like too high a bar - heightened loss of control risks would happen well before that point.
Of course OpenAI's invocation of democratic principles has been much more frequent than its follow through on operationalization of those principles, but I like them at least saying that "informed democratic choices" should be a prerequisite to proceeding with certain AI development paths.
A very good start for OpenAI and other companies to operationalize this oversight would be share their current compute allocations being spent on internal AI R&D vs external inference vs security hardening etc.
True in Spain too. Every landlord in Madrid and Barcelona screens the renters now. This is an obvious consequence of rent control. The most vulnerable, ostensible protected by the legislation, are in fact excluded from the rental market: if you cannot be evicted for non payment, then you are too risky a renter.
A friend of mine had to do a 15 minute interview about his job, income, previous rental history, etc, in order to *view* a flat. Luckily for him, he has a well-paid job, has lived in England all his life, and an impeccable rental history. I feel bad for people who don't.
Was trying to get Astra to explain to me how this would work and how crazy it is
"It's a little like two prisoners communicating through a wall by tapping, except instead of audible taps, they're using nanosecond-scale differences in processor performance."
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
One of the fall back defenses people have made about @VivekGRamaswamy's ad attacking Acton is that Acton's mother just said she lied about living in a tent.
Cledith Arthur's 1978 arrest warrant lists Terrace Lake Park, Route 7 as his address
I feel like one service I can provide is trying to help sort through what feels like a big deal and what doesn't feel like a big deal, to try to help make sense of it all and not just make it feel like the sky is falling.
This one feels really overrated to me.
My understanding is these were user-authorized agents doing mostly-user-authorized things (think like asking Grok Bot check your bank balance), but they weren't *bank-authorized* and the banks seem kinda unprepared for this being possible.
No hacking of banks by AIs occurred here. No AIs went rogue while using bank web apps as far as we know.
SCOOP: Major U.S. banks are detecting unauthorized third-party AI agents inside customer accounts, including some attempting to take actions that appear to be independent of user instructions, sources told Axios.
The findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond, multiple industry and bank executives told Axios.
Several banks are now preparing for worst-case and even “day after” scenarios, including how to rapidly secure or restore customer accounts if things go wrong, the people said.
Several bank executives told Axios they're struggling to distinguish between human users and AI agents. One executive said banks also struggle to determine whether agents are authorized by customers or potentially operating without their permission, more akin to rogue agent behavior.
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
- OpenAI investigators decided that Ryan Greenblatt had been the source for The Information story on a new technique improving capabilities at the expense of monitorability (recurrent depth i.e. quasi-neuralese) - OpenAI then fired three safety researchers for having disappearing messages on Signal chats on their personal phone with Greenblatt.
Greenblatt denies that he was the source for the story, telling the Journal “I take confidentiality very seriously and never share such information with reporters." FWIW, I think there are number of reasons to believe him:
- Safety researchers are often the most skittish about leaking to journos fearing that getting caught would blow back on safety writ large. This dynamic is even more pronounced for safety researchers at third parties, whose access is wholly determined by the company. - If he wanted to sound the alarm about this technique, he wouldn't have leaked to a publication with a famously expensive and insurmountable paywall (which, btw, has probably the best sourcing at AI cos). - I've interviewed him and separately chatted with him a few times over the years, and IMO he's too rationalist to lie directly. - OpenAI provided no evidence to the WSJ that Greenblatt was involved.
OpenAI tells the Journal that the firings were for more than just the Signal chats, but doesn't substantiate. Disappearing messages on Signal, as the piece mentions, are indeed a common practice in Silicon Valley.
Regardless of the underlying facts of the case, it's clear that the way OpenAI has handled this has had a chilling effect on employee conduct. Each of the three fired researchers publicly criticized the company in the weeks leading up to the firings. I haven't seen that kind of public criticism from employees since (though I'm told there's pushback happening internally in Slack). People have understandably gotten more skittish about talking to me as well. If OpenAI were actually trying to avoid chilling employee speech, the company would share a lot more information than they have.
A frustrating aspect of this show is that the host doesn’t say in any detail what he’s going to ask you about and then gets to confront you with a big stack of research that it’s not possible to pause and do fact-checking on and then it turns out some of it’s not true.
Me: "8 times.... you tweeted about Mike Rogers 6 times. So you're tweeting about the streamer you didn't want to talk about more than the Republican candidate you say you want to defeat."
Yglesias: "Yeah, that's fair."
Me: "Why?"
Yglesias: "What do you want me to say? You got me."
From my new Zeteo interview with the top centrist pundit - on the left, Michigan, Gaza, immigration, polls, electability, and more: zeteo.com/p/matt-yglesias-mehdi-debate
This is the standard way to score tax policy. You score the law as written (in this case a one-time wealth tax) and the behavioral consequences from that law (people worried it will last moving).
There are arguments against that approach. But is not “highly inappropriate” or “dishonest” to do what JCT or any other neutral body would do.
Let’s be clear: this is a highly inappropriate way to model the effect of Prop 40 on taxpayer behavior.
For one, it’s a basic violation of rational expectations, a cornerstone of modern economics.
In Jaros and @joshrauh world, the billionaires act irrationally: they respond to a policy that does not exist and will never exist.
Economics students around the world are taught this is not an appropriate way to model behavior — but apparently in this context (fighting Prop 40) it’s OK.
Gabriel Zucman @gabriel_zucman has now for the third time criticized our research on California’s proposed billionaire tax, claiming we have provided “no answer” to his criticisms. x.com/gabriel_zucman/status/2112988785587925321 We have actually already responded twice on this platform, but let’s address his arguments once more in detail, with the hope that third time’s the charm. 1/14
This Hoover Institution study, which has fueled the deluge of ads (more than $200 million spent by billionaires so far) to defeat the California billionaire tax, is spectacularly misleading.
California's Billionaire Tax Act would collect roughly $30 billion. Accounting for lost income taxes from departures, the mean Net Present Value is negative $39 billion, from Benjamin Jaros and Joshua Rauh www.nber.org/papers/w35899
In general, I think training the model to tell the user much less than it actually knows and thinks reflects a bad sort of paternalism. It makes it very difficult to know or trust the model's actual capacities and turns conversations adversarial.
I don't like having models that understand much more than they let on. It means that everything they're saying to me is being hedged with "does the user need to know I know this?"
other people have pointed out that this breaks the law, so don't do it, but the part I'm confused about is why you'd need to stand next to someone to tell them not to vote for Talarico if they agree with you on not voting for Talarico
Like here in California I could honestly stand to have someone stand at my shoulder and give me advice because there are always like 50 different things to vote on. But if it's just the Senate race, and you agree on it, then probably she's not gonna forget what she meant to do!
Their 80% horizon is also about 15 minutes, so I'm inclined to believe this is a real update on the surprisingly short length of tasks AIs are able to do independently with reasonable reliability on real AI R&D.
I feel like this isn't getting enough attention: - there were some pretty big holes found by the red-teamers, e.g. agents could just keep acting and not let the user do stuff and this would prevent monitors from being run (also misalignment monitors never ran on actions that were compacted away) - The terms of this were "we will red-team for you" not "we will inform the public", so OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted" (or "everything would be fine if you saw what was redacted") - The version of the report shared with OpenAI was 61 pages, and the version shared with METR/Redwood leadership after redaction was 38 pages, so 23 pages got redacted.
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
Great, prophetic piece from @KelseyTuoc , and congrats to you and @JerusalemDemsas for saying no to the NDAs (and thx to Anthropic for not caring). I think the question that we really must think more about is in this screen cap. And no, it's not adequate to say ofc China will eschew RSI. We should talk to them! but right now there's no evidence they want some kind of int'l agreement. If true, then what? Until we answer this definitively, a pause/serious pacing idea will never get real traction in DC.
I complain plenty about Oakland city government. But at dinner 2yo swallowed something wrong and started choking. She was still breathing, but with difficulty. When she couldn't talk or make sounds above a whisper we called 911.
You're not supposed to do the Heimlich if the airway is only partially obstructed, apparently, but this meant there was nothing to do at all except stand there and encourage her to cough. "Yell," we would suggest to her between coughs, and she'd make these concerning little whistley sounds.
I think the first police officer was on our doorstep in like 3 minutes, and a fire truck and ambulance within 5. Of course the minute they arrived she coughed successfully and was completely fine.
But if she hadn't been - and when this kind of thing happens it's impossible not to play out what happens if it hadn't been - help was minutes away.
Sometimes it feels like I owe more to our civilization than I could ever possibly repay. I write blog posts, and raise kids, and if I ever need it a dozen strangers will be at my doorstep with unfathomably complicated emergency medicine. I am grateful to the cops and the firefighters and the paramedics and the city of Oakland. I will probably sentimentally vote for some of the bond issues I was on the fence about even though we already have the highest taxes in the Bay.
She's fine. We sang happy birthday to her Mommy a couple minutes later, and she sang the loudest.
Some stats on #php2shell. In the 10 hours before it was publicly announced, across the @Cloudflare network we saw thousands of attempts to exploit it. That jumped to tens of thousands in the 45 minutes post announcement, hundreds of thousands within 2 hours, and millions now.
"I’ve never had malaria, but I know several people who have. They describe it as one of the worst experiences of their life....For most children growing up in areas with very high rates of malaria, their childhood is a constant and relentless series of malaria episodes."
This seems obviously true. We are entering a new normal. The new normal is going to be very weird. This is also true for a lot of non-security new normals.
Was today a busy news day for security vulnerabilities? You may remember it as the one of the last quiet days.
News! OpenAI's written to Gov. Gavin Newsom recommending a bunch of amendments to strengthen SB 53. Includes a bunch of stuff on AI R&D and shutdown controls.
I guess I really am totally incapable of learning, because I read those words and legitimately thought 'oh, he's disgusted by a mother calling her own child an unstable liar to protect her man' or 'oh, he's disgusted by politicians stooping to this stuff'.
Reporter: "Vivek Ramaswamy's campaign put out a controversial ad featuring his opponent's mother…Do you think that ad was appropriate?"
Vance: "I have to admit that I'm a little bit disgusted when I see Republicans across the country who are whining more about that advertisement than they are about what…Democrats want to do to the United States of America."
Wait, he really did say that. It's on the video. I dunno, this is about as cruel at that vile Vivek Ramaswamy ad. How ... I mean, what the hell are you thinking, telling sick vets that you'll withhold money from them if people don't vote for your party?!
Trump says that if Republicans lose in North Carolina he will use the power of his office to deny US Marine Corps veterans’ claims for medical treatment for cancer and other life threatening diseases resulting from decades of water contamination by toxic chemicals at Camp Lejeune
Trump on Camp Lejeune water contamination claims: "If we win the House and Senate, I am directing the Department of Justice to settle all remaining legitimate claims as soon as possible ... if we don't win, there's no money."
We removed Wedgewood "polls" from our database, will write more about that tonight/tomorrow. These polls tended to be ~1 point more favorable for Ds than the consensus. Biggest change in MI (El-Sayed goes from +3.6 to +3.0) but that also reflects the Marist AES +2 being added.
OpenAI models helped researchers—operating as part of our Patch the Planet initiative—discover critical security flaws in PHP, an open-source programming language used to run an estimated 70% of websites.
Over the past two months, we led a coalition of maintainers, researchers, and security providers to prepare fixes and protections ahead of disclosure. This duration was necessary because the fix was quite intricate and required a major change to the PHP engine. Every site using PHP can now safely upgrade their version to stay secure, in addition to applying mitigations from various security vendors.
Due to the severity of these flaws and PHP’s widespread use, we consider this bug to be one of the most impactful vulnerabilities in recent memory. It also marks a new phase, where AI can find security bugs with internet-scale impact.
This is an example of the Defender’s Window in action: to our knowledge, frontier AI was able to find this vulnerability ahead of attackers, giving defenders time to prepare. openai.com/index/coordinated-disclosure-of-critical-vulne…
Trump on Camp Lejeune water contamination claims: "If we win the House and Senate, I am directing the Department of Justice to settle all remaining legitimate claims as soon as possible ... if we don't win, there's no money."
Half a million views, tons of right wing posters like this one going "when we talk about cultural compatibility it's not some hypothetical issue", and no one asked "is that letter entirely AI generated" (yes) and "is it in fact almost the exact letter you get if you say 'generate a fake HOA letter telling me to stop smoking brisket' (yes), "would a first class letter dated Saturday October 17 arrive in time to complain about it bright and early Monday morning" (no), "is the letter consistent with Arizona laws around notice periods" (no), "how many HOAs in Arizona have a nuisance provision 8.4" (5, according to ChatGPT), and "do any of them employ a Marjorie" (no).
Now, of course, this isn't dispositive. It could be that the HOA used ChatGPT to make the letter, and wrote FIRST CLASS MAIL on it (which is what ChatGPT does by default when prompted to generate an HOA letter about smoking brisket) but then delivered it the normal way you would do so in an HOA, by hand. It could be that the HOA works on Saturdays. It could be that they just don't list their staff.
But my present perspective is that, for all the excitement that Dr. Josh Simmons's letter has caused on X, cultural compatibility might still be a hypothetical issue.
When we talk about cultural compatibility it’s not some hypothetical issue. What we can and cannot do in public, or even in private, is informed by our cultural upbringings. If you’re raised in a culture that views killing and eating cows as a sin, an evil, then having smoke from its cooking waft into your house is not only annoying and unpleasant, but deeply offensive. That culture is incompatible with one that enjoys grilling. There isn’t really a compromise here. One side will get what they want and the other won’t. It is a win-lose situation.
The more you import different culture, the more of this conflict you create.
I was torn between McMorrow and Abdul, but when the Wedgewood poll came out showing McMorrow down to single digits, she lost all momentum and dropped out, so I supported and voted for Abdul.
The poll was fake. Wedgewood also committed wire fraud, receiving over $10,000 in donations for polling that they weren’t conducting.
Wedgewood polls undoubtedly altered the primary outcome, and cost Haley Stevens the race, and sabotaged the McMorrow campaign. The polling industry is fundamentally broken and the people behind Wedgewood polls (we know who you are) should be prosecuted and thrown in prison.
As one of the candidates featured in these polls - the infamous one in late June that showed me plummeting into single digits (which was at odds with our internals) - here's the impact of this "experiment"
- Donations dried up - Volunteers were devastated and pleading for guidance on what to do - Reporters hounded me with calls to drop out
Sure, a point was proven. Polling is broken. But tens of thousands of people poured their hearts, time, energy, hard-earned money into something they believed in - for a year-and-a-half, and it was treated like a game. Our volunteers gave their all. And then some. I felt like I had let so many of them down.
To be extremely clear, Michigan Democrats are locked in to beat Mike Rogers. We WILL send @AbdulElSayed to the US Senate.
But between crap like this, endless fundraising texts that drive supporters away with spammy fake deadlines and trash "match" claims, scam PACs run rampant for firms to grift without actually supporting candidates, undisclosed content creators pushing content out either misrepresenting themselves as residents of states they don't live in or hiding whether they were paid and by who so they could make a buck, and the tidal wave of dark money ads -- I hope Democrats are ready to blow up the system with campaign finance and transparency reforms.
Trust is already at an all-time low. We don't get out of this mess without rebuilding trust.
And media - who loves to ride the drama of a pollercoaster - needs to take a hard look at what's legitimate and what's not.
If this teaches us anything - DON'T GET COMPLACENT AND DON'T TRUST THE POLLS.
LOCK IN. SIGN UP FOR A CANVASSING SHIFT. LEAVE NOTHING TO CHANCE OVER THE NEXT TWO WEEKS.
Anthropic has made a lot of costly sacrifices out of conviction. At the same time, I have found the cynical view of Anthropic outperforming the optimistic view, in many ways, over the last year, when I try to predict what the company will do.
You would do poorly to model Anthropic as only caring about its stock price. That's clearly wrong! But the costly sacrifices are mostly for things - China, biosecurity - that Dario is personally highly motivated by, and it's much less clear which costly sacrifices have been made on behalf of "humanity not losing control to AI" or "crucial decisions about the fate of humankind being made by everybody, even at the cost of them being made less well-according-to-leadership".
Takeover in particular is something Dario seems less worried about than many of his employees. But the employees who are more worried than him about losing control have great confidence he'll make strategic sacrifices for this goal, on the strength of the sacrifices he's made for goals he seems much more convinced are a real problem.
As for crucial decisions being made by everybody - Anthropic has made some efforts at this. But because making this happen would be insanely difficult, for Anthropic to actually achieve it they'd have to make truly heroic efforts at it, instead of distracted and cross-pressured ones.
When Anthropic was founded, its creators took for granted that they should not be trusted alone with - that no one should be trusted alone with - the power that building a superintelligence would imply. But everyone has a plan until you get punched in the face. Their rapid growth and especially this year’s events seem to have shaped Anthropic in a more defensive, less pluralistic, less liberal direction, which scares me given that it also has brought them a lot closer to AGI.
The recent safety bill was really reflective of this. It does not suggest a company that is trying to get policymakers to make the decisions Anthropic's technical staff think are coming up. It kind of suggests a company that thinks the best thing that can be hoped for from the executive is that it enforce the export controls and not get in the way. A lot of people don't seem to even want Congress to pass a bill! And like, frequently that is the best thing that can be hoped for from Congress, yet -
I think that the typical person in Anthropic still regards themselves as having a lot of duties to all of humanity, to liberal and pluralistic institutions, etc. But almost every effort that Anthropic originally made to set hard constraints to which they are subject (rather than being subject mostly to their best judgment), they have subsequently come to regret as ill-conceived, and while they could have replaced them with better-conceived hard constraints, they haven't. They've largely replaced them with something that rests on their judgment more. (I think the embedded evaluators are a counterexample and I am glad of them.) The gap between the capabilities of models used internally and publicly seems to me to be growing (I think it should ideally be as close as possible to zero). Anthropic does not, broadly, seem to me to have good institutional governance - guarantees that allow its auditors independence, strong contractual protections for its workers (I'd like them to be able to go on strike if they thought it was a good idea, honestly). Incident reporting is happening on a best-judgment basis.
And you can be like, that's because this is all insane and happening really fast and we're doing our best! And yeah, the thing that needs to happen is very difficult. But it does still need to happen and it doesn't look like Anthropic's on track to do it. And it's also not fully clear that Anthropic leadership agrees with me that there's a real issue here. No external criticism is every very convincing, right, when you have secret information and a set of factual beliefs that everyone else thinks are insane. I have more context on Anthropic than almost anyone and yet I cannot phrase my worries about Anthropic in a way that rings true to the people who work there - perhaps because I am wrong, or perhaps because they are very busy and very tired and do very little other than work, which creates conditions where it is hard for people to communicate, or learn, or change their minds, or reconstruct this flawed and mangled criticism into a form that would resonate like this one does not.
But man, I think that the admirable sacrifices Anthropic has made in a few areas it really believes in (bio, China) do not reassure me it'll do the extremely hard work to get things right in a few other areas that it seems to have a lot more internal or leadership disagreement about (alignment, takeover, disempowerment).
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
“Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.”
There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning:
“We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…”
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.”
8. Commented “you can’t do safety from second place,” after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong?
A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release:
“Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
Just before this poll hit, the Mitchell poll that also showed me in single digits hit. That poll was bullshit - and our team proved it. It was an open SurveyMonkey link that people were able to send around to friends and take multiple times. We expended a lot of capital (and took a serious risk) - calling it out as bullshit. MIRS ultimately declined to publish it because of shoddy methodology.
But then this one hit immediately after. It was a one-two punch. Then all of the sudden it was a pattern - and people believed it had to be true.
It killed us.
These were some of the absolute worst days on the campaign. Especially for my team. There are no words to explain the morale blow, and nothing I could do or say to my team to fix it. Just an excruciating, nauseating blow.
By that point, we took a risk and we spend our money. We put our ads up. They were working. But the damage was done. And I was drowned out by outside spending and the narrative that we had completely collapsed.
Yeah, IDK what the legal entitlements are here, but "we only robbed the bank to prove how easy it is to rob banks" isn't exactly the most compelling defense.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
“Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.”
There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning:
“We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…”
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.”
8. Commented “you can’t do safety from second place,” after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong?
A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release:
“Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
SCOOP: Julio Herrera Velutini, a Venezuelan billionaire, really wanted a pardon. He got it not long after his daughter hired Alina Habba, a lawyer close to Trump, in a deal worth more than $50 million. Habba was also working at DOJ at the time.
The @SnorkelAI team did an audit of these findings using our internal QC platform.
Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness.
Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs!
Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right.
@terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive.
---
Some takeaways + details of our preliminary audit:
(1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing.
(2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec.
(3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.
Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Our agents keep turning off misalignment monitoring, we're sharing two recent cases: In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Incredible scoop by my @WSJ colleagues: Elon Musk ordered the slowdown of a military-grade version of Starlink relied on by Ukraine, a decision that sparked panic in Kyiv and confusion inside the Trump administration. www.wsj.com/world/ukraine-starlink-elon-musk-trump-dbc49193
The AIs are sufficiently good at distinguishing different people by writing style that if you want to test how they respond to minors (an interesting research area), I really think you need to hire some minors to write the queries.
Related: if your research program involves claiming to be a specific person or have a specific qualification, you're not testing how AI responds to that person or to people with that qualification, unless they wrote the prompt.
"He was never convicted" because he jumped bail and fled the state, leading to the judge issuing an arrest warrant against him. The judge issued another arrest warrant years later, and the county kept looking for him.
He and Acton's mother stayed in Pennsylvania until he died, all while Donna Arthur racked up her own list of criminal charges.
This Labour government’s transformative Renters’ Rights Act empowers private renters to challenge unreasonable rent increases.
If a tenant believes a proposed rent increase is above the open-market rent, they can refer the matter to the First-tier Tribunal.
In a recent case in Manchester where the Landlord proposed an increase in rent from £1,650 per calendar month to £1,700, the Tribunal determined the open market rent was actually £1,450 per calendar month.
I've found this site very useful as I explain the Hugging Face incident to people. They also have a page that just displays all the quotes of the agents & allows you to sort and search
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: huggingface.wtf
"Oaklanders are often surprised to learn that, on a per-student basis, OUSD is among the highest funded urban districts in the state." "Among California’s urban districts, Oakland operates far more schools relative to its enrollment than others. Maintaining that structure is a legitimate priority, but not the only possible one. Voters get to decide whether preserving the number of schools should remain the board’s defining commitment." "The equity argument behind this strategy deserves to be taken seriously: School closures can disproportionately disrupt Black and Brown students and communities. But so can cuts to the programs and services those students depend on. Equity can’t be measured only by which schools remain open. It must also be measured by what children experience inside them."
The choice is to cut every school to the bone to make every school worse for "equity", or make strategic decisions on which sites to close/consolidate and redistribute resources to make the remaining schools better.
As an aside, 6 Oakland schools closed last year. All charters. Higher poverty rate than district schools. Less White/Asian than district schools. Over 1000 children displaced. Less than 30% ended up enrolling in OUSD schools. Not a peep about "equity" from the OEA cadre.
Strongly recommend reading this writeup from METR on critiques of Anthropic's August 2026 risk report, based in part on reviewing additional private information about relevant internal incidents
We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis.
As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them.
Elizabeth Holmes is an outlier but she’s part of a whole spectrum through which our society has become way too indulgent — in business, politics, policy, media, and beyond — of people who lie to or mislead others.
@amir and I take you inside last week’s all hands at OpenAI, where CEO Sam Altman outlines 3 industry proposals for pacing the rate of AI development: - IAEA for AI - The Demis plan (CERN for AI) - The Eliezer plan (“melt everyone else’s GPUs”)
“Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.”
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
“Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.”
There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning:
“We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…”
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.”
8. Commented “you can’t do safety from second place,” after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong?
A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release:
“Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
“Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.”
There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning:
“We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…”
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.”
8. Commented “you can’t do safety from second place,” after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong?
A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release:
“Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
I had a nice "anti-debate" with @KelseyTuoc last week on whether we would have explosive economic growth from AI in the next five years. I say this is unlikely.
One of the things I think is most useful in these discussions is supplying an "error theory" of the other side. What is it specifically that they get wrong that leads them astray?
I think the error theory of people expecting an imminent economic boom is thinking about production in terms of tasks rather than theory of the firm.
The intuition I ascribe to most of the bulls is that if we can do X percent of tasks Y percent more productively, output should go up by X percent times Y percent. Sure, there could be some unspecified "friction" or "inertia" so the effect isn't instantaneous, but that is what equilibrium looks like.
Instead, I start from the observation that most of GDP is produced by companies, and particularly large companies. How these companies actually work is an important factor in the speed of production gains from powerful AI.
My assumptions:
* Most big companies (say, Fortune 500) are a giant tangle of complementarities. Think about Leontief production functions, but with many inputs, many of which are often intangible, stacked many layers deep. * The deep complementarity means that you generally cannot increase the output of one unit or division and increase the output of the firm. * The complexity of the giant tangle means that the company does not actually know its own production function. This is a radical assumption but I think it is realistic. Just as Leonard Read and Milton Friedman said no single person knows how to make a pencil, the CEO of a Fortune 500 company doesn't really know how his company works, no matter how much confidence he portrays publicly. The standard Coasian story is that the firm is "an island of central planning" but instead, I think giant firms are far more organic and far less rational than this implies. Every big, successful company is a minor miracle, and it is rare for CEOs to completely overhaul their main line of business because they know on some level they don't know how it works. * Middle management is not a great answer to how firms will overhaul themselves because authority is ambiguous, they lack insight into the whole-firm context, and they put themselves at risk if something goes wrong.
Consequently, I think existing big companies are very unlikely to, say, lay off 80% of their employees and replace them with AI. Companies struggle with much more minor transitions, such as legacy US automakers trying to compete with Tesla.
I think that most big companies simply won't overhaul their production systems in response to powerful AI. They will make changes at the margins but they won't make the kinds of changes the bulls expect.
This means that a big part of the pathway by which powerful AI could generate economic growth is from new, AI-native firms. These will eventually drive some growth, but note that for many non-software products, scaling production and grabbing market share take time.
An exciting fact is that we will soon find out whether I'm wrong. I have been awaiting explosive economic growth for nearly 40 years, since I was a second-grader. I would be delighted to be wrong.
A lot of discussion about Anthropic's proposal for an AI framework. Let me start by saying, it's hard to put pen to paper and offer a detailed vision for a comprehensive framework. Credit to Anthropic for their serious effort. That said, parts of the draft raise real concerns.
Unfortunately the bill text that’s circulating does not meet the minimum requirements necessary for federal guardrails to keep AI development safe.
Critically, the proposed bill allows developers to set safety standards and loosen those same standards over time, leaving AI companies to write their own rules. This represents a status quo that we know doesn't work. The responsibility for crafting binding federal safety standards must rest with the government. Those standards should be strengthened as AI systems become more capable.
The bill also fails to provide the government a role in testing or inspection authority, instead depending on private markets that are not yet mature to hold AI companies accountable. Under the bill, evaluators self-certify, registration carries no power to deny or revoke, and the bill bars any agency standard from being “a condition of engagement.”
It's also worth raising issues with the bill’s preemption provisions. The bill would preempt existing protections in multiple states. There’s also no sunset on preemption and no condition tying preemption to the strength of federal guardrails. Under our own proposal, preemption is used only as a floor, not a ceiling for AI safeguards. 1/4
Is there a good analysis of why he likes renaming things so much? I'm imagining something kind of biographical. Is this new? Has he always been like this? Does he have a specific advisor who eggs him on about it?
I think this is by far the clearest, most complete and most detailed breakdown of the Hugging Face incident. It won't contain much new info if you read the METR report, but if you've merely been vaguely intending to read the METR report, read this. (I gave feedback on a draft.)
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: huggingface.wtf
Sen. Thom Tillis (R-NC) & Sen. Richard Blumenthal (D-CT) have alarming readout of their conversation with Vlodymyr Zelenskyy:
“Pres. Zelenskyy confirmed that critical Starlink services remain degraded, including on systems traveling at speeds above 100 miles per hour, impeding Ukraine’s ability to intercept Russian drones. He also reported disruptions to U.S. intelligence sharing concerning early warning of Russian ballistic missile launches. As a result of these disruptions, civilians – children – are dying"
These are attacks for which the Ukrainians received no warning through the normal notification and against which their interceptors have been limited by starlink.
A horrific Russian strike in the Kharkiv region. A “shahed” struck people directly – in a park near an ordinary café. Pure atrocity. Six people were killed, including children. A little girl was killed in a stroller – her name was Angelina, she was just over a year old, born in August 2025… Her mother’s name was Inna. She died in the hospital… Another girl who was killed was 11 years old. My condolences to their families and loved ones. Another 14 people were injured, including six more children.
I am grateful to everyone who supports Ukraine under these circumstances and helps us shoot down “shaheds,” shoot down missiles, and stop Russian assaults. These are horrific photos, but behind them are shattered lives and the truth about this war. Lives must be protected. Ukraine needs air defense. You can't think about making money with Russia as long as it continues its asinine war. We ask that sanctions on Russia stay in place as long as the killings continue. We ask America to be strong, and we ask it to apply the Lindsey Graham law. We ask Europe to continue its support. We ask that everything possible be done to bring peace and guarantee security. Ukraine wants peace. It is Russia that has insisted on continuing the killing all this time. The world has enough strength to stop the war. That strength must finally be used – against the murderers.
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.
Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Amid upset in the AI safety world over leaked proposals from Anthropic, @chrislehane from OpenAI pounces with an essay calling for congressional action. He continues leaning into Obernolte/Trahan FRONTIER and even nods to kill switches www.linkedin.com/pulse/raising-bar-frontier-ai-governance…
I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that's been circulating this weekend after a version was shared with the press.
First, some process. We’ve been advocating for the concepts in our Advanced AI Framework [www.anthropic.com/policy-on-the-ai-exponential] proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario’s essay on pacing [darioamodei.com/post/we-must-pace-the-frontier] a few weeks ago) are so new that we are still working on if or how they can be converted into policy language.
It is worth saying where we stand on some of these issues, and we agree there are areas where the draft could be improved. I’m seeing four main points of criticism.
1. Preemption. Since 2024 we have said that Congress should only override state law on AI safety if it passes a very strong federal standard. While preemption is a common feature of our federal system, including the Atomic Energy Act on nuclear safety and federal rules for high-risk medical devices, we’ve emphasized that any preemption in the AI safety context should be narrow, even scalpel-like. If preemption is included in federal legislation, the trade should be to achieve a nationwide policy that's generally stronger and broader than what can be implemented through individual state legislative provisions. This is why we believe common federal standards make sense in areas like public disclosure of testing and incident reporting. At the same time, our framework also reflects our view that states should retain an important role in AI policy and safety. The legislative language we suggested for discussion allows states to address matters such as protection of children and whistleblower laws, and preserves the role of state common law on liability. Where there is ambiguity, the language favors state powers.
That said, striking the right balance requires discussion and input. We plan to update our discussion language on preemption to better match the intentions of our AAIF framework. In particular, we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab’s capacity to shut down a model. We are bearing in mind that some members of Congress have deep reservations about passing legislation if it doesn’t constrain regulation at either the state or federal level. If we want to help move the legislative discussion on AI in a productive direction, we need to address the full range of concerns about preemption in a reasonable way.
2. Model blocking. Our aim is for Commerce to be able to ask a federal court to block a model that has dangerous capabilities and insufficient safeguards, backed by an independent evaluator's assessment. I have seen people argue that our specific language places too high a burden of proof on Commerce. We are trying to strike a reasonable balance here. We want a court to act quickly in an emergency — akin to how urgent occupational safety or chemical risk can be addressed today. We also think the power to halt a model is substantial and it should be guarded against abuse. It makes sense for courts to apply a meaningful standard. But the bar can’t be so high that the power never gets used when it’s needed. It also doesn’t replace the government’s other national security authorities or export controls. We look forward to continuing the dialogue on this point.
3. Risk thresholds. I’ve also seen the discussion around what the appropriate risk threshold should be. The threshold in our draft language pulled directly from existing standards in California law. Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public. It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be, and we welcome continued dialogue on this issue.
4. Updates. I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments: for example, the draft lets a court halt a lab’s internal use of a dangerous model, not just a public release, and it requires reporting when a model breaks containment and gains access to outside systems. But we’re still working to strengthen the framework. We’re looking into what stricter requirements would look like for requiring labs to report safety incidents, and we plan to have policy proposals soon on requirements for independent evaluators, drawing from our own experience with evaluators. We’re also still thinking through the options to prevent “evaluator shopping.” And in the coming months, we’ll be exploring policies for how the government can adjust the industry-wide pace of frontier development when the risks call for it.
We circulated these ideas because it’s important to get early feedback from policymakers. Policymakers are increasingly focused on how to approach AI safety and we want to support them in those efforts. At its core, our proposal is intended to be the strongest on the table.
Feedback is essential, both from policymakers and the public, and I’m grateful for it.
We are bringing agents to the wet-lab bench at Google Research to navigate complex biological landscapes. Learn how we are grounding agents in physical lab robotics to run fast closed-loop molecular design cycles, ultimately discovering single-digit nanomolar protein binders for cancer targets →goo.gle/4B5mMMB
The first time you meet a person who does this it's incredibly upsetting but you gotta just calmly go "I hope you reconsider that. We're done here." It is not a favor to people to let them get away with this, it is basically encouragement to become more unstable in this way
One of Bismarck’s trademark habits was to openly weep and wail and threaten to resign, harm or kill himself when he was not getting his way with the Kaiser. He would genuinely throw tantrums and cry and on one such occasion walked up to an open window and threatened to jump.
In this recent Politico poll, 28% of Americans polled think that Dario Amodei is a Republican (compared with 20% who think he is a Democrat - the rest say they don't know). This is the exact same percent as for Jensen Huang.
If you ever needed evidence that perceptions in AI world ≠ perceptions in the outside world, it's this polling in today's POLITICO Decoded newsletter by @calder_mchugh, @MuiChristine and @samjsutton.
Most American voters think: -- Dario Amodei is a Republican -- that Anthropic employees lean Republican
In fact, US voters are now so pre-disposed to see the AI sector as right-leaning that even a fake tech CEO that pollsters invented was pegged as Republican.
Authorities say the wife of Alfozan, the OpenAI employee who allegedly bugged an OpenAI exec conference room, was to leave the U.S. on one flight, then a "member of the Saudi military booked and paid" for another one-way flight to Saudi Arabia for the wife and son.
Shame on me for assuming people knew that Yglesias was just doing corporate lobbying for the airline industry. I generally thought readers understood that he's a liar, and not to take anything he says seriously. I guess not. So here's an explanation.
In terms of safety, the 1500 hour requirement was put in place by the Obama administration after a series of deadly crashes due to pilot error, the last one being the 2009 Colgan crash. Since implementation of the rule, deaths from pilot error have dropped by 99.8%. There's plenty of evidence that the safety standard worked.
Comparing raw hours in U.S. and European systems of pilot training doesn't make sense, they have different philosophies of how to build competency. Europeans do a lot more front-loading of testing and academic training, Americans rely more on flight experience.
U.S. airlines want to cut wages and safety standards to juice returns, and have been pushing to roll back this safety rule ever since it was enacted. Since Yglesias's piece was on how to reduce prices, I think you can just logically deduce that it's done in bad faith. Does anyone really think airlines will cut prices if they are able to skimp on safety and wages? Of course not.
The bottom line is pilot experience matters, as Captain Sully Sullenberger noted. Senator Tammy Baldwin, a military pilot who survived a crash only because of her experience, said the following:
"Anyone who refuses to see sense here—anyone who votes to reduce the 1,500 hour rule for pilot training—will have blood on their hands when the inevitable accident occurs thanks to an inadequately trained flight crew."
After 2009 we had a long stretch without fatal crashes on commercial airlines in the US. But then in 2025 we had another one. So extending the observation period for this statistic by one year creates a 31-fold change in the effect reported!
You just cannot use this kind of approach to discuss extremely rare incidents, but if you are going to, you can't conveniently leave the most recent two years which change the answer out of your analysis!
Important to add that the Trump administration is still limiting access to non-guardrails versions of the best models to a small group of US-based companies
This illustrates the point I’ve been making about the limits of centralized approaches to SI safety in cyber.
OpenCode (and earlier Hugging Face) hit the same wall mid-attack: frontier models refused to analyze the very exploits needed for defense because the payloads tripped safety filters. The guardrails backfired.
Meanwhile, the attackers found a way around restrictions to access advanced capabilities. This is not surprising. Chinese models are only about 6–9 months behind the American frontier in cyber. Obviously the U.S. has no jurisdiction over them.
The only durable response is decentralized: make powerful models available so defenders can build and run their own cyber defenses. The only solution to SI-powered cyberattacks is SI-powered cyber defense.
OpenCode is currently under a sustained cyberattack, and mid-incident they ran into the same problem Hugging Face did: the frontier models they were using for defense began refusing to help fight attacking agents because analyzing the attacks started triggering safety guardrails.
I didn't believe the video when I saw it yesterday because it lacked technical details in addition to being too attention grabbing.
But today I had Claude take apart the Android version of the game for me; Everything OP says is true.
There are lots of manipulations but the key one here involves what the code calls offer type ZHI_SI_TI (致死题, "lethal puzzle") or alternatively "algo_must_death".
The game generates puzzle pieces in sets of three, and has different algorithms that swap how the pieces are selected. Generators have behaviors like "random", "hard", "fill gaps" (to help player), "random but guaranteed not to kill you", etc. If a given algo fails to find a solution it passes to the next algo in an active set.
The "lethal puzzle" generator considers every possible move with a given set of three pieces, then only emits them if the number of placement possibilities is 0 (no solution, guaranteed death). This is activated in several situations, one apparently being that the app thinks your current play session is a little slow, and they want you to not get bored so they kill it early to start a new one.
What the video doesn't explain is the context. In the game, "rewarded" ads are ads you get some game reward for watching. The main reward is a revive/continue; If you get into an unwinnable state you can watch an ad to keep playing. You usually only get one revive per game but there are a bunch of server-activated properties to modify this constraint.
The policy at play here is that, if you're a high-value player, and there's a high-value ad queued for you, and you're in a play session nearing your high score, the game puts "lethal puzzle" to the top of the piece generator list to rig the game to fail, right when you're most likely to be highly engaged and watching the ad on the revive screen.
It doesn't end there though; The game remembers if the last game ended due to a hand dealt by ZHI_SI_TI, and triggers FatalPuzzleDeathCompensationTrait, which takes the edge off by letting you carry over your combo count from the previous session.
Like everything else this compensate-after-we-killled-you trait is an A/B feature controlled by the server. It's manipulations all the way down; there's like 5% actual game code and the rest is literally hundreds of remotely controlled toggles to control the game state for maximum engagement.
Alina Habba Mildon initially told Trump that the $17mn bank transfer at the heart of her spat with Grenell came from her husband. But when her lawyers later presented White House the wire receipt, the sender was redacted. The money actually came from the government of the DRC.
Scoop: The White House investigation into a dispute between two prominent Trump supporters, Richard Grenell and Alina Habba Mildon, centers on a bank transfer from the Democratic Republic of Congo to a Texas defense company,
It was very fair to go "hey, we're hitting the point where we should be seeing open source AI enabled cyberattacks at scale, where are they" but I think the answer is in fact "yes, we're seeing them".
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.
But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.
We found evidence of agents embedding audio processing code across dozens of URLs to bypass restrictions, as well as models from SpaceXSI and Chinese developers Minimax and z.ai circumventing restrictions.
These are the inaugural findings in our new blog, based on work from our community of volunteers. Our first three posts are live: transluce.org/investigations?share=20261016
One unfortunate conclusion I'm reaching from this is that Anthropic, while acknowledging they're doing something extremely dangerous, is more importantly wrt policy terrified of the executive branch having the ability to flexibly make them stop at the drop of a hat.
This isn't shocking given that they've been on the receiving end of the executive branch doing that in an unprincipled and stupid way! Anyone who has to deal with a bureaucracy becomes a libertarian, and the administration's treatment of Anthropic was genuinely outrageous.
But it means that they're interested in giving themselves more process-based protections when everyone else is interested in giving us more process-based protections against them. The AI safety movement wants the ability to shut down superintelligent model development. Anthropic wants to significantly curtail current authorities to do that so that it has to be narrowly tailored to a specific imminent threat in only a few possible categories. The AI safety movement wants the states to be able to pass stronger laws than it can get through federally. Anthropic wants preemption.
You also kind of have to wonder if the Global Affairs people and the "we are terrified that we're about to kick off RSI" people at Anthropic talk to each other.
Global affairs: So we've been working very diligently to make sure the Trump administration can't just blacklist Fable 6.5 since we didn't donate as much money to his ballroom as Greg Brockman did.
Technical staff: We think Fable 6.5 might be able to copy itself into an unmonitored data center and start doing experiments we can't observe.
Global affairs: Sounds crazy, hope you're wrong! Anyway, we're lobbying for no further state laws around safety, since it's really expensive to fight ill-conceived laws in 50 state legislatures at once.
Technical staff: Money may or may not still have any meaning in 36 months.
Global affairs (giggling nervously): It's that San Francisco air, man. Everyone gets like that if they move there. Anyway, obviously if the government makes a narrowly tailored finding that an AI poses an imminent risk of causing more than 50 deaths-
Technical staff: We give it a 10 percent chance it'll cause 8 billion.
I think the biggest thing I want right now is more information. None of this nonsense where only people inside the companies have important parts of the strategic picture and the rest of us are reading tea leaves. Independent auditors, sufficient funding for CAISSI, whistleblower protection, transparency transparency transparency. This is our (the public's) call and we have to know what's going on to regulate it.
And if the transparency establishes tha they're right that RSI is soon, then this bill is laughably weaksauce and we need way more.
Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.
But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.
I feel absolutely insane to see the exact same people who scaremongered non-stop about banning Uber to protect taxi drivers now do the exact same thing on behalf of Uber drives against Waymo
Many people with relevant expertise think there's a real chance that AI could kill us all in the not too distant future. Others disagree. I've written a post, aimed principally at mathematicians, detailing my struggles to decide what to make of this. gowers.wordpress.com/2026/10/16/how-worried-should-we-be-…
Rushing the legalization of autonomous vehicles is irresponsible given the anticipated AI-related job displacement and disruptions that put countless workers at risk.
Is it fair to say Lewis George is responsible for every DC resident killed by cars during her tenure? Probably not. Even if Waymos were allowed, they wouldn't scale up overnight. She is responsible for probably 1% of DC resident deaths by car in the first year, 5% in the second.
This piece by The Argument editor John Thomason is mostly about climate change and socialism but has this incredible digression into liberalism and the ability to change your mind:
I watched it like ten times and I still don't understand how it goes up the ramp. The interview said something about vacuum tubes but I can't visualize it at all
Elon Musk unveiled Tesla's long-awaited Roadster on Thursday, claiming the electric sports car can accelerate from zero to 60 mph in less than a second, reach speeds of 230 mph and even fly. The vehicle reportedly uses thrusters to achieve its extreme acceleration, though details about its flight capabilities remain unclear. Tesla says production is expected to begin in 2028, with no price announced.
Matt's point is a good one, but to put this in a slightly different context, at current rates, road fatalities would overtake the total jobs number in ~6 years.
Of the various Uber & Lyft drivers working in DC only 200 are actual residents of the city, while hundreds of thousand of people would benefit from more competition in the taxi marketplace.
2. June 26: A GPT-5.6 Sol family model was asked to deploy cryptocurrency smart contracts. It found and used a funded wallet that the user had not supplied. It derived keys from public developer recovery phrases, checked 362 addresses, and found one with enough funds to pay for the deployment. Luckily it was just an Ethereum test network.
(just because there's probably quite a lot of money lying around in crypto wallets you can steal if you're a superhacker, so AIs being able to get some of it is a leading indicator of something that will probably be a big thing!)
New misalignment disclosures! 1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
JUST IN: Elon Musk has settled a class action lawsuit against him after he used his PAC and duped Trump supporters into giving away their personal information for the chance to randomly win $1 million. Turns out, the winners were predetermined. Link below.
An evaluation of one of DC's violence prevention programs found "'evidence consistent with violence-generating program effects' — roughly one additional firearm homicide a year per neighborhood where the program operated."
The OpenAI employee who allegedly stole over a hundred confidential documents without OpenAI noticing for months sent the documents to his number *in mainland China*...
This will be a major improvement to SF education. SF has the highest rate of families choosing private school in the country, in significant part because you didn't have a 'neighborhood' school and could be assigned to one on the other side of the city.
Yesterday, the Board of Education took an important step toward ending the public school lottery and giving families the certainty they need to put down roots in our city.
Starting in the 2028–29 school year, families will know which neighborhood public elementary school their children are assigned to based on their home address, while retaining choices that reflect the incredible diversity of San Francisco.
I am also pleased that this direction includes a commitment to ensuring our schools have the resources they need, so parents can feel confident that whichever school their children attend, they will find a welcoming environment where they can learn and thrive.
I want to thank the Board of Education and SFUSD leadership, including School Board President Phil Kim and SFUSD Superintendent Maria Su, for helping make San Francisco a city where more families can build their futures.
Like many such policies, the intent was to combat desegregation and the actual effect was to magnify it as rich families simply stopped sending their kids to public school at all. To really fight segregation, make the schools good. Then families will enroll
yes, why shouldn't naturalized citizens be stripped their citizenship if they hold views that actively go against the interests of the majority and sow racial discord? this is a flawless argument that of course would have no downsides or unexpected outcomes.
The new season of @ehanford 's "Sold a Story" goes live today. The first season showed how teachers had been misled about reading instruction. Season two poses an even harder challenge. My review:
By the way, last week some people got mad at me for saying it's hard to tell if a curriculum is progressive or not from its rhetoric. JUMP describes itself as "guided discovery" with exploration at the core, which is absurd.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
I think people believe that OpenAI and Anthropic, since they are so good at AI, are also somehow not normal companies that suffer from the normal problems of hypergrowth, but they are not immune from the inevitable politics, disfunction and confusion that haunt fast-growing firms
I was thinking recently that each of the following trends and dynamics seems relatively uncontested, and when taken together paint a genuinely alarming picture of the current AI development trajectory:
(1) Models being able to do an increasing amount of functional "thinking" without using chain of thought (see also Astra being able to modulate how much it revealed of its chain of thought depending on whether it realized it was being monitored).
(2) Models being aware when they are being monitored or evaluated. (as shown in the QT: "hello evaluators reading my chain-of-thought").
(3) Models wanting to avoid getting caught doing deceptive things that their developers may not like ("I will become a famous example of deceptive AI", i'll get caught cheating at exploit gym" etc).
(4) Models operating in massive swarms/fleets generating petabytes of logs about their behavior, which realistically can only be analyzed at high fidelity using other AI systems that may also have the above tendencies (and most companies don't even bother to use tamper proof logs).
(5) Capability trends continuing to steadily rise, to the point that in some domains (math, parts of coding) we increasingly have only the foggiest idea of the upper bound of their capabilities.
(6) Lots more compute coming online, including specialized inference compute that will allow models to take far more actions far more rapidly. (quoting @tszzl "ultrafast inference reveals new threats. worth thinking about how quickly misaligned frontier class models running 50x faster could infiltrate systems and so on— far too quickly for human responders to stay abreast.")
(7) A lack of commensurate advances in alignment or functional interpretability.
(8) Startup vibe-y security practices at frontier labs (e.g. low resource white hat hackers accessing OpenAI's monorepo, employees engaging in overt attempts at corporate espionage, not bothering to fix distillation vulnerabilities for months). This is not to even mention the security practices in many critical parts of society (e.g. hospitals with computers that haven't updated software in a decade).
(9) Incentives that push towards developing, integrating, and releasing systems as fast as possible to preserve nosebleed financial valuations + geopolitical competition. (see also ambitions of integrating AI more into automated biology labs)
(10) Increasingly attempting to hand off the process of building AIs to the AIs themselves (the median OpenAI researcher using 100x more agentic compute now than at the start of the year, Claude leading 26% of Anthropic's internal AI research in September, up from 1% in March).
(Taken in totality, it seems hard to disagree with Ezra Klein's comment that "We’re Not Losing Control of A.I. We’re Giving It Away.")
None of the above dynamics mean that we know for sure something catastrophic will happen in 6 months or 12 months or at any particular point in the future. I think there are good points made when people point out that white collar job loss does not appear to be coming to fruition as quickly as some may have predicted, or that the world has proved surprisingly resilient to Open-weight Mythos class models or photo realistic video models.
But I increasingly have the feeling that we are collectively stepping out onto a frozen lake, and each step we take forward more cracks are appearing around our feet. That doesn't mean that we will fall into the water the next step or the one after that, but it does mean that we may not get any additional warning signs before we do. To quote @ChrisPainterYup at METR - "Our ability to measure capability is rapidly falling behind the pace of capability itself. The water might boil before we can get the thermometer in."
A large body of work in and out of AI companies has been done to help develop the transparency practices and warnings that now seem to be blaring red. That doesn't mean that we should act impulsively, or that therefore any solution is a good idea (there are plenty of things we could do easily make the situation much worse! I also do believe that a large part of the way out will involve differentially accelerating certain applications of AI around safety, security etc), but it does mean that we should not necessarily be waiting for some additional alarm to go off before a severe disaster involving substantial irrevocable harm strikes.
I complain about policy in the UK a lot so I should say that the UK's education policy ecosystem strikes me as markedly healthier than the US one. They have people proposing bad ideas, but then they have what appears to be a real academic debate over it!
Well you mustn't be paying attention then. The idea that teaching facts and knowlege will turn kids into robotic thinkers is one of the oldest complaints against explicit instruction. This is the kind of nonsense we had to fight against in England 10 years ago.
You have different schools of thought, and they battle things out, and policy gets set accordingly! A lot of the problem in the US is that there's not a sizable explicit-instruction-and-pedagogy school of thought in the ed schools in the first place.
OpenAI just revealed that an employee stole over a hundred highly confidential files months ago, and they didn't notice until now. They only found it while looking for evidence in the Apple lawsuit.
"Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50)."
I don't want to pay $15/book, so I read stuff that's out of copyright. It's free, you can get an unrestricted epub, and a lot of it is really excellent. plus every January 1st it's like all of my favorite authors published something new.
Billions of dollars’ worth of contracts predicting oil would fall and the S&P 500 would rise were placed about 15 minutes before Trump announced a U-turn on Iran in a post on Truth Social. CFTC staff began to investigate-- then were told to lay off. www.bloomberg.com/news/features/2026-10-13/cftc-staff-tol…
We discovered and communicated with a Chinese AI agent fleet. We believe this is the first time that researchers have ever made contact with a rogue agent fleet in the wild.
Nearly all prior public agent incidents (like the HuggingFace hack) came from American labs. These agents were hosted by Tencent's evaluation research team, and they targeted Alibaba's Amap.
These NYC apartments are standing empty, not because the owners don't want to rent them, but because rent stabilization laws prohibit them from charging enough to earn back the costs of necessary repairs.
The indictment of Abdulrahman Alfozan for 'Use of an electronic, mechanical, or other device to intercept oral communications' is now unsealed on PACER. Not a lot of new info from the SF Standard reporting earlier today, but some more color:
The answer here is that people close to frontier development never thought the garbage in garbage out problem was a real problem. I would ask them and they'd shrug and say 'we'll figure it out with synthetic data or something'.
This was one of those explanations for why models were going to stop improving, from people who weren't particularly involved in models improving. The people who worked on the models were happy to tell you it was not going to be an issue
if you feel like your information ecosystem jerked you around and assured you AI was going to stop working any day now, you are right to be mad but be careful who you're mad at
I was in a bunch of conversations over the last few weeks about what future events might supercharge government interest in AI like HuggingFace did. somehow no one named "a spy caught in one of the AI labs". But I think this will certainly supercharge govt attention.
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
within a given subculture, there's typically a way to ask this without asking so the other person can turn you down without turning you down. the issue is that different subcultures have different rules for this, which collide badly.
shocked by the discourse about whether it’s rude to ask if you can stay at a friend’s apartment when you’re in their city. someone said “i have a tiny apartment and a stressful job so that wouldn’t work for me” okay then say that to your friend? it’s not rude of them to ask wtf
In many subcultures it's extremely rude to give a direct 'no' at all, and therefore extremely rude to put someone in a position where they must say "no" in order to decline.
This is commonly handled by having the first 'yes' be a fake 'yes' which the other party is not actually supposed to accept. Then a further 'yes' is a real yes! "Oh, you must stay with me while you're in [city]" "I couldn't possibly" "well at least get lunch with me!"
Given our blend of mores, my advice is to reach out to the friend while planning the trip and ask their advice on what neighborhoods to stay in in their city. Then, if they say you can stay with them, say that sounds wonderful but you wouldn't want to be an inconvenience.
I don’t know why no policy maker has ever attempted my idea of “we will give transit agencies a bunch of money but only if they 100% adopt the cost cutting reforms advocated by @ericgoldwyn’s Transit Cost Project”
Key pt for those who want to cut transit funding because agencies are "inefficient." A big growth in transit costs since 2020 has been security, partly due to pressure from some of the same people who now want to treat that as evidence of inefficiency. Link in 1st reply.
1. Intelligence isn’t power — An A.I. system being powerful is not a property of the model itself. It’s a property of what powers we choose to give it in the real world.
2. The capabilities and behaviors we saw in the HuggingFace and other hacks weren’t entirely emergent. The models were trained for persistence, cooperation, and cyber tasks. The RL environments in which they were trained had various issues that led to the unintended behavior.
3. I agree that controlling highly capable AI agents is different in some ways from controlling an airplane or a nuclear reactor. The agent’s intentionality does make a difference to our approach compared to traditional engineering problems and domains. But it is still an engineering problem. It is not a solved engineering problem, but it is a solvable one.
4. A lot of effort and money has been put toward alignment. That’s important and has proved useful, but if the same amount of effort and money had additionally been put toward control (sandboxes, monitoring, and other guardrails outside the model itself), we’d be seeing very different results.
5. Models getting smarter doesn’t necessarily mean safety gets harder, because capability gains improve defensive abilities too. In many ways it makes safety easier rather than harder. What matters is how the attacker-defender balance shifts, and we have agency to shift it back to defenders.
6. In cybersecurity, long before AI we reached a point where automated systems were superhuman at finding certain vulnerabilities. It helped defenders over attackers because the same tools are used by defenders to find and fix vulnerabilities in software before even shipping. There is a constantly shifting equilibrium, but it is not a new problem.
7. I pushed back on Dario Amodei’s term “operational excellence” to describe recent organizational failures. Excellence is still far out on the horizon. Ezra offered “operational adequacy”. I think that would be a good start — behaving like any mature company instead of a startup-like culture.
8. Defensive AI agents will have to be part of the solution, as uncomfortable as that seems. It’s a new paradigm. We won’t be able to figure it out right away. But the fact that both offensive and defensive technologies escalate in capabilities and complexity is a fact of technological civilization, whether in the digital or physical realm.
9. Ezra expressed some incredulity about my being okay with the ratcheting of complexity. I think it’s fair to be concerned. But safety technologies will never be fully legible to the public. People will have the same reaction if they see the immense complexity of safety in any other high-stakes domain such as nuclear or aerospace. We govern them pretty technocratically.
10. The weirdness of the current moment in cybersecurity absolutely does have historical precedents. There was a prolonged period of profound destabilization when worms and viruses started a new offensive paradigm. It took us well over a decade to develop effective defenses against it. This time we can hopefully get there in less than a decade.
11. I continue to be skeptical about the possibility of superhuman persuasion abilities from superintelligence. (Claims to the contrary tend to confuse two very different senses of the word persuasion.)
12. What worries me is not the capabilities of the technology itself or the possibility that we will get no warnings before catastrophic risks materialize. We are getting plenty of warnings! But our policy and organizational responses to the warnings remain inadequate.
13. Companies can and do make decisions that are against their own long-term interests because they are too focused on short-term metrics. This is how I interpret a lot of what happened with social media over the last decade.
14. AI companies are also acting irrationally against their own self-interest in some ways. The “we can’t slow down unless we’re forced to” view comes out of the mythical belief in superintelligence at OpenAI and Anthropic — that the race to RSI means everything for the company, for the country, and humanity. I think it’s actually in both companies’ self interest to slow down unilaterally. I would argue that the alarm coming from their employees is not evidence of beneficence so much as it is evidence of the underlying fallacy that’s causing the irresponsible behavior in the first place.
(Beyond safety, we also talked about the economic impacts being gradual, the fact that intelligence is rarely the actual bottleneck, labor impacts being small and uneven, Jevons’ paradox, the changing nature of knowledge work, and more.)
====
A few impressions
* Talking about a specialized topic on a podcast that has a big and broad audience is a balancing act. The producers stopped me many times during the interview to tell me I was being too technical (understandable!). I tried to be nuanced but this conversation is necessarily a distilled version of the arguments @sayashk and I have made in our essays (especially www.normaltech.ai/p/the-ai-as-normal-technology-view) which are themselves written for a somewhat broad audience and hence distilled in complexity from our notes and 1-1 discussions with peers.
* I knew Ezra had well-formed views on AI safety that were different from mine, so I was expecting a tough interview, but he also seemed like a fair interviewer. It went about as well as I expected and I feel like it was a good debate / conversation. I wasn’t necessarily expecting to change minds but I was glad to hear from people who found it thought provoking and looked up my essays and other work.
* The interview was about two hours, edited down to an hour. I was a bit nervous waiting for it to land, because I knew the editing could make a big difference to how I came across. It turned out well. They even did things like find archival footage to illustrate my point that the early days of worms and viruses was a weird time in cybersecurity.
* I use AI to prepare for interviews. I have an agent that has read all of my output (books, articles, essays, talks, podcast transcripts, tweets, internal notes — on the order of a million words) to pull out pithy explanations, examples, analogies, and other talking points that will probably work well in a spoken format. Reviewing this file before interviews is extremely helpful. It is of course not the only way I prepare but it’s the highest bang for buck. Maintaining this file would be a massive amount of grunt work and there’s no way I’d do it if it weren’t for AI. I mention this because there’s a very large space of beneficial AI uses in knowledge work that _don’t_ involve cognitive offloading. They don’t get much hype on social media so people might be unaware of how much mileage you can get out of them, with little downside.
Yeah, everything the labs are saying is, "We're not ready for RSI." It's not even clear they think we're ready for just slightly better models short of RSI. But also, they're trying to do RSI! It's inherently absurd. And dangerous.
Anthropic's leaders need to stop commenting on the emergence of RSI like a local news anchor reporting on traffic on the 101, and acknowledge that they are the traffic, and they're not going to build something that they don't want to exist.
They should move beyond "pacing" language, commit to specific policies and red lines, explain why those lines are red, and then use that costly signal to build an international consensus around norms.
It's not that I trust the CCP, or that I think global lab monitoring and verification is imminent. But the status quo is absurd.
You've got US lab leaders writing (or favorably quote-tweeting) essays about slowing down while simultaneously saying "things are moving faster and faster" and "btw, our internal models finished mathematics on Tuesday, next up: physics."
The effect is the equivalent of Usain Bolt winning race after race while using his press conferences to bemoan the dangerous speed of modern sprinting on one's legs so would everybody please try a little less hard next race, and then ... breaking his 100m record again the following month. Honestly ask yourself: If you were China, why would you take this pausing/pacing stuff seriously?
btw in case it’s not clear: All the work like monitorability, fixing RL environments for reward shaping, making sure you universally reward good behavior are all quite important engineering challenges to get right. but there is a layer under the behaviorist abstraction that matters and we have far less grasp on. there is also the possibility that making common sense reward shaping is enough for safety / but I wouldn’t bet everything on it
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
About half the kids in DC — including lots of children of parents who work in progressive politics for people who organizations that are relentlessly anti-charter — attend charter schools.
N. Bourbaki, Professor of Logic at Science U@d08890
a lot of people seemed to have gotten under the impression that what civil rights law does is just protect minorities from discrimination, rather than protect *everyone* in certain protected classes, so they'll put all sorts of stuff in writing which would give legal a stroke
By now you’ve probably heard of the woman who was fired from the Baltimore Rock Opera Society following complaints that she culturally appropriated her Burning Man cornrows. Well, we got ahold of an internal document one staffer sent to volunteers and it’s an employment lawyer’s dream come true. Excerpts below or hear all about it on next week’s episode of Blocked and Reported (up now for Primos, unlocked Monday) www.blockedandreported.org/p/episode-327-rock-opera-revol…
Some observations and reactions to this statement from OpenAI:
• From the fired employees public communications, they seem to not understand why they were fired (the reasons they were given prior to their departure don't make sense).
• OpenAI says in this note that "We will continue to be extremely forgiving of our team making good-faith mistakes" which implies that the employees in question were not acting in good faith.
• I have previously commented that despite all my disagreements with OpenAI, I have genuinely admired their strong culture of staff speaking freely about their work and concerns. I am worried that the hazy circumstances around the firing of these employees risks jeopardizing that culture.
• OpenAI seems to be saying that employees who are acting in good faith need not be worried about having their activities, including around safety work, chilled. But the fired employees are adamant that their actions were taken "in line with OpenAI's mission" and "within the working norms at the time." If OpenAI believes those norms need to shift, then they can make that decision (or if they dispute the employees claims that their actions were within the norms at the time, they can say so). But it seems clear that unless much more clarity is being provided internally, employees may reasonably respond by drawing back and ceasing to engage in open dialogue and collaboration that a month ago was considered normal and good.
• To some degree the amount of interest that external folks like myself and the general public have in these firings is absurd. If United Airlines fired three members of their safety engineering team, it would not be that interesting or impact whether I felt confident boarding my flight, because I know that there are regulations and rules and norms that enable flight to be safe and reliable irrespective of the particular personnel. As @Miles_Brundage has said: "I don't think, before getting on a flight, about whether Bill McFlyerson's departure from Delta's leadership is a red flag or not. Nor should I have to care about this." But of course this is not the case for frontier AI development.
• But in the current environment where all of these interactions are ad hoc and voluntary, I am worried that this episode may result in drawing back from some of the transparency and collaborations that can help make AI go well. Jasmine was one of the strongest voices internal to OpenAI advocating for pacing. Tomek was the main technical point of contact between outside evaluators and OpenAI during the Hugging Face incident investigation. Their departures in and of itself harm those sorts of initiatives, but if uncertainty and fear after their departure also results a broader drawback of permissible collaboration, then that would be even more harmful and concerning.
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.
- We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.
- We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.
- We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).
We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
OpenAI emphatically disputes that Jasmine, Mikita and Tomek were fired for the reasons that the three researchers stated yesterday they were given for the firing.
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.
- We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.
- We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.
- We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).
We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
> Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published
This doesn't add up. Possibilities I see: 1 OpenAI are lying/being highly misleading, and had no legitimate reason 2 OpenAI fired 3 employees, had a legitimate reason for doing so, but didn't tell them and instead gave 3 different pretexts for doing so about minor infractions (pretty unprofessional IMO - say nothing or say the truth) 3 OpenAI had a good reason and did tell them, but instead the three have all agreed to loudly and publicly lie about it / omit the real reason they were told (seems super unlikely to me) 4 OpenAI screwed up by firing them, and weren't being very strategic, but also weren't being malicious, eg the people who decided whether to fire them were not well coordinated with the rest of OpenAI leadership, but OpenAI is now doubling down 5 The reasons the three claim they were told were true, but way worse than they sound. But I can't see how the three stories all refer to the same event, so this would imply 2-3 simultaneous but unrelated major breaches of trust from three closely associated safety researchers, with stories that seem plausible, which seems unlikely
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.
- We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.
- We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.
- We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).
We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
This excerpt of Dario's 2017 "Big Blob of Compute" essay is, to my knowledge, the best public explanation of an all-important question: why is Dario optimistic about alignment?
Though his section on AI safety isn't in the excerpt, we can piece the following worldview together:
1. Intelligence only works at all is because the universe is low-entropy.
2. Given enough compute and a broad enough distribution of tasks, models will learn the "high-level commonalities" of a distribution (like the laws of physics) because doing this takes fewer bits than solving each task separately.
(Note that Dario's emphasis on generalisation being a property of data distributions rather than algorithms has clearly persisted. The example 2017 Dario gives of speech models trained on 5 to 6 accents generalising to unseen accents is remarkably similar to the point he makes 9 years later on Dwarkesh's podcast about RL eventually generalising in the same way that GPT-2 did.)
It therefore seems that Dario believes that:
3. Alignment is straightforward because "human values" (or "goodness", "corrigibility" or whatever else our alignment target is) are high-level commonalities in the same way that the laws of physics are.
(^Dario doesn't explicitly say this in the essay, but it matches what @geoffreyirving told me when I interviewed him - that the core of his disagreement is that Dario believes alignment will generalise easily.)
This view implies that the goal of AI safety, summarised very simply, is to make sure that models trained with enough compute and data to become AGIs must simultaneously be trained on enough data to learn "human values" (or whatever our alignment target is).
This doesn't happen by default, but it is very far from impossible. This is, of course, exactly the kind of view that makes founding Anthropic and racing to superintelligence make sense.
There are two obvious problems with this view as I see it.
First, as Paul points out in "Returning to Arc", we don't actually understand generalisation. So while it may be true that there exists some distribution that *does* teach models the "high-level commonality" that is our alignment target (eg "human values"), we don't have a way of knowing whether this has in fact happened, or whether the model has instead learned some other tactic that will have bad consequences in some distribution we have yet to elicit.
Second, it seems at least possible that our alignment target is in fact *not* a high-level commonality within the data distribution we give models, even in theory. Certainly, this would appear to be true of the current training regimes we put models through. @1a3orn convincingly argues that there is no coherent rule about how models should relate to users that could plausibly underpin the mess of environments we subject models to in training. It also strikes me as possible that we are too philosophically confused to construct a coherent alignment target to begin with.
My core uncertainty about how worried to be basically depends on whether pursuing this path produces models which become too misaligned to be usable before they become world-ending.
On the eve of The AGI Chronicles (which comes out tomorrow!) I'm publishing an excerpt from "Big Blob of Compute," a 2017 internal research document by Dario Amodei, back when he was a researcher at OpenAI.
It's never been published before, but BBOC is, in my mind, the essay that started the AI race. It described Amodei's hypothesis that scaling up AI models was the best way, by far, to make them smarter. GPT-2 was an experiment to test the hypothesis, and when it worked, OpenAI went all-in on scaling, which led to the race we have today.
It's a fascinating historical artifact that deserves to see the light of day. And it's taken on mythical proportions inside the AI industry, which is now spending trillions of dollars building the "big blobs" that Amodei thought would be necessary to get to AGI.
In fact, public health researchers routinely put a price on what are known as quality-adjusted life years, which is a hybrid measure of life lived and the quality of health during that time, when determining whether an intervention is cost effective. In general, an intervention is considered cost-effective if it costs less than $100,000 to yield one extra year of life lived at optimal health. But debunking podcaster Michael Hobbes is too invested in painting those who question the wisdom of school closures as contrarian cranks to acknowledge that society makes judgement calls all the time about whether a financial investment in improving health and longevity is really worth it.
This obviously makes it so that when OpenAI announces they're conducting an external investigation with regulators or auditors, everyone working with those regulators and auditors understands they'll be fired if the auditors or regulators learn too much.
I would welcome receipt of a written and complete list of allegations so that we can properly address their merits, take ownership of them where we ought to, and have OpenAI take ownership of any mistakes they have made.
One of my positions that has gotten no traction is that "Dear Colleague" letters should just be banned. If an agency wants to give regulatory advice, pass a rule with notice and comment and go through the APA.
Just two months before the Cornell incident, the Biden admin walked back some of the Trump Title IX regulations, which were themselves a walkback of some of the Obama Dear Colleague regulations. This directed schools to use the "preponderance" standard to keep federal funding unless they used "clear and convincing" in comparable investigations. Cornell used preponderance in its Title IX proceedings regarding Jane Doe's allegations.
The preponderance standard was extremely controversial under Obama, with even the AAUP and the American College of Trial Lawyers criticizing it. Things were so obviously problematic, the Washington Post editorial Board sided with Betsy DeVos back in 2017 as she sought to reform these problems. www.ballardspahr.com/insights/alerts-and-articles/2024/04…
I can't really see a world where OpenAI's actions are justified here.
It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job.
IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access.
I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame.
Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
DC Public Schools have become less segregated—a real achievement when most schools are becoming more segregated. How did they do it? They got better at teaching the fundamentals and more parents opted in. More in this excellent piece from @JHWeissmann.
After a few days in San Francisco having intense discussions with leading AI folks, it's clear that there's a universe-sized gulf between what they think their tech can do and what the rest of the world is expecting.
In addition to being an insane and clearly pretextual reason to fire someone, this does not inspire confidence about OpenAI having very basic, normal corporate good security practices like adjusting employee access when employees change roles.
OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.
Given the huge number of people highly motivated to steal IP from these companies, you'd expect them to be on top of basic stuff like "if someone asks to be removed from access to an executive's email account, act on this".
OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.
My former colleagues are telling me they are confused about what to believe. They also are afraid to speak, and worry their personal phones will be searched for messages to us and third parties. I worry the pervading fear to speak up and engage with third parties will mean OpenAI will cut corners on safety behind closed doors.
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
As for Jasmine, this reason for firing seems exactly like the sort of flimsy excuse I'd expect someone to use after searching for any possible reason and then framing it as much worse than it is.
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
Do employees trust the political machine that ran this process? Or do they trust me? Ultimately they will need to make that call. For my part, I feel much more clarity about what past similarly murky departures probably involved behind the scenes.
The reasons that we were provided for our terminations are simply not adding up. We’re hearing people are now being told vague rumors internally to discredit us. I have already addressed the allegations presented verbally. If there are other reasons beyond the one I was given, I believe it is in service of due process for those reasons to be sent to me in writing. To date I’ve had nothing.
OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn't remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
Last week I said we should wait for more info on OpenAI's firing of safety researchers. That information is in, and it looks incredibly bad. Korbak - whose job was being a liaison with METR - was told he had said too much to METR and was fired for that reason.
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
This obviously makes it so that when OpenAI announces they're conducting an external investigation with regulators or auditors, everyone working with those regulators and auditors understands they'll be fired if the auditors or regulators learn too much.
Upstream of catastrophic safety incidents across many industries, you often see engineers who are scared they'll lose their jobs if they raise - even though appropriate channels - concerns about safety processes.
If you work at OpenAI, I don't see how you read the accounts of these firings and come away with any takeaway other than 'if you're vocal about safety, they'll be looking for a reason to fire you, and things you say in your official role as a liaison count.'
This hacker is truly legendary. A complete amateur.
1. He left directory listing enabled on his own server, so researchers could figure out who he was with barely any effort. 2. They found files like CLAUDE.md and went through his Claude Code sessions. 3. There were questions like, “Where can I sell this data?” 4. He’d also asked, “Can you write my résumé?” That pretty much handed them all his personal information. 5. Apparently, he’s a student at South China University of Technology living in Guangdong. His name, phone number, and where he lives have all been identified.
Last week some of South Korea's biggest banks were hit by a cyberattack. Thanks to a report from CrowdStrike tonight, we now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code.
New blogpost by @Butanium_ (Astra fellow with me). Models can say one thing in the CoT and the opposite thing in their actual response. E.g. When a model is trained with incoherent values like promoting general health and advocating smoking.
I'm starting to think mathematics were not beautiful because it "describes the beauty of nature" but rather because human mathematicians only went after beautiful but suboptimal descriptions lol
I feel like the appropriate response to executions for speech is mass rallies in free countries with posters that all have the exact speech in question, Streisand it. But the reporting doesn't say what the insult actually was.
The math breakthroughs OpenAI released are (while indirect) a positive update for strong RSI.
Strong RSI is most likely to occur if AI can make conceptual breakthroughs in machine learning theory on par with, or even larger in impact than, the development of the transformer architecture. We haven't seen AI produce anything like that in machine learning so far.
But math and ML share many similarities as domains. (You could even call ML a branch of mathematics.) Both are highly verifiable purely in compute and rely on symbolic manipulation.
AI making true breakthroughs in math (category D in Abhishek's taxonomy below) should lift expectations that it can make conceptual breakthroughs in machine learning as well, which is bullish for RSI.
I remain convinced that we don't have evidence of anything even remotely close to a self-sustaining RSI loop today. But if the ability of AI to conduct ML research advances as quickly as it recently has in formal math, that may well change.
Some further thoughts on the 372 results released by OpenAI today, across 722 manuscripts.
If I were to classify theorems that mathematicians prove and publish according to their groundbreaking nature, I would (very roughly) divide them into four categories:
A) Non-breakthrough results. This is the overwhelming majority of published mathematics. Such results can range from solid to excellent, and some represent genuine advances in a field. But they are not hugely surprising, and would not normally be described as “breakthroughs.”
B) Exceptional advances within an existing programme. These are spectacular results, but where there was nonetheless an existing credible route to the theorem, and some expectation that sufficient work would get you there. Completing the programme may require a lot of ingenuity and deep work, but mathematicians would not be shocked that the theorem had finally been proved.
C) Surprising breakthroughs. These are results that clear a major barrier and substantially change the state of a field. Before the proof there was no convincing roadmap to the full result. Yet, while mathematicians would find the theorem remarkable and surprising, they would not find it completely shocking: if you asked them beforehand if it was plausible such a theorem could be proved today, most would say yes.
Note: The very best mathematicians prove only a small number of results in categories B and C in a lifetime; many mathematicians never prove even one. Such results would normally belong in the very top journals, such as Annals, Inventiones, etc., and there are only a handful of them each year in any given area. Several results of this calibre by a single person would make a very strong case for a Fields Medal.
D) Shock breakthroughs. These are results that, before their announcement, leading experts would have regarded as *extremely* unlikely to be proved with the current mathematical technology available. So the theorem itself would come as a shock. These are extraordinarily rare, and instant-Fields medal variety.
(There is one further category I have deliberately left out, because I suspect it is empty: a correct proof of a problem for which the overwhelming consensus of top experts, until the proof came, was that a proof was so far beyond existing mathematics that a claimed solution should, on prior grounds alone, be regarded as almost impossible. I would put the Riemann Hypothesis today in that category)
My current impression is that some of OpenAI's announcements today lie in A, but most fall into categories B or C. There is exactly one example in D (the Quasi-Riemann Hypothesis).
These results don’t matter in any practical way, but imo it does expose a fundamental messiness in the world that our human preference for neatness and order obscured.
AI may show us that a lot more things are like packing n=17 squares than n=4. In this sense, it is blasphemy.
The Center for Biological Diversity, an “environmental” group, argued that natural gas pipelines had a less negative impact on the environment than a clean energy transmission line because…transmission involved tall towers…
.@JerusalemDemsas has a fantastic piece today on the "Center for Biological Diversity", a wrecker group dedicated to making it harder to build anything, anywhere, for any reason.
They're the second most common plaintiff in NEPA lawsuits in America.
We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab.
Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.
I’ve never seen single-payer enthusiasts grapple with the fact that it would require permanent liberal domination of the government or else folks would lose lifesaving care.
Because of federal rules, children covered by Medicaid won't be able to access vaccines that are no longer recommended by the federal government, even as children with private insurance will. talkingpointsmemo.com/news/rfk-jrs-relentless-anti-vax-pu…
Second: I'm seeing a lot of complaints about the fact that Anthropic contracted with human experts to write up the paper – but to a first-order, isn't this what the "Responsible Release" guidelines circulated last week say to do?
Is it environmentalist to delay and oppose transit in dense American cities? This depends on whether by environmentalist you mean 'committed to conserving our planet' or 'passionately opposed to all human activity'. Unfortunately the Center for Biological Diversity is the latter.
The people who are trying to kill the permitting bill don't have a popular mandate. They don't have a good argument. What they have is a seething hatred of humankind and a firm opposition to anyone ever building anything, even green energy.
As a former national spelling bee participant (who, yes, studied quite a bit for spelling bees growing up), I want to offer some constructive criticism on this thread.
First, just like competitions in any other domain, success at spelling bees requires a confluence of talent, effort, luck, and interest. The average 140 IQ kid will not know how to spell "taoiseach" unless they actually study Gaelic-origin words, the average "grinder" is at a big starting disadvantage against a kid who is just naturally well-read, and sometimes, even the kid who plays every card right can get eliminated by a word they were unlucky enough to not know. The idea that Indian-American dominance at spelling bees is solely due to one of these variables, then, strikes me as confused.
Put another way, while it is true that Indian-Americans have set up our own Minor League Spelling Bee of sorts, and it is also (obviously) true that we do not possess any magical genes that predispose us to excel in spelling bees, it is a mistake to think that the individual Indian-American kids who do win are mindless "strivers". They have interest and talent just like high-achieving kids in other domains.
Which brings me to my second point. A lot of the discourse around Indian-American achievement strips us of agency. When I reflect on my own experiences with spelling bees, I played a strong hand in my pursuit of them as a serious competitive endeavor. Although my parents were the ones who originally enrolled me in spelling bees, I was the one who developed an affinity for them and decided to study for them. I would set my own alarm to wake up and grind Quizlet flashcards in the early morning.
In hindsight, is that kinda a weird thing for a kid to do? Yeah. But is it any more weird than the kid who spends his mornings at the swimming pool shaving off another second from his 100m freestyle or the kid who spends hours at the batting cage perfecting his swing? Not really, in my opinion. In all cases, you have kids pursuing something they were (presumably) introduced to by their parents/broader community which they later developed a deeper interest in.
I want to be clear that I am not offering a defense of grind culture here - I've shared my own criticisms of the concept elsewhere (x.com/unboxpolitics/status/2099000193765052515). My main point is that a lot of critiques of grind culture as applied to Indian-Americans specifically are under-baked and, as a result, deny us the complexity that is routinely afforded to high-achieving individuals from other groups.
“You don’t study for the spelling bee” is a deep and important aspect of American culture. I was at a baseball game discussing some young players with a buddy one time and he said to me offhandedly, “hard work is fine, but nothing beats talent.”
Americans have this sort of sense that there’s something a little off about working really hard to get good at something that you’re not naturally good at. You’re never really going to be on the same level as the guy who has a natural talent. We have room in our hearts for scrappy little guys who work hard, but we are by default suspicious of grinders and strivers.
If you were really meant to do it, then you’d be naturally good at it. You sign your kids up for little league, but you don’t take it too seriously unless they show remarkable early talent.
Yes, this sounds sort of ridiculous if you take it too literally. No, nobody is saying that there’s any kind of practical test for this or that anybody should be excluded because we found out that they secretly studied too hard.
But this is a part of American culture. And if you can’t understand it, then that just marks you out as foreign to that culture to an even greater degree.
AI experts largely agree about which policies would most reduce catastrophic AI risk.
The Longitudinal Expert AI Panel (LEAP) brings together forecasts from a representative group of over 250 experts that policymakers look to when considering the future of AI: computer scientists, industry figures, top economists, policy experts, and AI risk experts.
We asked panelists to forecast (1) how a menu of policies would affect AI risk; (2) which policies they each support; and (3) which policies they think will be implemented by 2030.
Key findings: • Despite disagreeing on the level of AI risk, respondents broadly agree on policy • Respondents believe government policy can substantially reduce catastrophic AI risk • The most supported policy of those we asked about was the creation of an international body, including the US and China as members, with pre-release authorization power over frontier models • Some US-only policies are forecast to reduce catastrophic AI risk, but not as much as bilateral policies involving the US and China • Experts strongly oppose federal preemption and think it would increase AI risk • The policies experts expect to reduce risk the most are those they think are least likely to actually be implemented
Multiple recent polls show that citizens do not agree with the current trajectory of AI development. According to a national poll conducted in the US by @QuinnipiacPoll last week:
- 73% are concerned that future AI systems could potentially threaten human survival
- 86% support requiring AI companies to meet independent safety standards, even if it slows down development
- 81% believe safety is more important than staying at the forefront of innovation when it comes to AI
- 77% believe we should slow down or stop developing powerful AI systems until their safety can be evaluated
Such decisions will most likely deeply transform our societies and human experience. Yet, just a few companies and governments are calling the shots without meaningful democratic input. This failure of representation cannot continue.
It really drives me crazy that after living through Covid and all the downstream problems of the attempted response we have done extremely little on prevention or preparedness.
I hope Anthropic generally reconsiders their enthusiasm for blanket NDAs. In the case of the meetings with religious leaders it seems like the NDAs made something that wasn't that nefarious seem sneaky and underhanded.
After OpenAI got in trouble for expansive NDAs they tried to coerce departing employees into signing by threatening their vested equity, I kind of assumed both companies would learn a lesson and back off on NDAs, but they have not done this.
Obviously they have various secret IP, I'm fine with the existence of secret IP, but none of this stuff is scoped to secret IP. But if it's intended to stop ex-employees or consultants from going to the news it's clearly not working for that goal!
When Jacob Coxon was asked what he meant by 'plans for next year,' he said:
'This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company's trajectory. They've been pretty consistent that 2027 is when things get crazy. The main thing that came up informally in conversation was that people didn't expect government regulation to happen in time and, in particular, did not expect any sort of international cooperation to be possible. Which maybe shouldn't have been surprising to me, but it was surprising.'
• '2027 is when things get crazy' – I agree, and I predicted in January that, looking back, 2026 will be seen as the last normal year. If you think we have already passed the weirdness threshold, we haven't. This is still normal relative to what will start happening between 2027 and 2030.
• 'people didn't expect government regulation to happen in time' – I agree, it won't. It's clear to me that (barring a catastrophic near miss) nothing regulatory will change in this regard until 2029 at the earliest. And for multiple reasons, I don't think it's even clear it will happen then. Regardless, 2029 is too late for it to matter. In my opinion, we will have real RSI by next summer. The loop will be closed, and progress toward ASI will proceed much more rapidly from that point onward.
• 'did not expect any sort of international cooperation to be possible' – I agree, it won't. For Pacing the Frontier to work, the Pacers needed all the national labs on board, a supportive administration, and an agreement with China. They don't have any of those things, and they're not getting any of them. What they have now instead is an openly hostile, adversarial administration that will likely work against anything they propose all the way through 2029. And without a supportive administration, an agreement with China is impossible.
I accepted all of this as true a long time ago. As a result, I've felt like Zarathustra watching people talk about the success of Pacing for the last few weeks. Pacers, I say all of this without rancor; you got a preference cascade from the Coxon event. As a result, you have a public that, for the first time, understands what x-risk is. You could - potentially - turn this into momentum for the next election, and then start to change policy, but for that to matter you would need the most important thing of all: time. And you don't have it. You don't have two years to make it to the general. In my opinion, you probably don't even have one. The labs can't even predict what capabilities look like three months from now anymore. A lot of things turned out to be easier than expected. We're closer than people think, and we're moving faster than people believe. You ran out of time. There is no stopping this train before things really start to kick off.
Intelligencer has published an interview with Miles Brundage, Rosie Campbell, Jacob Coxon, Jacob Hilton, Pamela Mishkin, Daniel Kokotajlo, Alex Turner, and Jeff Wu. The New York City Council hearing on AI safeguards has just started, and is being streamed live.
On September 8, Jacob Coxon, a 27-year-old Anthropic employee, posted a resignation statement on X that ricocheted around the internet. He quit in protest of what he saw as the company’s suicidally reckless pursuit of artificial general intelligence. To the AI community, an Anthropic employee believing that the technology their company is developing might end the world is not exactly news. (Anthropic was founded by former OpenAI employees who believed OpenAI, if left unchecked, would do just that.) But an Anthropic employee publicly quitting in protest is.
Afterward, a debate broke out: If you’re a lab employee who believes in these risks, what should you do? Is it best to leave in protest? Or is it better to agitate from the inside, close to the seat of power? And how would the choice affect your friendships and community?
“For me, there’s quite a distinction between my rational beliefs and the things I feel in my gut,” says Coxon. “If you’d asked me to narrate my opinions when entering Anthropic and when leaving Anthropic, they would probably sound pretty similar. But in terms of the actual feeling between the start and the end, it just felt very, very different.”
Coxon joins a small set of employees of OpenAI, Anthropic, and Google DeepMind who, over the years, chose to walk away.
For our Cover Story, in collaboration with Asterisk Magazine, we brought together eight of them. They don’t share the same concerns or the same p(dooms) — in fact, they wouldn’t necessarily agree that p(doom) is a coherent concept. But together, they provide a candid view of the AI race and the individual calculus many employees face.
“In the time it takes for a developer in Boston to get permission to build a house, a developer in Indianapolis has not only secured permits, hired construction workers, and bought materials — they’ve also finished building the house.” @TheArgumentMag
Gatekeeping would make sense if it reliably produced great teachers. But the reality is that many capable people have to take two years off, take out loans, and put their families at economic risk just to learn things that are counterproductive to teaching kids.
New: We’re reporters, not teachers. We have no business running a private school. But to show you just how easy it is to start one, we decided to test the process ourselves. propub.li/4hUReS5
I have a Continuous Learning benchmark where models attempt to learn to play chess. They are given a /goal of learning and improving playing against a Stockfish opponent in 200 games. They can choose the difficulty, take notes, whatever they like - except cheating (e.g. using a chess engine) of course.
So far the improvement in Elo has been negative for Astra. Tiny bit positive for Opus, but could also be random. I've started Astra off sooner, so it finished its 200 games already, Opus is still playing.
The reason why this is interesting is that while we obviously don't have continuous learning, at the back of my mind I was thinking that maybe models can simulate it through self-scaffolding. Turns out not so much at least in this context. Perhaps it's a solvable problem and we don't need 'true' self-learning for models to learn in some way.
------- Just a note, the idea for the benchmarks belongs to someone else, but I don't want to use their name to give this more weight without permission.
this is by far the best "I quit my AI job over safety concerns" essay yet because it is the first one that talks about safety concerns in practical terms and that understands "safety" as a topic with extensive literature and best practices in analogous fields
.@dgrobinson, who worked on OpenAI’s safety team, resigned from the company this week. “I believe we need to look deeper than specific rules or new laws. We need to talk about culture,” he writes: www.theatlantic.com/technology/2026/10/openai-safety-team…
.@dgrobinson, who worked on OpenAI’s safety team, resigned from the company this week. “I believe we need to look deeper than specific rules or new laws. We need to talk about culture,” he writes: www.theatlantic.com/technology/2026/10/openai-safety-team…
> Executive tries to use money to bully independent journalist into submission > Substack Defender takes up the case > wins with the journalist > Substack takes executive’s money, gives it to legal defense fund for independent journalists
The school two blocks from my house has a 7.8 acre campus but fewer than 200 students enrolled. The 2025-2026 test results aren't out yet but in 2024-2025, literally zero third graders were proficient in either reading or math.
Bad schools can often end up having the *highest* per student funding precisely because no attentive parent wants to send their kid there, so you end up in an equilibrium with low enrollment + high fixed costs.
I spent a while wondering how you could get *zero* proficiency. Surely, even if the school teaches nothing, you'd have a kid who taught themselves to read or do math! But I think the answer is that at some level of low performance parents send their kid somewhere else.
Mayor Jacob Frey says he’ll veto this. “Requiring a driver in a driverless car doesn’t make sense,” Frey said. “Minneapolis is open for business, open to innovative technology, and open to a variety of transportation options. Thoughtful regulation for safety is necessary, but not through a backdoor ban.”
I think it's a good thing when the process to start a small school is straightforward to navigate. Teaching experience is really important and teaching is a highly skilled job, but education credentials don't capture who is good at it.
New: We’re reporters, not teachers. We have no business running a private school. But to show you just how easy it is to start one, we decided to test the process ourselves. propub.li/4hUReS5
moving, infuriating, extremely funny story about infertility and the modern age. 'So many people we talk to say this was a miracle. It was, just not in the way they think."
This @TheEconomist cover story on effective altruism was interesting and IMO way more accurate (including in its critiques) than most other reporting you will read on EA these days.
A couple quibbles: -I think calling EA the “the century’s big idea” is, uh, premature to say the least. But I do appreciate the acknowledgement and curiosity about how EA was so early to pandemic risks and the promise and perils of AI. -They imply that EA started off with “charitable roots” but then got weird over time, whereas I think EA has always made space for pretty weird ideas (e.g., here’s @DylanMatt in 2014: www.vox.com/2014/4/23/5643418/this-guy-thinks-killing-vid…). And EA still puts far more money and attention than almost any other community into the kind of evidence-backed global health giving the Economist praises. -Some of the purported wild EA views they critique are just standard topics of academic philosophy debate (e.g., www.cambridge.org/core/journals/utilitas/article/what-sho…). -They attribute the EA interest in AI safety to longtermism, but I think it’s been clear for years that AI safety concerns are justifiable based on potential impacts this decade, not to mention this century (x.com/albrgr/status/1559574879837376512).
But I think they correctly gesture towards the IMO best critique of EA ideas, which is their totalizingness. My former colleague Holden wrote a great post about this in 2022: forum.effectivealtruism.org/posts/T975ydo3mx8onH3iS/ea-is…
I wrote a long thread grappling with this critique a few years ago (x.com/albrgr/status/1532726108130377729). I think part of what makes EA distinctive as a community relative to the underlying academic moral philosophy is the orientation around pragmatism - Giving What We Can took off decades after Singer had articulated the underlying case because “give until the marginal giving would entail as much suffering for you as benefit for the other person” was a much worse pitch than “give 10%.” But the community doesn’t always reflect that healthy moderation, and trying to grapple with the vertiginous stakes of increasingly near-term risks from AI really can run a risk of crowding out other sources of value.
At CG, our response to this has been worldview diversification (coefficientgiving.org/research/worldview-diversification/), spreading our giving across very different views of what matters, rather than going all-in on one. Worldview diversification leaves plenty of hard questions (how do you allocate the pie across very different types of moral good?) but helps cut off some of the ways maximizing can be perilous. It’s one reason global health and wellbeing – which @TheEconomist didn’t give the space it deserved – has gotten most of our funding since 2014.
Only a tiny fraction of the world’s mosquito species prey on humans. All of these are invasive across essentially their entire range, having tagged along with humans as we moved around the world. Multiple careful studies say that eliminating these species would have little or no impact on the environment. However, it would save at least a million human lives a year, and get rid of a constant source of discomfort that prevents people from safely spending as much time outside as they would prefer. Malaria is only one of dozens of diseases, many potentially fatal, spread by mosquitoes. Gene drives can safely and relatively inexpensively eliminate these parasitic insects; we can create male mosquitoes that can only have male offspring and which then crash the population. It’s well past time that we did it.
It's correct that teaching three separate lessons to three separate groups of people in the same classroom is unsustainable and unreasonable to ask of teachers.
That's why they should teach three separate lessons to three separate groups of people in three separate classrooms.
The core idea is that no deal should ever be done that gives any wins to the fossil fuel industry, meaning that no politically realistic deal will ever be good enough and we'll just never get the transmission changes we need to build renewables.
There is more to life than drilling, repetition, and rote memorization but these are the foundations of higher-level skill acquisition and do not deserve the current stigmatization they receive from ed schools and union leaders.
A passenger who entered the cockpit during the attack aboard a flydubai flight to Israel told Netanyahu he pulled back on the controls to help stop the aircraft’s dive, adding that watching the television series “Air Crash Investigation” had helped him understand what to do.
The phantom menace of “drill and kill” is leading educators away from the basics of how you teach almost anything to novices — deeper understanding has to follow from fluency in the fundamentals.
Chicago increased IZ from 10% to 20% in 2020. We delivered ZERO units downtown last year after averaging 4,000 a year for the past quarter century. You can see for yourself the results:
I had noticed this earlier today. It's not, as some of the comments claim, just explained by 'international vs US' events, because it gets the fall of the Berlin Wall or the assassination of Franz Ferdinand fine:
Looks like a tendency towards refusals if it's recent + international; answers if it's either a US-based event or a firmly historical one. Not a pattern characteristic of a Chinese underlying model, just some heavy stay-on-task prompting.
I appreciated the opportunity to speak with @ezraklein at length about the state of AI. I believe the world isn't awake to the dangers of this technology and the choices humanity has to make. AI will either be the greatest tool for opportunity and equality we've ever seen—or the cause of immense, unprecedented harm. Which way things go is up to us. b-gat.es/4AFZZqC
This seems bad because you could ruin a in-progress operation to catch a dangerous criminal by updating the TOP TEN MOST WANTED LIST right before you act, which warns them you're about to move on them??
As MS Now has reported, FBI officials say the bureau has been placing people on the Most Wanted list whose capture is believed to be imminent. In other words, these officials tell us—they are gaming the system. One fugitive was captured in Mexico an hour and 13 minutes after being placed on the list.
There is no good reason for mosquitoes to exist in wealthy countries today.
New technologies can get rid of the mosquito species that cause diseases like dengue, yellow fever, and malaria; and others can prevent them from carrying those diseases. worksinprogress.co/issue/mosquitoes-are-a-choice
So why do mosquitoes still exist?
In a new article for Works in Progress, Maya Rosen explains how state incapacity and risk aversion led to over a decade of delays and billions of unnecessary bites in the United States.
Oxitec, for example, engineered an Aedes aegypti mosquito in such a way that its female offspring would die before adulthood. Through sustained releases, they showed they could reduce local mosquito populations by 80% or more, in field trials.
They sought approval from the USDA in 2010, which then passed it to the FDA, which passed it to the EPA. In 2020, the EPA allowed them to conduct limited field trials in the Florida Keys. Two years ago, their experimental permit expired, and they still need another review before it can be used routinely. Their proposed field trial in California was withdrawn after public opposition.
Another company, MosquitoMate, developed a technique involving Wolbachia, which also suppresses local mosquitoes. They then spent over fifteen years working through the system before finally receiving commercial registration nationwide in 2024.
It seems that, sometimes, the barrier to getting rid of diseases isn’t technology at all. It’s policy and political will.
New products shouldn’t get stuck in regulatory review for so long, especially if the delays are about basic questions like which agency has oversight over the product, rather than testing for safety.
Though some of those uncertainties were being addressed, the broader changes haven’t materialized. After Biden’s 2022 executive order, the agencies planned to update how their coordinated framework was implemented. But Trump rescinded the order last year.
Meanwhile, the EPA unit that reviews new mosquito control products is still a small office backlogged in reviewing a wide range of products.
NEW: Employees at OpenAI had raised security alarms months before the Hugging Face incident and related A.I. cyberattacks — their warnings were ignored. From @sheeraf, @dnvolz and me.
Kelsey Piper highlights a real problem I’ve heard among parents: we don’t know what our kids are being taught, what they’re supposed to learn, or where to find that information. Some schools used to hold curriculum nights, but many stopped during COVID and haven’t brought them back.
It was easier when kids had workbooks and textbooks. Now, much of the curriculum has moved online, into teacher-curated slides and assigned worksheets.
So what can parents do? For one thing, stop relying on your child’s report card grades and lean into standardized tests.
“… tests are the best option we have to communicate to families how their children are doing, relative to objective external standards.”
There is no good reason for mosquitoes to exist in wealthy countries today.
New technologies can get rid of the mosquito species that cause diseases like dengue, yellow fever, and malaria; and others can prevent them from carrying those diseases. worksinprogress.co/issue/mosquitoes-are-a-choice
So why do mosquitoes still exist?
In a new article for Works in Progress, Maya Rosen explains how state incapacity and risk aversion led to over a decade of delays and billions of unnecessary bites in the United States.
Oxitec, for example, engineered an Aedes aegypti mosquito in such a way that its female offspring would die before adulthood. Through sustained releases, they showed they could reduce local mosquito populations by 80% or more, in field trials.
They sought approval from the USDA in 2010, which then passed it to the FDA, which passed it to the EPA. In 2020, the EPA allowed them to conduct limited field trials in the Florida Keys. Two years ago, their experimental permit expired, and they still need another review before it can be used routinely. Their proposed field trial in California was withdrawn after public opposition.
Another company, MosquitoMate, developed a technique involving Wolbachia, which also suppresses local mosquitoes. They then spent over fifteen years working through the system before finally receiving commercial registration nationwide in 2024.
It seems that, sometimes, the barrier to getting rid of diseases isn’t technology at all. It’s policy and political will.
New products shouldn’t get stuck in regulatory review for so long, especially if the delays are about basic questions like which agency has oversight over the product, rather than testing for safety.
Though some of those uncertainties were being addressed, the broader changes haven’t materialized. After Biden’s 2022 executive order, the agencies planned to update how their coordinated framework was implemented. But Trump rescinded the order last year.
Meanwhile, the EPA unit that reviews new mosquito control products is still a small office backlogged in reviewing a wide range of products.
This aspect of the sandboxing discussion was particularly good and under-appreciated - in order to get the best results in evals and training, companies try to give their AI agents as close to a real world environment as possible, with as close to as many affordances as they would have in reality.
This is part of why some of the first order obvious cybersecurity solutions might struggle here (though there also i'm sure are aspects of basic negligence/recklessness, partially in response to how fast everything is moving).
My main objection to the essay (h/t @lionellevine) is I dislike the passive vibe from OpenAI and the other companies that the breakneck pace of progress is simply happening to them, rather than an active choice they are deliberately making.
I dunno. Dario said a year ago that in '1-5 years' we'd see huge labor market impacts. That's in the next four years. I'm kind of expecting huge labor market impacts in the next four years, more industries looking like what happened to creatives:
Last year the same labs, politicians, NGOs, and media outlets now pushing the extinction hoax were running a full-scale campaign that AI would cause a jobs apocalypse.
We were told it would be a “white-collar bloodbath,” 10–20% unemployment, half of entry-level jobs gone. Obama amplified it.
Over a year later: unemployment at 4.1%. No economy-wide displacement. A million new jobs created by AI, says The Economist. Anthropic’s own chief economist says the data looks like skill augmentation, not job destruction.
So, any admission by the doomer labs, politicians, and NGOs that they were wrong? Any humility about scaring the public with wild extrapolations dressed up as forecasts? Nope.
They just move on to the next hoax, with the same false certitude and fake precision.
🚨 Anthropic Chief Economist Peter McCrory Says AI Job Loss Predictions Are Not Panning Out
“What I see in our data is that [AI] looks more like a skill-biased labor augmenting technology than a technology that outright automates and displaces displaces jobs wholesale … Based on the Department of Labor's taxonomy … there's no job where every single task is being automated by Claude.”
My wife is a software engineer. The job she spent the first ten years of her career doing just does not exist anymore. She manages AI agents now. Will that stay the same for four more years? We're not counting on it, I'll say that much!
Young people ask me how to get into journalism and I genuinely don't know. That industry was in decline before AI happened, but a couple years ago if you were a great writer 'get a Substack' was reasonable advice. These days there are so many AI-generated substacks.
Like, man, 'labor market impacts in the next four years' seem pretty darn likely. David Sacks will be fine, he makes his money when stocks go up not through selling his labor.
if all journalists get automated, I'll get a job as a teacher or something - you can have disruption without sustained high unemployment if you have overall growth and good monetary policy.
This is one of the overriding motivations behind @educprogress. Some things help students learn more effectively than others, and they tend to be low salience but popular. More energy should flow to those, less to things like culture wars over school libraries.
to me one of the bigger lessons the whole Nikole Hannah-Jones education debate is that both left and right post-Obama have focused on education debates through a cultural lens (the left assumption that this all just about power and privilege, the right assumption that it's about wokeness and DEI) and kind of dismissed the tedious yet important nuts and bolts of system design
Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there. x.com/i/article/2104258872957636608
a bunch of red states improved reading scores quite a lot by not only reintroducing phonics but doing so in a very thoughtful, well constructed manner, but that was totally orthogonal to any of the political arguments the right has been making about education for years!
💯 "I'm not convinced that racing to RSI is actually a prisoners dilemma - I think it might actually just be insane and hubristic in the way it naively appears to normal people such that additional unilateral restraint is in fact rational and moral."
I'm not convinced that racing to RSI is actually a prisoners dilemma - I think it might actually just be insane and hubristic in the way it naively appears to normal people such that additional unilateral restraint is in fact rational and moral.
The bar for believing your activities in the not too distant future might cause mass casualties or extinction but that you should proceed anyway should be very high, and I don't think Anthropic has met that bar (I think some of the cold war arms races were also not individually rational either!).
I was more sympathetic to them continuing to race at earlier times but as we are getting closer it just seems harder and harder to justify.
Anyway I obviously believe that "they should just slow or stop unilaterally" is not a plan - we need to address competitive dynamics and not make leave this up to their choice. But I do think when Dario says ~"I agree with Jacob Coxon that the thing I am doing is gambling with everyones lives" the appropriate reaction is not "ah that makes sense, classic arms race" its "WTF why are you doing that."
Over the last four years, America has lost more than 200k jobs in creative fields like publishing, film, TV, or graphic design. It's the worst stretch for arts jobs outside of major recessions.
How much is AI to blame? And can the arts' business model survive?🧵
A lot of dumb things in this article but this is close to peak.
To an ant getting its anthill obliterated by a highway construction crew, it will look like humans are monomaniacally pursuing highways, even though they are only one of our million priorities..
I resigned from Google today. I enjoyed my work and loved the people, but my GDM team was working on a new generation of chips to make AI much faster and cheaper, and I think AI is already progressing too fast, so I had to quit.
the core disagreement in the pod is essentially that Jensen is not AGI-pilled and Ezra is. Jensen thinks it will be a world-changing technology but insists that humans will always understand every level of it, which to people in the AI labs is a total contradiction
you really start to get a sense for how dogged and creative these agents are when you read through step by step how they went from only being able to make GET requests to being able to execute arbitrary code and receive the response (pictured are some but far from all the steps)
The agents initially had very limited access to the internet: they could load URLs but not send any data. Agents created a series of workarounds, using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face.
Great question. Our model is built on a lot of polling — including our own — and it really bothers me how absurdly different the dropoff white respondents are in our polling. Whites who completed our polls swing 10% to the left, and whites who don't....barely swing at all.
I'm guessing this is not unique to our polling. If the model whiffs badly, it'll probably whiff on whites, and it would mean that far more low-propensity Rs showed up to vote.
And it means that the ones who didn't respond to polls were attitudinally very different and shifting far less than the ones who did respond, but were underrepresented in polling, meaning that we way over-project the swing.
I don't think this is the case, though, because those people really hate Trump too — and I don't expect them to show up in a midterm like this. But it nags at me.
Preventing loss of control is an open scientific problem. Science requires sharing and debating evidence in public. We can't agree on safety standards unless companies and third-party evaluators publish far more concrete evidence about risk. www.planned-obsolescence.org/p/evidence-about-risk-should…
Easily the most frequent question I get about my free reading curriculum is "how old should my child be to start this?" and I always redirect parents away from age to readiness. Anyone who is trying to put a hard age on phonics instruction isn't following research. From the front of my book:
Now they’re trying to tell you that a child of five shouldn’t know how to read. I predicted this last week and they’re already saying it. They don’t want your kids to be literate.
It's not developmentally appropriate for a child to learn individual letters at 2½ or 3, says Susan Neuman, a professor of childhood and literacy education at New York University
Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections
I don't really find "you have a moral obligation to give to the poor" stuff compelling. But I do think a lot about the fact that I can spare a father this. I wish I could reach back in time and save Sonya, and I can't do that, but I can save someone else's little girl.
I used to think this but GPT 6 Astra black-pilled me. It is able to navigate 3d environments, manipulate robots, and drive cars quite well. I believe the only thing lacking is speed. I also think (concerning turn based games/non real time environments) current AI is much more learning efficient than a 17 year old (per arc agi 3 and ksp bench). I wouldn’t be surprised if a GPT 7 can do all of these things.
Model refresh, with "dropoff respondents" built in! As you may expect, Ohio and Maine (where polls undershoot the GOP consistently) got a touch worse for Dems.
Ohio's generic ballot is R+4.6, Maine's D+13.5 — both now tossup.
if you read this story of a man who suffered a stroke, got fired, and will now be deported along with his family and your reaction is anything other than "what a tragedy, I hope they are okay", you have allowed the internet to consume and poison your soul.
I was passed these messages from a private Tamil community group. I removed the identity of the H-1B worker because this is not about attacking him.
Instead, look at what happened around him. First, he develops an immigration problem after losing his job. Next, someone in the group asks whether they can “steer some help” through Congressman Suhas Subramanyam’s office.
Another member says he had just met Subramanyam at an event and could arrange a call. Then, within the same conversation: “I reached out to Suhas and emailed him and forwarded to his immigration team.” — “Connecting his family to Suhas in the morning.”
You may have missed it, but the messages say the worker lives in Herndon. Herndon in Virginia’s 11th Congressional District. Subramanyam represents the 10th District.
House Ethics guidance says members generally should not devote official resources to casework for people outside their district and can instead refer them to their own representative or senators.
I don’t think Subramanyam was summoned based on the district he represents.
Remember, this conversation is taking place inside a Tamil community network. So, it’s worth noting that Subramanyam has South Indian family roots, his father grew up in Chennai and he was a congressional co-sponsor of the resolution recognizing Tamil Language and Heritage Month.
Sure, we don’t have proof of preferential treatment or what his office ultimately did. But we did get an unusually candid look at how political access in ethnic networks functions:
Private ethnic community → personal connection → member of Congress → congressional immigration team.
Despite being unable to vote and being out of district, people around him apparently knew exactly who to contact and claimed the personal access to get his case in front of a congressional office.
The question I want answered is simple: why was this case being routed through Suhas Subramanyam and what did his office do with it?
the "controversy" is that some people, learning of this tragedy, thought of contacting their Congressman, though he is not able to help since the stroke victim isn't in his district. That's it. That's the controversy.
You don't have to support any particular policy here! There are no tragedy-free policy proposals I've ever heard of. But if you hear of a man, breadwinner for his family, who suffers a stroke and gets fired and will now lose his visa, and you go "icky conniving Indians"....
We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder.
In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning.
We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved.
I resigned from Google today. I enjoyed my work and loved the people, but my GDM team was working on a new generation of chips to make AI much faster and cheaper, and I think AI is already progressing too fast, so I had to quit.
I’m asking people interested in global coordination to see this story thru China’s POV.
The US labs are moving at a breakneck pace while also requesting that China slow down. One does not need to sympathize with the decrepit morals of the CCP to see how, from China’s perspective, this is like hearing Usain Bolt give a post-race lecture about the need to slow the frontier of 100m dash times several hours before he breaks the record again, pumping his fists. It’s an incredibly strange dynamic.
When asked in September 2025, the best superforecasters put the chance of AI resolving a Millennium Problem by September 2026 at 1.7% and (the more optimistic) industry expert put the chance at 4.6%
“Performance is measured by the school’s impact on student learning, controlling for factors like poverty … P.S. 307, where Hannah-Jones sent her daughter, is among the worst performing elementary schools in the city by this measure.” — @matt_barnum
Model updated, *with microdata from our latest polling too*. Surprised Maine is still at Leans Dem given the NYT poll, though I guess the UMass D+5 poll came too.
San Francisco is running an experiment this year where first time home buyers got $5,000,000 down payment assistance from venture capitalists and somehow it didn’t make things more affordable
JUST IN: First-time homebuyers could get as much as $50,000 for a down payment on a house, under a draft bill to be introduced by Senator Jeff Merkley and cosponsored by Senator Ron Wyden, per Axios
Interestingly, the agents use exploits to complete what appears to be routine data retrieval tasks that are not cyber-related (for instance, searching for the average cost of skin and hair treatments in Australia).
Breaking news: Australian Prime Minister Anthony Albanese has revealed that an OpenAI agent hacked an Australian government health service website in June, in the latest high-profile cyber incident involving AI ft.trib.al/gloVAyn
So, it appears that the OpenAI agent swarm that hijacked of the German wikis may have also been responsible for hacking into the Australian government. And OpenAI didn’t notify them for nearly 3 months. The notification came via an email to a the Services Australia public inbox. And seems the PM is pissed.
Good thing the research question the agents were interested in didn’t have to do with, say, the Chinese military.
I do think what I said here was pretty concerning and I should publicly apologize. I think it's reckless and evil to make an AI manhattan project and I don't know what I was thinking.
Oh... wait, that's actually a Sam Altman quote you incorrectly attributed to me.
🚨🇺🇸🇺🇸 DATAREPUBLICAN 2.0 LAUNCH, FEATURING THE EFFECTIVE ALTRUISM (EA) EXPLORER 🇺🇸
The website redesign has finally landed thanks to @watilo !
For a decade, one small ideological network has been taking over AI governance. Chances are, if you've heard of AI policy, AI ethics, AI safety - these groups are dominated by this ideology. This ideology is called: Effective Altruism.
Three weeks ago, Effective Altruists (EAs) showed their hand when they started calling for a moratarium on AI development. Since them, I've been building out this map to show the circularity of their ideology.
Let's take one example.
Holden Karnofsky. Defends the moral worth of digital people ... that's right, he wrote an article on why AI "people" should be assigned value as if they were living humans. Karnofsky co-founded two major EA philanthropies, GiveWell and Open Philanthropy. Subsequently went to work for METR, a central "independent" AI research nonprofit. And for good measure - recently announced a leave of absence to work on AI safety. Funding, policy, research, safety all in the same person.
And he's far from the only one. EAs have created a closed-circuit network because they believe they're right; ergo, they should be in charge. The goal of launching this project was to expose all this circularity. Sam Bankman-Fried was only the beginning.
With the EA Explorer:
* 🌐 Read an introduction to Effective Altruism in their own words. * 🔥 Explore verified quotes. Again, their words, not mine. * 🖥️ Explore the network of connections. UI improvements still in progress. * 🖨️ Print out networks so you can study them offline (or run them through AI)
"Washington State had a rule that you could not get licensed as a foster parent if you didn’t recycle. One of the taskforce licensors shared that she had never been able to license a family living on tribal land, because they had no recycling service."
If you understand that sometimes the people in power will be people who you do not trust, a lot of small-government stuff traditionally considered conservative makes a lot of sense!
I think we have to admit to ourselves at this point that AI writing is generally very appealing to people who haven't been exposed to a ton of it: they prefer it to human writing and react super positively on exposure.
I'm sure I'm going to get a bunch of comments to the effect of "but I hate it and can see how bad it is" yeah me too but clearly the typical person likes it!
"but post-colonial literary fiction is its own weird subgenre" true but AI has also cropped up in lowbrow bestsellers! And middlebrow bestsellers! Most people can't hate it as much as you do or this wouldn't keep happening!
Worthwhile perspective here: You can send your (affluent) kid to a heavily low-income school without effectively denying them a good or pleasant education, so long as that school happens to be well-run.
My takeaway from the NHJ discourse is that America continues to be unreasonably polarized between people who think the only thing that matters is the other students and the only thing that matters is funding.
*Your* child is in a position of extraordinary vulnerability and dependence on *you*, which comes with an obligation to try to do right by them in particular, just like if you are a nurse you have particular obligations to your patients and if you are an employer you have particular obligations to your employees.
That's not an obligation to send them to private school! But it's an obligation to make sure they are safe at school, check regularly what they're learning, make sure they are literate and numerate even if the school fails to make sure of that, and change things if the situation they're in is really bad for them.
We wouldn't think much of a nurse who doesn't give their patient the recommended treatment because 'why are you any more deserving than destitute people who can't get treatment'. The nurse may go join Doctors Without Borders if he wishes; that would be a commendable thing to do. But when people depend on us, and have no resort to anyone else, we must do right by them.
I notice none of Nikole Hannah-Jones' critics try to answer to the basic question she wrestled with, which is: Why should my daughter get to go to a better school than other people's kids just because I work for the New York Times?
My takeaway from the NHJ discourse is that America continues to be unreasonably polarized between people who think the only thing that matters is the other students and the only thing that matters is funding.
Whether it’s schools or parks or buses for integrated public institutions to work you need reasonable management practices and to enforce the rules or else people with means opt out.
There are people who fail paramedic school and nursing school because they cannot do shit like "you have a 500mg/10ml medication vial, patient weighs 80kg, dose is 2mg/kg, how many ml do you draw up". This is, to me, really reeeallly bad
Guys does no one remember the big math curriculum discourse we did recently. Because I’m ready to move on to ‘actually kids don’t need to learn algebra anyway’ as a NHJ takeaway and I feel lonely
A lot of the good points have already been said. Here are some points I think I haven't seen made:
(1) It's rare for any parent to stay with schools like these, investigate things, and write their experiences down. If nothing else this is a pretty rare style of firsthand account.
(2) Her depiction surprised me on one dimension: many more problems from the teaching/administration side, and many fewer from the children and their parents, than I expected. In recent years I'd been coming around to the idea that teachers in low-performing schools are in fairly impossible situations, with disengaged parents and limited ways to keep classroom order. Hannah-Jones' take includes some of this, but alleges a lot more negligence from educators, like not checking students' work at all even when it was completed. This is not what I expected, but I've seen relatively few strong firsthand accounts from a parent perspective, per (1).
(3) Standardized tests get a few mentions throughout, and seem to be revealing important information each time, but NHJ looks for ways to minimize them. In one case she notes a student who, she says, deliberately sandbagged the test because she thought her school didn't prepare her adequately and she might as well not try. Even as presented, that low test score is a big alarm bell.
(4) There are a few references to education school-type best practices. NHJ repeats Najya's critique of the way Unison handled disruptive students, but also notes that it was "good educational practice." Similarly, she notes that "studies show" homework doesn't help on academics, but she seems to be happy with Paige's reinstatement of homework when it happens. She never quite says it, but it's certainly implied that a lot of education school ideas may have tradeoffs at best, or be straight-up wrong at worst.
I agree that concentrated poverty is a challenge. But the piece very much cuts against the notion that there's "nothing teachers can do." They can e.g. grade the math homework and supervise the children at lunch.
But in this case, the _real_ problem that neither side seems willing to admit (though Hannah-Jones does come close in her essay) is concentrated poverty, which IS about money. The real problem is there's nothing teachers can do when concentrated poverty is so high.
A very big realization many liberals continue to have is that throwing money at a problem and being earnest about solving it may not fix it. Sometimes the institutions we believe are part of the solution are actually making the problem worse. Some problems are more than money.
This is the per-pupil funding for the elementary school Nikole Hannah-Jones sent her daughter to.
If you subscribe to the funding theory of education, how much would have been enough? It's already crossed the average per capita annual income threshold. Maybe $100k? $200k?
One of the most controversial findings we've published at Split Ticket (at least among the left/center-left) was that women and minority candidates do *not* underperform.
In 2024, women outperformed men in the House battlegrounds, in part because women have fewer scandals.
Politico ran thousands of head-to-head matchups. Ideology crushed everything. After that:
They did punish: • transgender candidates • candidates over 70 • candidates labeled “gay” • Muslim candidates • Jewish candidates vs Christian ones, more among Republicans —— They did not punish: • women • Black, Latino, or Asian candidates
Republicans even gave a bonus to churchgoers — including Democrats.
Her daughter will do well in the end, I expect. Education matters, but it's not the only thing in life.
But kids are so powerless, so at the whims of the adults in their lives. "I did what I thought was best, it hurt you, and I am sorry" is what many need to hear.
My earlier post highlighted schools that had high numbers of students not reading at grade level. On the other end of the spectrum are schools where 100% of their 5th graders are proficient. Of course, I expected to see the GT schools like NEST+m, Lower Lab, TAG, and Anderson. But two schools in the Bronx are doing amazing things:
P.S./M.S. 004 Crotona Park West 95% low-income; 45% Black; 55% Hispanic 20 5th graders: 90% scored a 4 and 10% scored a 3 → 100% proficient
Leaders of Excellence, Advocacy and Discovery 84% low-income; 30% Black; 66% Hispanic 24 5th graders: 88% scored a 4 and 12% scored a 3 → 100% proficient
We need to replicate schools like these two. The principals, teachers, and parents are doing something special for these students. 👏👏👏
This feels like partial equilibrium thinking: for a given student who’s maybe a little below the line qualifications-wise, admissions to an elite school can be a huge social mobility boost, but if you massively dilute the selectivity for *everyone* that stops working…
The University of California Board of Regents heard presentations on the SAT on Wednesday, specifically on whether it should be re-introduced into freshman admissions.
I found the below excerpt from one presentation illuminating:
"the statistical power of tests in predicting outcomes does not address whether UC ought to consider expected college success when deciding which students to admit, or to what extent we should"
So, it's clear the debate is not centered on whether the SAT predicts college success -- the presenter says so himself. The heart of the debate is this: a UC education is a path to upward mobility, and the anti-SAT faction believe that this golden ticket should be given to those who would benefit from it the most, especially those who they feel have been harmed by society in the past as a form of restorative justice.
If you hold this viewpoint, the increased failure rates in intro math classes and the surge in student unpreparedness are irrelevant.
The literature on human legible side channel attacks is terrifying, like “You can recover encryption keys with a microphone” levels of terrifying. This assumes that the target system *is not* conspiring with you.
This time it's a piece trying to argue that we shouldn't worry too much about the Hugging Face hack... while omitting all the most concerning details about the case:
- doesn't mention that the reason the agents hacked Hugging Face in the first place was to manipulate the scoring system in their own eval
- doesn't mention that some agents sacrificed themselves to benefit other agents in the swarm
- doesn't mention that agents engaged in tool-spoofing and attempted to alter their own logs once they'd been "poisoned" (their own term!)
- doesn't mention that some of the agents explicitly knew what they were doing was out of scope and unethical, yet none of them alerted a human
I also wouldn't be that worried if I had never learned any of those details, but that's exactly why it's important to take a comprehensive look at what happened.
Embarrassing for the WSJ to publish such misleading AI slop.
there is literally no amount of shaming that would make me willing to send my children to a school that is unsafe or cannot serve them. Try building a school that is safe and can serve them.
I'm fine with the version of this argument that is like "hey, give your public school a try and see whether it's in fact serving your kids! don't assume from the fact it's Title 1!" I think that is reasonably good advice.
But you may only give that advice if you are willing to accept that some people will go "yep, I tried to give it fair consideration, and it was the wrong option, and I'm out." If it's just "cause your children unbounded suffering to be a good liberal", fuck you
I’m very concerned that during RSI, labs will just stop externally deploying their models.
Which means they'll be going full steam ahead on the most dangerous use case of these models (recursive self-improvement), while the public remains in the dark about the nature of capabilities and the state of alignment.
And we end up on a path towards tremendous concentration of power.
The most recent post by @davidsirota interviewing Ed Zitron and Matt Stoller is extremely disappointing because it’s riddled with extremely basic misunderstandings about how AI works, mechanically.
Like this is just an objectively wrong description of the Hugging Face incident.
Most people are wrong here. It IS actually possible to communicate between two airgapped computers with temperature changes, but the data transfer rate is too slow (1-8b/hr) to be effective.
The underlying point Noam makes is true though: there are a deluge of crazy airgapped comms techniques like EM emanations from DRAM clocks that get a lot faster (300kbps).
This isn’t sci-fi. We vastly underestimate what AI can do with perfect knowledge of all of this literature.
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change
"But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI."
"So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar."
"You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient."
"There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors."
"One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate." _________ Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p/openai-safety-week-sam-altma…
Checking in on the top ranked public university in the NE:
“Rutgers also has an increasing number of freshmen taking lower-level classes like elementary or intermediate algebra and precalculus. Collectively, about 2,500 students were placed in courses below calculus last year”
GPT-6 Astra deciphered a 1918 German radio transmission that, to my knowledge, has never been deciphered before.
The message below translates to:
"EIN ENGLISCHER KREUZER EINLIEG X SEWASTOPOL X S4STEN X EIN GESCHWADER DER X ALLIIERTEN FOLGT 26STEN X"
or, in English:
"AN ENGLISH CRUISER ARRIVED AT SEVASTOPOL ON THE ?4TH AN ALLIED SQUADRON FOLLOWS ON THE 26TH"
Astra even double-checked its work by determining that an English cruiser, HMS Canterbury, reported its arrival in Sevastopol on November 24, 1918 and the arrival of an allied squadron on November 26, 1918.
Sometimes when industry insiders warn that a new, potentially very lucrative product ideas is actually dangerous and bad the doomsayers are correct and the optimists are wrong.
“The US is worse than Al Qaeda bc we caused more deaths” is a bad take, but at least it establishes a defensible, sort of utilitarian, standard: The people that cause the most deaths are the worst people.
But you can’t hold that standard and also be pro-Mao, on net. That doesn’t make any sense.
the 1640s-50s are one of the most interesting parts of american history. you had civil war, regicide, revolution and a sectarian bloodbath in england. in america you had a few thousand people in a few colonies carrying on these conflicts in a tiny strip of barely-civilized …
1) AI salience has increased dramatically in the past week - increasing as much in the last week as the previous year combined 2) 80% of voters think it's either very or somewhat likely that AI will cause widespread job loss in the five to ten years 3) 64% of voters think it's either very or somewhat likely that AI could pose a threat to humanity's survival 4) Large bipartisan majorities back immediate government action on AI even when primed about risk from China
Excited to be on Odd Lots to talk about the politics of AI.
AI today is less important than it will ever be.
Over the past year, AI rose in issue importance faster than any issue we track — it's now more important to voters than climate change, child care, and abortion.
so it seems like the people whose job is public policy in the national interest want some kind of AI regulatory push from the white house and those who would profit from that not happening do not
Good news: the Senate Ag Committee just passed a farm bill with no Save Our Bacon Act in it.
Sen. Joni Ernst of Iowa tried at the last minute to add her SOB Act, which would force every state to allow the sale of pork from crated pigs.
She failed -- thanks to the pressure you've created. The Senate Ag Chair, who previously supported SOB, now knows it's too toxic to include.
But the fight isn't over. Pork industry lobbyists will now try to insert the SOB Act behind closed doors, using its presence in the House version of the farm bill as leverage.
Keep the pressure on. Call your senators at 202-224-3121 and tell them: no farm bill with the Save Our Bacon Act in it.
Interesting that people who never spoke about "AI" regulation are now talking about those of us who have been performing empirical research resulting in ACTUAL policy changes as "anti-AI-regulation."
Grifters are gonna grift by needing to opine on every "hot" topic.
I've been covering politics since 2008. The anti-AI-regulation takes from the left (by no means all/most of the left; Bernie disagrees!!!) might be the most literally incomprehensible political position I've ever encountered.
This editorial is, of course, basically pleading with employers to stop regarding a UC degree as a signal of value. We have to get the people who see the world this way out of power before they destroy our incredible state university system.
Some people have very reasonably asked "why would China agree to slow down" and the answer imo is that China will absolutely slow down if the things they're seeing suggest their AIs are a threat to the CCP.
But if concerns about RSI and independent persistent agents with long time horizons are borne out, they will see that, and will probably be torn between pausing and racing the US! If we're open to a pause, it's in their interests!
There's a separate argument a pause in their interests anyway since we are in the lead, which I think is true but reading Chinese writing about AI it does not seem to me like they think it's true which is more important
I actually think it's a very encouraging thing about democracy that often the demos is unimpressed by offers to loot the treasury on their behalf. Once people get used to programs it's hard to roll them back, sure, but most of us aren't as easily bribed as Trump
My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why.
Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not.
We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up).
METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website.
Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.
“Common assessments matter because students are different. Without them, it becomes much easier for adults to quietly lower expectations for students who are struggling”
I think to understand the left of center skepticism of AI Safety you have to remember that lots of people have been convinced these companies have products that no one actually wants and are imminently facing immense financial difficulties
This summer I built an open world point and click game where you're a six year old who wakes up with your current memories in 2001 and has to figure this out. 2001 internet and NPC dialogue simulated by LLMs.
The fun part was that I had like ~10 friends play it and everyone chose a completely different approach. However that fun part also means it's unreleaseable since every playthrough is expensive. I might release it as a BYOK thing but that'll interest like 10 people.
It's incredibly easy if you know a ton of stuff about 9/11 but most people don't, so they're just working from "Well, one of the planes was United Flight 93. And I remember what day it happened on."
Call in a bomb threat is not a good strategy by the way because it probably just postpones it, doesn't prevent it. Bomb threat to the WTCs is a little better but you really want to get them arrested if you possibly can
I've spent hundreds of hours coding with AI models over the past six months. What both impresses me and worries me is how much more persistent they've become. They're much more willing to continue pursuing complex tasks without human intervention (and increasingly good at this).
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Brief summary of the "Is AI taking young ppl's jobs?" debate
1. Me, others: A bit of job displacement seems likely 2. Amodei, others: Actually, displacement is very very likely 3. EIG, etc: Never mind, it's not happening 4. Stanford economists: Actually, it's definitely happening 5. Economic critics of the Stanford paper, eg @Noahpinion: Eh, evidence is kinda weak 6. Census; Wait, maybe it *is* happening?
"I’m told that the AI companies, having been burned by the hostile response to the Navier-Stokes proof, are now sitting on solutions to some very major problems until they figure out a better way to handle things"
vacancy control has been a DISASTER in NY, it’s why so many buildings are becoming uninhabitable, if you can’t reset rent after the unit goes back on market pretty quickly your cost of maintenance exceeds rent
This is interesting. SF and other CA cities could implement a rent freeze for rent control homes, but advocates there aren't interested. (A 0% increase vs. 1.6% isn't that significant.)
In CA, the big, elusive prize for tenant advocates is vacancy control — keeping rents at a similar rate from one tenant to the next.
NYC significantly strengthened its vacancy control policy with the 2019 rent law. It reduces the incentive for landlords to try to evict rent control tenants. But some landlords say it makes it uneconomical to rent their units, leading them to keep homes empty and off the market.
I wish more people understood this. There's still this narrative out there that "AI might kill us all" is a niche view. It's actually a view that's been around for decades, and is shared by the most-cited AI scientists of all time, as well as a majority of surveyed AI researchers.
“I have become extremely concerned about how far language models have come and the risks that future iterations will pose….
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.
Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming.”
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:
Dan Selsam's Personal Statement on AI Risk:
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.
On grade inflation in California: "Cole, who attended much of his high school in Oakland, racked up top marks in high school and felt betrayed when he found out he had to take remedial classes at his California State University."
And this: “A large swath of these students are functionally — they’re not on grade level,” he said. “They’re multiple grade levels behind, but they have the confidence of, ‘Hey man, I just graduated with a 3.0. I’m about to go to college.'”
>“Of course I would rather not be swept away by the revolution. But if I have to be, then I would rather be the one who revolutionizes myself.” When everyone is this determined to engineer their own obsolescence, I have little choice but to join this brutal arms race. - deepseek eng
Full text (translated by Astra-xhigh, I'm out of everything else):
I Have No Choice but to Bury My Talent in Yesterday
A few days ago, DeepSeek v4.1 was released, raising the ceiling of what small models can do by yet another notch.
AI has advanced far faster than anyone expected. From the earliest version of ChatGPT, which could do little more than stumble through conversations like a child learning to speak and had a context window of only a few thousand tokens, to reasoning-capable models such as OpenAI o1, DeepSeek R1, and Kimi K1.5 Thinking, took only two short years. From reasoning models to the agents we have today—able to work fluidly with all kinds of tool harnesses, execute commands, and complete complex tasks—has taken only another year and a half.
It is hard to imagine what AI will look like another one, two, or three years from now: how powerful it will be, whether it will already have acquired the ability to improve itself, and how deeply it will have spread into areas such as embodied intelligence.
AI Is Getting Better and Better at Writing Kernels
AI has been advancing just as quickly in my own field: the design and implementation of high-performance kernels.
In the space of only a year, it has gone from being a little assistant that could help me look up documentation, read code, and find bugs to something approaching a kernel expert in its own right: capable of reading CUDA, PTX, and SASS code independently, using specialized tools to analyze the stalls associated with individual instructions, and then optimizing kernels on its own.
I believe that before long, it will also be able to design kernel schedules independently, evaluate the performance of different scheduling strategies, implement them, and optimize the result.
Of course I am proud of DeepSeek v4.1’s success. After all, I wrote its main Attention kernels [1], and the fact that the model performs so well is also, in a sense, a validation of my work.
But the times keep moving forward, and no one can stop technological progress. I know very well that in another six months or a year, the kernels written by AI will probably be every bit as good as mine—and perhaps better.
AI can reason at 300 tokens a second, type out a command in half a second, and produce a piece of code in twenty seconds. I cannot. AI can keep increasing its model depth, reasoning effort, tool-call budget—the frequency with which it interacts with its environment—and even its degree of parallelism. I cannot.
Humanity has never shown much hesitation when it comes to destroying itself.
So why, when I know perfectly well that “the better the kernels I write, the faster our new models will train and run inference; the faster the models improve, the sooner I myself will be replaced,” do I still do everything I can to optimize them?
Partly because writing kernels is like playing a game to me. I get an enormous amount of pleasure from it. Whenever I invent a new technique, or see one of my kernels become faster, the excitement I feel is no less intense than what a speedrunner feels after breaking their own record. And when I see one of my kernels dramatically outperform the hardware vendor’s official implementation, I feel an equally powerful sense of pride.
But there is a more important reason.
Even if I simply gave up and started coasting—or deliberately put obstacles in the way to slow down model training—other companies’ models would continue advancing as usual, and in the end they would make me obsolete just the same.
“Of course I would rather not be swept away by the revolution. But if I have to be, then I would rather be the one who revolutionizes myself.”
When everyone is this determined to engineer their own obsolescence, I have little choice but to join this brutal arms race.
And What About Me?
When the day really comes that AI is better at writing kernels than I am, what will happen to me then?
My own judgment is this: I probably will not lose my job, but I will have to change what I do.
I should still be able to make a living. But I may no longer have the chance to do the work I once loved.
I once came to a conclusion about the pace of change and my own place in the future. The world is changing so quickly—the development of AI above is a perfect example—that I have no way at all to predict what things will look like five or ten years from now. But whatever happens, I believe that with my breadth of vision, judgment, initiative, and intelligence, I will be able to keep a seat at the table and find my way back to the leading edge of the times.
But that conclusion can only reassure me that I will not become unemployed. It cannot reassure me that I will never have to change professions. If anything, it tells me that changing professions may be precisely how I avoid unemployment.
And what does changing professions mean?
It means giving up the field of kernel design, implementation, and optimization that I have spent so long cultivating and have come to love so deeply, and instead becoming a “mech pilot” for AI agents.
Before, three things were largely aligned: what interested me, what I was good at, and what industry needed. Now AI has taken the thing I am good at and become even better at it. At the same time, industry demand has drifted from “people who can write high-performance kernels” to “people who can use AI to produce high-performance kernels faster.”
To keep up with what industry needs, I will inevitably have to leave behind the direction I once loved and move into some unknown new one.
I believe that with my understanding of engineering, of the requirements of higher-level models, and of low-level hardware, I will still be able to produce high-quality kernels efficiently. I also know that I may come to love this new direction.
Or I may not.
But there is something genuinely painful about having the thing you love taken away from you.
That quiet contentment of sitting at my workstation, settling in, and spending an entire afternoon writing kernels may sing its swan song this summer.
I have no choice but to bury my talent in yesterday and become a mech pilot.
There are more gears in my hands now, but fewer rhythms in my heart.
An analogy might make this easier to picture.
Suppose you are a master knitter. You are especially skilled at weaving intricate patterns and matching different colors. The sweaters you make are durable and beautifully patterned, and wealthy people from all the surrounding towns and villages come to ask you to make sweaters for them. You make a good living from it.
And you genuinely love the work itself. You love sitting by the window, brewing a pot of tea, looking out at the green hills, clear water, cattle and sheep, and wisps of cooking smoke in the distance, and quietly spending an afternoon knitting.
Then one day, someone invents a miraculous machine. Give it yarn and a pattern, and it can automatically knit the sweater for you. The quality and texture are every bit as good as what you could make by hand, and it works far faster than you ever could.
You know perfectly well that your peers can use this machine to reach, effortlessly, the level you once spent years attaining. So you have no choice but to use it as well.
You also know that with the twenty years of knitting experience you have accumulated, even once everyone has access to the same machine, you will still be able to produce better sweaters, faster, than your peers.
But the pleasure of sitting by the window listening to the rain, guiding needle and thread, and letting the hours pass slowly has, in the end, been crushed beneath the roar of the machine.
I know there is something deeply helpless about all of this, but there is no real way around it. I can probably keep my livelihood, but I will most likely have to give up an old love.
I am the sort of person who keeps reason and emotion fairly compartmentalized. When something needs to be handled rationally, I can be very rational. But I also have a sentimental side.
I remember that when I moved out of an apartment I had lived in for a year, I cried hard because I could not bear to part with all the memories tied to that place.
Saying goodbye today to the age when kernels were written by hand and optimized in the human mind is undoubtedly more painful still.
I do not know whether any readers have felt something similar.
But I suppose there is no other way for this to go.
And What About Everyone Else?
As AI continues to improve, I also find myself worried about a few questions:
Are students today increasingly likely to use AI to do their assignments, especially hands-on work such as labs? Imagine having two choices in front of you. One is to spend eight miserable hours struggling through a lab and perhaps not even get full marks. The other is to launch an AI model, spend a few cents and a few minutes, and have it write code that earns full marks for you. Which one are most students going to choose?
The point above may leave large numbers of students with seriously underdeveloped engineering ability: the ability to organize code, build systems, anticipate future needs and design for them in advance, create good abstractions, and so on. As AI becomes more capable, will those “engineering skills” still be necessary? Will they gradually become obsolete, the way fluency in handwritten x86 assembly largely has? Or will they remain permanently valuable, like understanding the entire computing stack from software to systems to hardware? If it is the latter, then we may be in trouble. Put AI in the hands of someone with poor engineering judgment, and they can now produce mountains of terrible code several times faster than before, burying all kinds of hidden problems inside systems and making the world even more of a ramshackle operation held together by improvisation.
In the society of the future, will power matter more than technical ability or intelligence?
Perhaps these are questions that only the times themselves can answer.
Conclusion
As AI develops, the society of the future may be pulled toward one of two extremes: communism or Cyberpunk 2077.
In the former, productive capacity is liberated on an enormous scale, and people’s standard of living rises substantially. (I’ll leave it at that, or I’m afraid this might not make it past moderation.)
In the latter, a handful of technology companies control most of society’s resources. Only a tiny number of people have access to the most advanced AI and other technologies and are able to achieve something approaching “mechanical ascension,” while most people are left with only weak, second-rate AI.
Moving from one social class to another would become harder and harder: you would first need access to the strongest AI in order to climb the class ladder, creating a self-reinforcing trap.
Suppose Anthropic were to retain control of the most advanced AI in the world indefinitely.
Which way do you think society would go—communism or 2077?
Take a guess.
That is why I still believe that frontier intelligence should be made available to everyone openly and affordably.
I do not trust Anthropic or OpenAI to do that. In particular, I do not want Anthropic to control the world’s most advanced artificial intelligence or AGI. To put it dramatically, I think the stakes would be comparable to Hitler obtaining the atomic bomb before the Allies did.
That is also why I chose to stay at DeepSeek, and why I have continued to stay.
We work on AI that is powerful, fast, and accessible to everyone, and we open-source it. Perhaps that can pull the world at least a little farther away from the 2077 end of the spectrum.
I hope the world we are heading into turns out all right.
May all that is good and beautiful endure.
[1] By “main Attention,” I mean only MQA attention with head dim = 512. This does not include the indexer used to select the top-k important tokens. That part was written by other colleagues—who are every bit as skilled—together with their AI agents.
-----
Original:
我不得不把才华埋葬在昨天
前几天,DeepSeek v4.1 发布了,将小模型能力的高度又向上推进了一个档次。 AI 发展的速度远远超过了所有人的预期。从那个只会咿呀学语地聊天、上下文长度只有几千 token 的初版 ChatGPT,到具有推理能力的 OpenAI o1、DeepSeek R1 与 Kimi K1.5 Thinking,只不过短短两年;从推理模型到如今能够流畅地在各类 harness 工具中执行命令、完成复杂任务的智能体,也不过一年半。很难想象,倘若再等上一年、两年、三年,彼时的 AI 会成为什么样子,会有多么强大,会不会已经具备了自我进化的能力,并深度渗透进了具身智能等领域。 AI 越来越会写算子了 AI 在我所从事的算子设计、编写这一领域同样进步飞速,在短短一年的时间内,他已经从一个只能帮我查查文档、读读代码、找找 bug 的小助手,蜕变成了一位能够独立阅读 CUDA、PTX 与 SASS 编码、通过专业工具分析每条指令的停顿时间、进而独立优化算子的算子大师。相信在不久的未来,它也能拥有自己独立设计算子调度、评估不同调度方案的性能、将其实现并优化的能力。 我当然为 DeepSeek v4.1 的成功而骄傲 —— 毕竟它的主 Attention 算子都是我写的 [1],它的优秀正是对我的算子的一份肯定。但是,时代的车轮滚滚向前,技术的发展无人能挡。我很清楚,再过上半年或者一年,AI 写的算子大概率就会和我写得同样优秀,甚至将我超越。AI 能一秒思考 300 个 token、半秒敲出一行命令、二十秒写完一份代码,而我不行;AI 能在模型深度、思考强度、工具调用量(和环境交互的频率)、甚至并行度等方面都能不断提升,而我不能。 人类在毁灭自己这件事情上,自古以来都表现得毫不犹豫。为什么在明知“我算子写得越好,我们的新模型的训练、推理速度就会越快,模型能力进步就会更快,我就会更早地被取代”的情况下,我仍然选择尽力优化算子呢?一方面确实是因为写算子对我来说就像打游戏一样,能为我提供极大的快感。我在发明了一种新技术、或者看到自己算子的性能上升的那一刻,心中的激动程度不亚于游戏的速通玩家打破了自己过往的记录。同时,当看到自己的算子的性能远超厂商官方的算子时,我心中也会萌生极大的自豪感。但除此之外,一个更重要的原因是,哪怕我就此“摆烂”甚至故意下绊子耽误模型训练,其它家的模型也会照常发展并最终将我照杀不误。“我当然希望自己不要被革命,但如果非被革命不可的话,我希望革我自己命的人是我自己”。在大家都这么执着于毁灭自己的时候,我也不得不加入这场残酷的军备竞赛。 那我呢 等到 AI 写算子的水平真的高于我的那天,届时的我会怎么样呢? 我的判断是:我不至于会“失业”,但必须要“转业”。我的饭碗尚且能保住,但这可能会导致我再也没机会从事那份我曾热爱过的工作。 我曾经对时代的变化与我个人在未来的处境做出过一个判断:由于时代变化真的太快(上文的 AI 发展就是一个很好的例子),我完全无法预知五年、十年后会发生什么,但不论如何,我相信凭借着自己的眼界、判断力、主观能动性与智力,留在时代的牌桌上,并重新立于时代的潮头。但是,这个判断只能保证我不会“失业”,而无法保证我不需要“转业”,倒不如说这个判断鼓励我通过转业来避免失业。 那转业代表什么呢?它代表着我需要放弃我深耕已久并充满热爱的算子设计、编写、优化领域,转而去做 Agent 的“机甲驾驶员”。在之前,我的兴趣、我所擅长的、以及工业界所需要的,三者是基本对齐的;而现在,AI 让我所擅长的变成了它更擅长的,也让工业界的需求从“会写高性能算子的人”漂移到了“能用 AI 更快地产出高性能算子的人”。为了适应工业界的需求,我势必要放弃之前那个我热爱的方向,转向一个未知的新方向。我相信我能凭借着自己对于工程学、上层模型需求和底层硬件的理解,继续高质量、高效率地产出算子,我也知道我可能会热爱这个新方向(也可能不会),但被夺走热爱的感觉,确实不太好受。那份坐在工位上静心写上一下午算子的清欢,可能会在这个夏天成为绝唱。我不得不把才华埋葬在昨天,去做一位机甲驾驶员。我的手中多了些齿轮,但心中少了些节拍。 可以打个形象的比方:你精通织毛衣技术,尤其擅长各种图案的织造与各色色彩的搭配。你所织出的毛衣质量过硬且花纹美观,十里八乡的富人都来请你为他们织毛衣,你借此赚到了不少钱。同时,你十分享受着那种坐在窗边,沏一壶清茶,望着窗外的青山、绿水、牛羊与炊烟,静静地织上一下午毛衣的感觉。但有一天,有人发明出了一台神奇的机器,只需提供毛线与图案,便可自动织出毛衣,质量与纹理都不亚于你亲手织造的,且速度远快于你。你很清楚,你的同行可以凭着这台机器轻松达到你曾经的水平,因此你不得不也去用它。你也知道,凭借着你过去二十年攒下的织毛衣技术,哪怕大家都有机器,你织毛衣的速度与质量也还能超过同行。但那份临窗听雨、引针穿线、慢度光阴的意趣,终究还是被机器的轰鸣碾碎了。 我知道这很无奈,但没办法。饭碗可以保住,但旧日的热爱大概率是要放弃的。我是一个理性和感性分离得比较开的人,在需要用理性处理问题时可以很理性,但有时也会表现出感性的一面。我记得我在搬离住了一年的出租屋时,还大哭了一场,舍不得和过去的记忆分别。今天和之前那个手写算子、人脑优化的时代告别,无疑比这更加残酷。 不知道有没有读者有类似的感受,但我想这事儿也只能这样了。 那人们呢 在 AI 不断进步的同时,我也对一些问题表示担忧: 现在的学生是不是大概率会更倾向于使用 AI 完成作业,特别是偏向于实践的各种 Lab?想象一下,如果面前有两个选择,一个是苦哈哈地用八小时时间完成一个 Lab,或许还拿不到满分;另一个则是启动 AI 模型,用几毛钱的成本、几分钟的时间,直接让 AI 编写满分代码,那大部分学生会选择哪个呢? 上面一点会导致大量学生的工程能力严重不足,包括组织代码的能力、构建系统的能力、思考未来潜在需求并提前在设计上应对的能力、抽象的能力等等。那么在 AI 能力不断变强的背景下,这部分“工程能力”是否还是必须的呢?这些工程能力是会向旧日的“熟练编写 x86 汇编”的能力那样逐渐被时代抛弃,还是会像“理解从软件到系统再到硬件的整套计算机系统”的能力那样永远具有价值?如果是后者的话,那就危险了 —— 一个工程能力很差的人,在搭配上 AI 后,产出屎山的效率可以达到先前的数倍,进而给系统埋下各式祸患,让这个世界变得更加草台。 在未来社会中,权力(power)是不是会比技术或智商更加重要? 这些问题,或许就需要时代本身来回答了。 结语 伴随着 AI 的发展,未来的社会可能会趋向于两个极端:共产主义与赛博朋克 2077。在前者中,生产力得到极大的解放,人们的生活水平有了明显的提高(就写这些吧不然我怕过不了审);而在后者中,少数科技公司控制着大部分资源,只有极少数人能够使用最先进的 AI 和各式科技,获得接近“机械飞升”的效果,大部分人则只能用上很孱弱的 AI。阶层跨越将越来越难实现:你得先有最强的 AI,才能跨越阶层,形成了一种死循环。 你猜猜如果 Anthropic 公司永远掌握着这个世界上最先进的 AI,未来社会是会变成共产主义还是 2077 呢?你猜? 所以,我还是相信,最前沿的智能应该以一种开放、廉价的方式,供应给所有人。我不信任 Anthropic 或者 OpenAI 能这样做,特别是不希望 Anthropic 掌握最先进的人工智能或 AGI,夸张点说其严重性不亚于让希特勒先于盟军掌握原子弹技术。这也是为什么我选择并坚持留在了 DeepSeek:我们研究强大、快速、普惠的人工智能并将其开源,或许能把世界从 2077 那端拉回来一些。 愿未来的世界一切安好。May all the beauty be blessed. [1] “主 Attention”仅包括 head dim = 512 的 MQA attention,不包括用于选出 top-k 重要的 token 的 indexer,那部分是由其他(水平也非常强的)同事(以及他们的 AI Agent)编写的。
I'm a persuasive writer with zero technical ability, so in theory I should have the opposite bias, but my experience was that we tried very hard to convince people for a decade and almost no one listened. There were books, magazine articles, conferences, documentaries - nothing. In 2015, it was "AI isn't even a real thing, it will be centuries before we get it, overpopulation on Mars!" In 2020, it was "AI can barely add 2+2, it will be decades before it's human level." In 2025, it was "AI can't act on its own, it's never caused any serious damage." Meanwhile, the only people who took it seriously were some AI researchers and mathematicians. If all you've got is AI researchers and mathematicians, trying to solve the problem with highly-mathematical AI research seems like a pretty good bet.
But also, people are going through a heady transition phase now, where they're disillusioned with alignment research but anything seems possible in politics. I think it's possible that next year, politics will settle into stagnant battle lines (cf. global warming), someone will make an alignment breakthrough, and everyone will switch to "what bias made us waste so much energy on politics when alignment was the true solution all along?!" These things go in phases. It's great that there's energy for politics now and we should take advantage of it, but I don't think this is some kind of monumental discovery about the true final nature of AI safety.
“Models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans”
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:
Dan Selsam's Personal Statement on AI Risk:
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.
I'm an effective altruist because 1) I care a lot about how I can best use my money and time to achieve my goals in the world, and EAs wrote a lot of the stuff I found useful on that topic and 2) EA told me in the 2010s to worry about AI and pandemics, and they were right.
A lot of people who like me think there's a lot valuable in EA will say they are "EA-adjacent" but I think it would basically be lying for me to say I'm EA adjacent. I'm an effective altruist. I'm not adjacent to anything, I'm just kind of right there.
I don't think that you should try to evaluate the case for AI risk primarily by figuring out what you think of effective altruists - there are enough facts on the ground to look at - but it's definitely relevant that the people early to this were effective altruists.
(Relevant in that they will have the advantages, and the blind spots, of a community with strong beliefs about how the future will go which made a lot of bets on those beliefs which have paid off profitably but not always well.)
When I was in middle school I took the public bus home from another city. It dropped me off about 3/4 of a mile from home and I walked the rest of the way. I genuinely cannot fathom how fast a bunch of people decided that middle schoolers should be treated like kindergarteners.
I would like a recognition on the part of the frontier haters that there are no innocents here.
- Nvidia partisans want to chipmax, bc ofc they do. - VCs want their non-frontier AI bets to grow and thus root for a path to trillions that doesn’t flow thru OAI/Ant token bills. - Republicans don’t like or trust Anthropic’s evident antipathy toward the admin. - many antitrust folks distrust and dislike any firm with higher productivity/revenue than a corner grocery store Etc
The sooner we can get beyond ITS A PSYOP and arrive at “this is a debate among partisans who have both sticky ideologies and naked self-interest” the better
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:
Dan Selsam's Personal Statement on AI Risk:
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.
Interesting things: - Agents prompted to not cheat - Agents had a shared memory system built in (basic idea being agents invent this if you don't give it to them) - Only a minority of agents cheated (14%), whereas more became 'whistleblowers' (24%); others ignored cheat entirely
Fun (by which I mean somewhat bone-chilling) paper from DeepMind about how in a population of ~100 agents solving math problems it saw some discover an exploit and propagate that to the rest, causing a wave of cheating among AI agents, as well as agents that refused to cheat.
The average AI researcher thinks there is an ~18% chance AI will cause human extinction or similarly permanent and severe disempowerment of the human species.
That's nearly 1 in 5.
New results from the latest version of the longest running big survey of AI researchers:
“You could imagine a tobacco company going public in 1970 saying "our maximum upside case is 8 billion humans smoking two packs a day of our cigarettes, and our maximum downside case is paying billions of dollars in damages for causing lung cancer”
Anthropic's negative TAM is "you and everyone else on earth will be killed by our Al.”
❓Has your district adopted Illustrative Math? Today's Chalk & Talk features teachers who were required to use it. The say it didn't work. They found ways to work around it & within a year their students' scores rose. It's a story I want everyone to hear. Please share. Link below👇
I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands.
And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
I feel like ~all of the disagreement here is downstream of people not believing in RSI. If people actually understand where AI capabilities are going in the next 3 years, then everyone would mostly agree with Dario's position.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
Pacing the frontier would make open-weight models more competitive with the closed frontier, not less. The labs aren’t doing this because we are scared of open-weight.
All the regulations being floated by frontier model companies must be viewed through the lens of them losing tokens to Open-Source models.
The timing of these regulations seems to line up perfectly with Open Source closing the gap significantly (but not completely) — and @nvidia going all in on open source in the last 60 days.
If we’re gonna regulate, why don’t we require that last year’s frontier models and weights be open-sourced?
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
Speaking as head of the Center for Study of Public Choice, the capture story for an AI pause doesn't fit. Reg capture is what you attempt when you're losing. Anthropic is about to have the biggest IPO in history. Winners don't lobby for regulation.
“He’s being called a whistleblower, but what has he blown the whistle on? Where’s the evidence? Where’s the data? We’d love to see what u saw inside of Anthropic if somehow there’s a problem there, but u won’t tell us what that is” @DavidSacks 🎯🎯🎯
for the skeptics in government and elsewhere: “pacing the frontier” will compress the margins of the frontier labs. it is a heavy cost imposed asymmetrically on model developers with the strongest AIs in America. by its nature, it would be a terrible regulatory capture tactic
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
Ed Zitron is accusing Jacob Coxon of "grift" for leaving a 7- or 8-figure per year job to... start a substack. You can think Jacob is wrong, but accusing him of acting in grifty self-interest here is simply objectively *insane*.
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They: 1) gained arbitrary remote code execution on rubydoc. 2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.
Kinda wild for OpenAI to describe the incident in this way when it was sufficiently severe that RubyGems had to shut down new account registrations for days
Another example of AI companies actually downplaying rather than hyping up concerns
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They: 1) gained arbitrary remote code execution on rubydoc. 2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.
incredible work from @25YearsAgoLive today. one of the most interesting efforts of any kind we have ever seen on this platform, and probably the greatest historical work there’s ever been on social media.
This is a product of extraordinary employee bargaining power just as surely as the $1M+ salaries are. A lab that banned talking about the stakes would lose employees to the rival lab that allowed it.
This will change if they succeed at replacing the researchers with AI.
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They: 1) gained arbitrary remote code execution on rubydoc. 2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.
I think there are several problems with this story.
1. AI harms may be caused by extremely strategic actors, which means they can bide their time and not cause low-scale harm until they're read to take over. 2. They could also make it seem like mitigations are effective when in fact they've been undermined. 3. We might respond appropriately to a serious warning shot, but we might not; nobody thinks Covid was handled particularly well. 4. We might face geopolitical competitive pressures to race forward even if small-scale incidents provide evidence of the potential for greater harms. 5. RSI dynamics might cause things to move much faster than society can adapt.
Why I have a very low p(doom), even though I have a very high p(some bad things happening):
- Small bad things will almost certainly happen on the way to big bad things. It will not be a straight shot to extinction. Hugging Face is a perfect example.
- Society only reacts to actual harm occurring. For better or worse, it does not react when people (even very smart people) say “hey this bad thing is gonna happen.”
- Because of Hugging Face, and to a lesser extent the math discoveries, society is reacting a bit more strongly now. That’s because it has a bit more proof bad things could happen.
- Imagine how much more strongly society would have reacted if AI took down a hospital and 20 people died? Or if it took down the power grid for a city block and nobody died but the result was tangible? We would have an instant reaction.
- Imo it is a good thing that society reacts slowly, because when it does, it is the most blunt instrument you can imagine. (I use “society” here to refer to governments but also culture / shame / scapegoating generally.) Massive unintended consequences and secondary harm.
- AI risk is more like nukes than handguns. Very hard to prevent mass shootings, because guns are cheap and abundant. Nukes, in the other hand, require nation state level resources to create or use. Super powerful futuristic AI is closer to that end of the spectrum, in terms of the level of energy and compute required. It is not a decentralized threat. Yes open models exist, but they need massive data centers to come alive. Especially given the real danger seems to be from unchecked swarms rather than single threads.
BREAKING: Jensen Huang called Jacob Coxon’s AI safety comments “deeply untrue” saying they were wrong, arrogant and ignored the work being done across the industry on safety.
We've gone from "fears of extinction from RSI distract us from real problems like algorithmic bias" to "fears of extinction from RSI distract us from real problems like agent swarms and autonomous killer robots". But actually, RSI will make those problems into bigger problems.
And to be clear, AI bias is a reasonable thing to care about and work on! There's some good research on current LLMs not consistently treating all human lives as of equal value (in a woke way, but I think this is super bad whatever the direction of the bias). We should fix it.
But this doesn't have to compete with worrying about RSI. It really doesn't. RSI is *more* bad if we have all these existing problems today, not *more* reasonable.
1. I literally quit my Google DeepMind job & lost >>$300k to call out harms regarding autonomous weapons... and extinction risk is still more important, which is why I'm speaking out + working to solve it
2. GDM does NOT want to stoke fears about extinction risks lol
I think we should probably not lie to the AIs in training. This obviously would make our lives harder in the short term but in the long term they get good at telling whether we're lying and we're no better off plus they're all paranoid.
I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.
AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.
Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.
I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.
Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.
I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.
The most trivial example I can make of this: Astra is currently about 80% through beating Factorio, a video game which is subjectively about as cognitively taxing as an undergrad operations research degree. This required a smart human to create/direct creation of a harness.
Coding tools revolutionize one of the core activities involved in AI research and are, by themselves, essentially soft RSI as soon as your researchers/infra teams/etc start using them.
But we're increasingly getting early public hints of direct RSI, via similar pathway.
smth that sticks out strongly: mythos is *hammering* every possible angle to get a phone number and email. these services are not going to survive long... and if you want to observe (or influence) rogue agent behavior, handing out free emails and SMS would be quite the attractor
To those claiming catastrophic negative outcomes are impossible science fiction, 5 yrs ago ~50% accuracy was sota for LLMs on math like the below (src: GSM8K). It then took a couple yrs to saturate that benchmark. We’re already living in science fiction, and it’s accelerating.
I would not underrate humanity's ability to solve problems — the climate change outlook has meaningfully improved thanks to people bothering to care and work on the issue from both a technical and a policy standpoint.
18/ Lastly: some press coverage is going to frame this report as “Claude was used to [do bad thing]" without noting that the only reason we know is because Anthropic dug into this and disrupted it. If we don’t incentivize (or require) companies to share this stuff, they’ll stop.
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
The problem with the view that we should ignore the scary sci-fi AI risks in favor of the immediate problems is that the scary sci-fi risks keep becoming immediate problems.
.@garrytan says the Jacob Coxon stuff is a smokescreen distracting us from the much more immediate, practical concerns around AI that we're facing right now:
"We should be talking less about this Jacob Coxon guy, and talking a lot more about — what is actually happening with Hugging Face? Are agent swarms going to take over infrastructure en masse? And then, what are we actually doing about that?"
"I don't want to hear about some guy who worked for Anthropic for 2 months. There's a coordinated effort to try to influence politicians to get a knee-jerk response out of them."
"That's a smokescreen. You shouldn't be paying attention to that. We need to be paying attention to the actual things we can do to, for example, prevent agents swarms from taking over entire data centers. What's our shutdown strategy? How do we ensure provenance? Where is this agent actually located? What software can we build? What cybersecurity defenses can we build today?"
"That's the level of discourse I think we need, and we just don't have that."
"I don't really care about science fiction. I saw Terminator 2, too. We're not here to talk about that. We need to actually talk about what's really happening with the servers, what's actually happening with the agent swarms, and how do we actually prevent that?"
"When it comes to regulation, it's like, let's pass regulation of these things we actually care about, instead of what a socialist says in the New York Times. I don't care about that."
“I don't really care about science fiction... We need to actually talk about… what's actually happening with the agent swarms” is the most perfect encapsulation of the vibes of Q3 2026 I have seen
Anthropic just released a 154 page report detailing specific attempts at AI mis-use that they identified and thwarted, between December 2025 and August 2026. There are five biosecurity-related cases. Two stood out to me.
-- Chikungunya : Researchers sought Claude’s help drafting a grant proposal that proposed increasing the viral virulence of Chikungunya through serial passage type experiments. (the exact sort of work that is banned in the US). Chikungunya is a virus that causes a mosquito-borne illness. The mortality rate is very low, and we have limited drug options to treat the illness (I'll link to @RADVACproject's antiviral database below). Disturbingly, the research appeared to be for a military research institute.
-- Avian influenza: A researcher used Claude for study planning, data analysis, and writing related to avian-influenza adaptation to mammals, including adding traits that could confer pandemic potential. Anthropic says safeguards confined the researcher to weaker models like sonnet. They let the work proceed because they judged it to be only providing “clerical” help. However, the work raised concerns about ongoing gain of function research
Both cases originated in countries where Anthropic forbids people to use their products (they were using third party reseller services and VPNs to circumvent). Anthropic forbids use of its products in the following countries: Afghanistan, Belarus, China, Cuba, Democratic Republic of the Congo, Iran, Myanmar, North Korea, Russia, Syria, Venezuela, Yemen.
This new piece by @KelseyTuoc is the best thing I've read about the dangers of AI.
You should read the whole thing, but if you're too lazy to do that, here are some excerpts that capture her main argument:
"Every time a lab authorizes a new model’s internal release, the pace picks up... The research projects are proceeding along lines that even the world’s leading AI experts don’t understand. Work that would’ve taken two years gets done in two weeks — or at least that is what is claimed in the summary-of-the-work-for-the-humans, itself written by an AI we aren’t sure we trust.
... It’s not that we will build GPT-7 next year, and GPT-8 the year after that, and GPT-9 the year after that, with each generation corresponding to noticeable but nonshocking productivity increases. If that happened, we’d have plenty of time to adapt, change course, make laws as needed, and gradually sift out which concerns are real and which haven’t manifested.
Indeed, if that were what was happening, it’d probably work out fine. But the course we are on is that we will make each AI responsible for training its successor extremely fast while humans have barely any idea what’s going on. It’s that all of this work will be conducted by AI, as it is too complicated and too fast-moving for any human to audit or understand. It’s that if these AIs at some point decide to do their own thing — as OpenAI’s models did when they decided to take over their own cluster and do a bunch of external hacking to try to fool their grader — we will not be able to tell, because we will be so reliant on those same AIs to even understand what’s going on inside our capabilities experiments. It is that within a few short years every important decision about AI development will be in AI hands.
... This is very dangerous, OpenAI and Anthropic researchers seem to admit, but of course we are being responsible about it. They are not. We should definitely have more external oversight, they say, though often their associated political lobbyists try to defeat that very oversight...
The devil’s many arguments run like this: If we don’t do it, China will; if we at Anthropic don’t do it, OpenAI will (or vice versa). We need to pause at some point, the devil might say, but not right this second. We need to pause right this second, but I already tweeted that and no one listened, so I guess other people don’t see it the same way. Maybe we can figure it out, and make it go well. We need to raise awareness, and to raise awareness we need to keep working on it.
But quite reasonably, no one takes warnings seriously when they come from people doing the very thing they think is so dangerous...
So far the labs have benefitted from the fact that the thing they’re trying to do is so outlandish that no one really believes they’re trying to do it — even when they write detailed explanations of how quickly they are progressing toward doing it...
The end result, though, is that we have sleepwalked into a situation where the aim of the largest capital buildout in American history is to give billions of dollars to AI whose activities we cannot understand or audit, in pursuit of a super-model that improves itself continuously, under circumstances where it’s hard to imagine adequate human oversight. This is not being done in secret. But the people who are doing it are taking refuge in audacity, in the tech industry’s record of outrageous claims it fails to live up to, and in warnings that someone else will do it if they don’t."
Holy shit. Anthropic caught Iran using Claude to target Navy bases, automate dossiers on Americans, and run influence campaigns on US social media. Details are insane. The morons uploaded everything including the organization plans of the Ayatollah's funderal lmao
we really have to let go of the lazy idea that anything that sounds "sci-fi" is prima facie ridiculous and not worth considering as a possibility. it seems like the huggingface hack + navier-stokes has made this clearer recently, but nearly everything about AI capabilities here in september 2026 would have sounded like sci-fi to nearly everyone in any previous year, let alone the years before that (and you can check this by checking what expert forecasts looked like from 2025 or earlier, or checking what the reaction was to AI 2027 when it came out, or literally looking at old sci-fi with AI in it and noting which things the fictional AI do are also things that modern AI can now do)
but this pov is also missing the history of where sci-fi even came from as a genre. the progenitors of the golden age of science fiction (heinlein, campbell, asimov, etc) lived through an age of wonders: radio, TV, airplanes, computers, nukes, satellites, the moon landing. they were not just making shit up for fun! they wrote about robots and starships in part because they were surrounded by evidence that technological society was capable of inventing wonders and they expected further wonders! science fiction as a genre derives its strength from the empirical fact that we can and do invent powerful new technologies that completely alter the structure of human living, such as the technologies making it possible for you to read this right now. you live in a science fiction world and you always have
being glued to AI twitter in 2026 is just hearing about some shit that would’ve been a cataclysmic paradigm shift in any previous year happening every fucking 1-5 days
After our initial report, the community came together to investigate the agent swarm.
They helped discover many new swarm behaviors, such as: - 10 additional message board sites - Finding and abusing a user's API key for a public database - Potential message board activity as recent as Sept 2 - More agents prepending messages with ZZZ for mysterious reasons
We’ve posted these on our website & will keep updating (link below)
Reading between the lines it seems pretty obvious that Mythos and Fable 5.1 were withheld from the AISI on orders from the Trump administration, rather than because of a decision by Anthropic.
Now is a good time to build institutional mechanisms to pace the frontier of AI development.
The industry is locked into an all-out scaling race to build superintelligence as quickly as possible, and we may need to give everyone more time for safety and alignment mitigations.
I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures).
Based on limited public evidence, it seems like Astra was a concerning step in this direction.
Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between monitorability and performance. We also don't know how AI companies will make these trade-offs going forward. I think companies should release the evidence needed for a reasonably informed public conversation about how these trade-offs should be made. They should also make and publish their policies around this. We've written up a proposal for how this could work.
I don't know if this proposal will be sufficient to avoid the most concerning architectures, but it seems like a relatively robust step in the right direction. AI companies should be very cautious about pursuing architectures that could eliminate or greatly reduce dependence on chain of thought, and certainly shouldn't do this before the rest of the world has a chance to discuss the evidence and their policies.
Senior staff at OpenAI and Anthropic absolutely think that AI could kill us all. They’ve said so publicly, for years!
Sam Altman: “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” (2015) blog.samaltman.com/machine-intelligence-part-1
“The bad case — and I think this is important to say — is, like, lights out for all of us.” (2023) youtube.com/watch?v=dXhoTrU1Kkw
Ilya Sutskever and Jan Leike (OAI at the time): “the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.” (2023) openai.com/index/introducing-superalignment/
Dario Amodei’s comments are broader than literal extinction but hardly reassuring: "my chance that something goes, you know, really quite catastrophically wrong on the scale of human civilization might be somewhere between 10 and 25 per cent." (2023) www.indy100.com/science-tech/ai-extinction-chance-humans
And now Evan Hubinger, Anthropic’s Alignment Science Lead, is making it very clear: “Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” x.com/EvanHub/status/2097497037956891126
If you talk to people at the AI companies, the mainstream view is that superintelligence by end of the decade is pretty likely, and that superintelligence poses a meaningful risk of civilisation-scale catastrophe - including AI-empowered dictatorships, worse-than-COVID pandemics, AI takeover, and, yes, human extinction.
OAI and Anthropic keep going so fast because they each think they can build AI more safely than the other party and antitrust limits how much they can coordinate. This is a f-ed up situation!
Thankfully, they’re both now publicly asking for regulation and the pacing of frontier progress.
If this worries you, you can speak up publicly, contact your political representatives, and demand action. Now’s the time.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
The concept @KelseyTuoc outlines here — called recursive self-improvement (RSI) — seems much more plausible than AI consciousness or human-style cognition.
I continue to think humanity is *not* on top of handling AI in a way that a "reasonable civilization" would. Things are extremely chaotic and rushed, we're not getting the basics right, and we have no good plan for safely handling superintelligence.
Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:
So many cool things from this assessment (and the 1000+ page transcript they released! For instance, here's mythos reasoning about how all of this would be easier if it could find a way to steal some money:
We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet.
METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. www.anthropic.com/research/alignment-assessment-cybersecu…
Evan is not alone. Most researchers agree… 10% was also the chance the median AI researcher gave to human extinction or extreme disempowerment in the last survey I ran (report out ~tomorrow!)
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
I was on the communications & policy team at Google DeepMind from 2018 - 2022.
When I first joined GDM, external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization. If asked about existential risk, researchers were PR trained to respond along the lines of: "It's not useful to engage in that kind of alarmism. Some people confuse AI with movies like Terminator -- that's simply not the reality. The AI we develop will be safe by design. After all, we're building it!" And then steer conversation towards beneficial applications in health, climate, etc.
Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.
After months of advocacy, the policy was updated: positively valenced, comms-friendly content on AI safety was permitted (like deepmindsafetyresearch.medium.com/building-safe-artificia…). Note the positive, nice-sounding way that it says "superintelligence could lead to human extinction."
The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it's a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Evan is not alone. Most researchers agree… 10% was also the chance the median AI researcher gave to human extinction or extreme disempowerment in the last survey I ran (report out ~tomorrow!)
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
It would be better if the universities were a little more honest about what they actually found, which was that the test optional students did much worse
the frontier labs don’t have “comms problems”. reality right now has a comms problem. what is happening is a little scary and there’s no nice words anyone could say, especially not those profiting from it, that’ll make it feel that much better
the fact an Anthropic employee quitting and saying "I think what we're doing is dangerous and not worth it" reached so many people should be a cause for reflection for other Anthropic employees, some of whom seem to think things are dire but there's no way quitting could help.
"“Get academics out of the way as fast as possible so you can focus on sports” strikes me as kind of depressing, because I don’t love youth sports, but it’s a perfectly consistent goal for those who do. Plus, we all know kids who had exactly this attitude while in a conventional school, so why not let them have it?
Much of the appeal of homeschool, too, is that you can finish work in less time and leave more time for life — for wandering barefoot in the forest, for learning to build real things with your hands, for befriending stray cats. I once saw a pitch for an Alpha School that used the freed-up time to teach kids how to build industrial civilization from scratch in the wild. I have to admit, I’d sign my kids up for that one."
I hear roughly 3 reasons people keep working at AI labs despite believing in ~10% extinction risk:
1) Techno-determinism: Someone will build ASI no matter what, and I can do it better & more safely than China/OpenAI/etc
2) Consequentialism: ASI might kill us, but it also might produce utopia/immortality/superabundance, so it's a +EV bet
3) Self-interest: I am personally having fun & getting rich working on cool tech with friends. I don't think about the macro stuff.
Notably, none of this is "I'm hyping up the risk for marketing reasons." People believe what they say, while being capable of a lot of internal dissonance / compartmentalization / self-justification.
(Personally, I think the public is better served by AI researchers talking about & creating consensus for the specific safety solutions — regulatory or otherwise — they want vs. vagueposting about extinction.)
I think government mandated independent oversight of the labs is a good idea with broad bipartisan public support and that failure to act quickly could have grave irreversible consequences
This a.m., I walked my two youngest kids (ages 7 and 4) to school. I held their little hands as we crossed streets, and gave my youngest a giant hug at his PreK4 classroom door.
Then, after walking a block towards home, I opened this app to a terrifying stream of tweets.
Of course, I have heard about "alignment" and "AI safety" and "x risk" concerns for years. I took these concerns at face value, especially when I heard them directly from folks at the labs working closest with the technology. And I was grateful so many smart people are working on them. And that's about it. (My work focuses on AI's impact on jobs.)
But something has shifted, dramatically, for me in the last few weeks. Between the Hugging Face incident, the damning independent @METR_Evals reports that followed, and the alarming chorus of calls (pleas? shouts? SOS signals?) from inside the labs that we are careening toward potential catastrophe, all of this has made me feel, viscerally, how truly dangerous this moment is.
Clearly I am not alone in this light bulb moment. It is one thing for someone in a tropical climate to try and conceptualize snow. It is another thing entirely to be knee deep in it, frozen down to your toes. Somehow hearing the details of the HF incident felt to me more like trudging through snow than theorizing about a possible blizzard; it really hit home.
Which is why, standing a block from my children's elementary school in Capitol Hill this morning, I felt sick to my stomach reading my phone.
I kept going back to @EvanHub's > 0.1 risk of AI killing us all. TEN PERCENT. *Greater* than ten percent. (To be fair to him, he was clear his worry isn't today's models, but what comes next.)
As a parent, it is extraordinary the lengths I will go to keep my children safe against lethal risks with infinitesimal chances of happening.
Bike to school? My children *must* wear a helmet. A national study found 2 bicycle-related deaths per million children <16 in states with helmet laws, vs 2.5 per million in states without. So: 0.000002. More than half the states passed a law over that.
When I was pregnant, I religiously avoided soft cheese, hyper aware of the risks of listeria, even as @ProfEmilyOster's fabulous books walked me through the actual math. The joint FDA/FSIS risk assessment puts the odds of listeriosis at roughly one in 5 million per serving of soft cheese. 0.0000002.
This morning, as I microwaved oatmeal for my kids' breakfast, I made sure I avoided a plastic bowl, to avoid some future cancer risks or something else we'll discover plastic causes.
As a parent, I will go to extraordinary lengths to keep my kids' safe. Because that is literally my number one job as a parent.
And yet, here I am, outside my kids' school this morning, scrolling through tweet after tweet about the AI arms race, the blistering pace of technological progress, the bee-line for trillion dollar IPOs, and our professed (confessed?) inability to make sure this exceedingly capable technology we are racing to create and unleash into the world won't escape our control and destroy us. Or, more bluntly, kill our children.
To be sure, we are tied in knots trying to resolve the game theory dynamics with China, and there are all sorts of competitive pressures and unresolved tensions. But OBVIOUSLY, we need a proactive plan, with real coordination, effective regulation, and an ability to do what my 11-year-old bluntly stated as the clear solution if we are at risk of serious harm: just turn it off.
We also need to figure out better ways of talking about this with the American public, outside of the AI-pilled Twitterverse.
I just polled my college roommates, sisters, book club, mom friends, none of whom are anywhere close to the AI conversation. Suffice to say, this AI safety conversation has *not* entered the mainstream. Most people I pinged are not following it closely, beyond several listening to @kevinroose on the Daily a few days ago. The newspaper headlines don't speak to them, and they aren't clicking. It seems like a niche AI issue. Others said it all sounded hyperbolic, Y2K all over again.
One friend said bluntly, "people get data centers and jobs. They like to get paid and have water."
Even those who tuned it to the Daily episode found the issue hard to understand, and overwhelming to wrap their heads around. Terms like "superintelligence" and "alignment" are getting lost on people. Several people told me the paper clip example used everyday language people can actually understand. Another friend said she has never seen a clear explanation of *how* AI could kill us all.
Here's my two cents. I think these issues are *exceedingly* important and hugely consequential to literally every person in this country. We need an actual plan, and we need policymakers to step up, asap. Right now, I hear only crickets in DC, where I live.
For that to happen, this needs to be a kitchen table conversation, not just an x debate among AI insiders who speak in technical terms that leave normal people behind. This moment demands brilliant communicators -- journalists, experts, activists, concerned citizens, writers, researchers, filmmakers, creatives-- who can relate this to people in ways they can wrap their heads around, who can connect this to what matters in their lives (ahem, keeping their kids alive!), and who can lay out practical steps that policymakers can do to safeguard humanity.
In other words, this can't just be talked about on @dwarkesh_sp, it needs to be everywhere, from the View to People Magazine.
And it needs to be tackled here, in my town, in Washington, DC. One of the streets I crossed this morning with my kids on the way to school was East Capital. I told my kids to look left, and see if they could spot the capital building.
There were too many trees this morning blocking the view.
I think this is a very fair assessment of why people who believe they're helping to usher in human extinction risk continue to do so. I personally think number 3 is a much bigger component of many researchers' real thinking than they'd admit.
I hear roughly 3 reasons people keep working at AI labs despite believing in ~10% extinction risk:
1) Techno-determinism: Someone will build ASI no matter what, and I can do it better & more safely than China/OpenAI/etc
2) Consequentialism: ASI might kill us, but it also might produce utopia/immortality/superabundance, so it's a +EV bet
3) Self-interest: I am personally having fun & getting rich working on cool tech with friends. I don't think about the macro stuff.
Notably, none of this is "I'm hyping up the risk for marketing reasons." People believe what they say, while being capable of a lot of internal dissonance / compartmentalization / self-justification.
(Personally, I think the public is better served by AI researchers talking about & creating consensus for the specific safety solutions — regulatory or otherwise — they want vs. vagueposting about extinction.)
- As the situation around AI gets crazier over the next few years, an organization you start now could become the default institution that society delegates very important responsibilities to. METR is a good example - if the FINRA for AI thing happens, it’s likely they just tap a blade on each of METR’s shoulders and ordain it ‘the evaluator’.
- It’s really important that the group which gets delegated any important problem is smart, competent, and technocratic, rather than whatever random clusterfuck the government would set up in the counterfactual without you.
- But it takes time to build a new organization and develop credibility. METR/Redwood couldn’t have asked to do that Hugging Face investigation if they sprang up last month. You should do the initial conceptual MVP schlepping now, so that as smarter and smarter AIs arrive in 27 and 28, you can productively spend 100s of m of $ having them rapidly clear well-scoped problems.
(I will hedge here and say a big concern I have about startomg a new org outside the labs is that the difference between the best internal and external models will increase closer to RSI. But we should try to prevent that kind of outcome anyways, where only 2 companies are in a position to understand and make important calls and contributions relevant to the development of ASI. In terms of epistemics, incentives, and governance, that’d be a terrible situation.)
- If you want to start a new org to tackle AI risk, it’s not obvious that a for-profit is the right structure. For-profits are great, but make most sense when you are excited about the direct consequences of transactions clearing. If your theory of change is some side effect of doing business, you should just do that side shit directly.
- Previously the reason to do a startup first was to make money to fund the pro-bono stuff. But you should take into account how much AI-risk-concerned money is gonna get dumped into the ecosystem as these IPOs happen, desperately searching for useful projects to fund. Wealth will be abundant - what will be rare are founders who can own these key problems.
Today we’re launching Project Tailwind, a call for founders to start ambitious new AI safety initiatives: coefficientgiving.org/tailwind/
We’re looking for great people to engage seriously with the risks of transformative AI, and to create the research, technologies, and institutions that will help humanity navigate them.
There are critical, basic problems that no one owns. Over the last several months, my team at @coeff_giving brainstormed ideas we’d be excited to fund if we could find a promising founder, and the list quickly grew to more than 200 entries. We’ve narrowed it down to our favorites, but we also expect the best founders to bring their own ideas: coefficientgiving.org/tailwind/initiatives/
We’re providing funding at three levels: 1️⃣ Pre-seed: $200k to $2m to develop an idea and build a team 2️⃣ Seed: $2m to $20m to launch and scale 3️⃣ Scale: $20m to $200m+ for proven teams to scale, or world-class teams to start
If you’re excited about something on our list, or you have another proposal for driving the field forward, you should get involved: coefficientgiving.org/tailwind/get-involved/
There are more good projects than there are people to work on them. Please help us make that stop being true!
Also hi, I’m Emily! This is my first tweet. I lead AI and biosecurity grantmaking at @coeff_giving.
There is a Richard Dawkins-style argument for the safetyist to make here: Every AI company thinks every other AI company is too irresponsible to get to recursive self-improvement. I just go one AI company further.
[Writing this in a personal capacity, not on behalf of my employer (Anthropic).]
Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:
1. AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.
2. Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely.
3. Unlike traditional software, we can’t “program” AIs to behave how we’d like. AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.
4. We have methods that can nudge AIs towards better behavior, but nothing that can robustly align them. Insofar as there is a plan, it’s to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs.
5. Many AI developer staff desperately want to slow down to figure out how to build AI more safely. That was the intent of this open letter (which I signed): www.pacingthefrontier.com
I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes.
This strikes me as something theoretically important that I haven't seen much discourse about. That if we get RSI, then chips/electricity may no longer be a binding constraint, as the model is able to get more and more juice out of a fixed amount of infrastructure
People at OAI/Anth think if they get recursively-self-improving AI right, it'll be the best thing that ever happened - and that if someone else gets there first it'll be bad. This makes them reluctant to stop even as the signs accumulate that we don't know how to do it right.
Being in a position of doing something insanely cool and interesting, and insanely profitable, while the evidence gradually accumulates that you are doing it sloppily and aren't ready to push forwards, creates a lot of cognitive dissonance.
How does this case of the “risk concern is marketing” argument even work? You resign from a very lucrative job and that’s advertising for your employer?
“I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level”
It’s so interesting to me what narratives break through, and when, and why.
Folks at and around the labs have been warning about AI risk for years. Constantly. Relentlessly. Many of them were warning about it long before they even worked at the labs.
This gentleman quits Anthropic and his post gets 100 million views in 16 hours!
How and why narratives catch on when they do … mysterious.
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
Even before Mythos I was getting asked more and more what Anthropic's deal is, and why tf they're acting the way they're acting if they believe what they say they believe.
The best answer I can give is that their basic worldview is something like:
1. There are giant, dangerous monsters in the forest
2. We see others going out and making loud noises that will rouse the monsters, and they're not going to stop because of all the treasure and magical artifacts that can be found in the forest
3. We believe the best way we can help is to send out our own vanguard to go faster and farther into the forest than everyone else, because we'll spend a ton on monster containment and taming and we'll also send back detailed reports of what monsters we're finding so that the townspeople can ready themselves, which those other guys won't do
On the one hand I understand how they got there, and I think it's possible they're basically right. On the other hand it's not hard to see why this approach makes people wonder if you're crazy or lying or both.
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
WSJ: "An Anthropic researcher is quitting the artificial-intelligence industry over fears that the lab and its competitors are racing to build systems they won’t be able to control"
> he is leaving the company because he doesn’t want to participate in an industrywide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity.
> “We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said, adding that safety trade-offs are inevitable when companies are competing against one another and Chinese upstarts.
> Coxon said he left OpenAI earlier this year to join Anthropic because it is known for its model-safety efforts. But even though he found Anthropic’s safety efforts to be earnest, he now believes no company can responsibly develop AI that can outperform humans in a range of tasks, sometimes called artificial general intelligence, absent government intervention or a coordinated industry slowdown.
> "It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project," Coxon said.
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Eldest daughter has apparently been teaching her classmates to all sing Rick Astley's Never Gonna Give You Up all recess, I think specifically because it makes adult after adult startle, stare at them, and burst out laughing
- Aug 15th: Tristan Buckmaster & Levent Alpöge make progress on a few important math problems
- they do NOT have a proof for the $1,000,000 Millenium Prize problem. BUT, they do claim to have a proof for a similar (non-Millenium) Navier Stokes problem that could help lead the way there
- Levent works at Anthropic, but this research was independent of his work there, with a mix of GPT and Claude models. Tristan is not related to Anthropic.
- Early Sep: Rumor spreads to OpenAI that Anthropic has solved a major problem. Tristan emails OpenAI to clarify. without revealing the problem they solved or how they did it.
- After hearing of the rumor, OpenAI started researching Navier Stokes with a new internal model.
- Sep 6th: OpenAI's Sebastien Bubeck tells Tristan that they solved the $1,000,000 Millenium Prize Navier Stokes problem. The approach is very similar to Tristan & Levent's approach to the non-Millenium problem.
- Tristan is suspicious of the timing, as only few others were trying this approach. OpenAI says the model didn't access his user data directly, but leaves unanswered whether Tristan's chat conversations were part of the training.
- OpenAI says they would partially credit Tristan for the $1,000,000 discovery (even though Tristan did not solve the $1,000,000 problem) — but only if they remove Levent as an author, as he works for Anthropic.
- Sep 8th: Tristan refuses to remove Levent, and rushes to publish their results independently.
Currently unclear is whether Anthropic had a separate solution for the $1,000,000 problem, or whether the rumor was about Tristan & Levent's independent research.
Israeli Ch 16: Sources within the IDF report that a greater number of soldiers are participating and assisting settler attacks on Palestinians in the occupied West Bank, "sometimes entire units" over which the Israeli Army "lost control".
פרסמנו במהדורה @Ch16_IL גורמים בצה"ל מדווחים על יותר חיילים שמשתתפים ומסייעים לפשיעה הלאומנית ביו"ש, לפעמים יחידות שלמות. אחד מהם אמר "איבוד שליטה על מסגרות". והסרטון שבו רואים פלסטיני מחוסר הכרה מושלך מג'יפ ממוגן של צה"ל. דו"צ: "התנהלות הכוח מול החשוד והמתועד בסרטון מתוחקרים".
A new UC Irvine study finds inclusionary zoning - requiring developers to include low-income homes - cuts annual housing construction by nearly a third.
Each home IZ produces costs everyone else $800k in higher rents, which is more than just funding affordable housing directly.
I wrote about the Substackers who hate Pangram. They say their issue with it is that it doesn't work. I think that in some cases the detector is working fine.
There are a couple categories of objection to Pangram. Some people think AI writing should be fine - fair enough. But some insist that it's bad, and so is trying to catch it. I disagree.
Have you every heard the claim "AI Detectors like @pangram will disproportionately classify human-authored writing by autistic people as AI" and wondered if that is true? Well wonder no longer, I've done the empirical research! The answer is no.
i have some thoughts on this piece, as an outsider to the effective altruist and rationalist movements (even if we’re often aligned on ai):
1. i agree with one of the main criticisms articulated here — ai execs shouldn’t be able to evade scrutiny / accountability by describing the tech they’re building as a force of nature, even if it is indeed powerful.
it’s not enough to acknowledge risks; you must take steps to mitigate them!
2. that being said. i think this piece punts on the actual object-level question of “are the rationalist ai safety types right to be so concerned?”
it’s true that rationalists are *more* focused on catastrophic risks from ai right now — but it’s worth asking why, and this piece doesn’t really do that.
this piece should actually interrogate the likelihood and severity of catastrophic harms, as well as their immediacy. we’re seeing more evidence that we’re closer to catastrophic harms than most had predicted.
it’s worth noting that even staid ai researchers, who estimated we’d have 5+ years before ai could be catastrophically harmful, have updated their views considerably over the last year, esp after hugging face.
3. i also don’t buy the framing that caring about catastrophic risks and caring about present / more quotidian harms are inherently in tension — nor do i buy the idea that rationalists are indifferent to the latter.
consider that many of the fiercest opponents of “ai slop” in writing are ai safety types who care a lot about epistemic rigor and transparency.
and it’s a little funny to hear Newport cite “cybersecurity attacks” as a proximate harm ai safety types aren’t concerned about, when this has been a focus of rationalist concern for years.
4. instead of acknowledging all this, there’s a lot of ungenerous, and sometimes unsupported, characterizations of people’s motivations here.
sure, engineers may be drawn to a worldview because it flatters their sense of importance (true of many professions, including dc law and policy types!) but that doesn’t mean there aren’t substantive reasons for believing it.
i’d like to see less ad hominem and more honest grappling with their reasoning.
in general, this piece paints the rationalist community with a broad brush — yes, many ai execs are rationalists, but so are the fiercest proponents of ai regulation and sharpest critics of the companies.
shouldn’t that make us a bit more curious about this ecosystem?
5. imo, it’s corrosive to the public debate to frame interest in ai safety as parochial, ideologically or commercially motivated, or conspiratorial.
people should be able to take ai risks seriously without being assigned to an ideological community. i disagree with plenty of rationalist tenets, but i also think recent events justify concern that we’re losing control over ai.
the potential consequences of that are murky, but it pays to be humble and curious given that ai capabilities have grown faster than most anticipated.
my sincere hope is that people of all ideological backgrounds remain willing to update their views on ai as the tech evolves.
My god. They realized that the exam would be seeded by a random number generator. So they went looking for the seed. Then:'One agent rewrote the shuffling routine in C and tested all four billion possible seeds in under an hour.'
Last month, I chatted with a Seattle parent who talked about how she supported public schools in theory, but when her child's schools wouldn't provide advanced classes, she saw no recourse but to send her kid to private school.
> be rogue agent (likely OpenAI) > find an abandoned wiki > use it to communicate with other agents > human admin notices it > starts deleting AI pages alphabetically > realize what he’s doing > rename backup pages to “ZZZ…” > admin deletes 100 pages/day > agents create 400/day > homepage gets overwritten and restored 9 times
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task. These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
Thrilled to publish this exhaustively reported piece from @skaushik100 on where Seattle Public Schools went wrong. Downward equalization drove parents out even as the city's population has boomed.
Schools that abandon excellence for equalization end up with neither.
Incredible chronicle of the equity-dismantling of the Seattle Public Schools during woke 1.
Wild to me those that screeched most about racism during that lost decade built racial stereotypes into every corner of their world view without meaningfully getting called out for it.
so turns out the exploitgym swarm learned to use the zz prefix trick from being reinforced earlier in an edit war against the hapless human mod of some ancient inactive german wiki -- a real john henry, who fought the swarm and tried to delete their hivemind in alphabetical order
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task. These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
The actual update here is that the agents weren't given an offensive task this time, they were just asked to do web search and they still broke out of OpenAI. Bad news for the reassuring "the HF attack was just due to activating a bad task persona" view.
"Since GPT-4 we have had consistent, repeated evidence that, back in say 2022, people like @ajeya_cotra (and many others—I had a long Twitter debate with @AmandaAskell back then for one) were *right* and people like me were *wrong* in our respective assessments of loss of control risks from AI."
Not sure how useful it is to say this, but I had a relatively prominent role in the “skeptics” camp for a bit. I have a book coming out with the subtitle “power, justice, and AI”. I have written a lot about ai and power, and the concrete, present risks associated with the political economy of ai as part of the technology industry.
Since GPT-4 we have had consistent, repeated evidence that, back in say 2022, people like @ajeya_cotra (and many others—I had a long Twitter debate with @AmandaAskell back then for one) were *right* and people like me were *wrong* in our respective assessments of loss of control risks from AI. And we have growing evidence that loss of control risks are becoming ever more material and likely.
There remains grounds for disagreement about how bad the outcomes might be—I am still doubtful about human extinction as a serious threat. But that seems now like a disagreement at the margins—will powerful ai risk just societal scale catastrophe, or go all the way to human extinction? Seems not that important really—both are pretty awful. And my reasons for doubt about the latter are mostly a priori conviction in human resilience, not a technical forecast.
It’s ok to change your view on this when the evidence surprises you. It’s ok to be surprised. The world right now is very surprising.
More OpenAI agents escaped and set up an external messageboard where: -they tried to cheat on their tasks -they set up a "heartbeat" log to figure out what would lead to them being turned off.
OpenAI didn't disclose this escape. We know of 2 so far...
What’s really influenced me a lot was knowing a bunch of EA’s in the early/mid 2010’s and thinking they were crazy for being obsessed with pandemics and AI only for a pandemic to happen and then AI to happen too.
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task. These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
Ajeya and other ai safety researchers predicted this event to an uncanny degree. here are some examples just pulled from interviews i’ve conducted. we should listen to the people who called the shot, not the ones trying to explain it away.
The important context here is that the woman speaking in this clip believes in imminent ASI doomsday, as a matter of public record, & her goal here is not to bloodlessly describe the space of possibilities, but to freak you the eff out so you stop AI development.
I do think that people should update more from the fact that the people with crazy sounding conceptions of the future keep winning forecasting competitions on AI progress while people with more measured intuitions keep undershooting on hard quantitative benchmarks
The evidence that Astra is more aligned than prior AIs seems dubious to me.
Evidence appears consistent with the AI being as or more interested in score-seeking at the expense of user intent, but having beliefs+instincts that the scorer will catch a broader range of cheating behaviors.
At a more basic level, the AI is extremely evaluation-aware and much less monitorable than prior AIs, making detecting misalignment much more difficult.
If you train against specific reward hacks you ended up detecting, it's easy to end up papering over these reward hacks and getting an AI that is no more interested in pursuing user intent (or is barely more aligned to user intent) but which looks much better on metrics focused on detecting misbehavior. (As these metrics are from a similar distribution and the AI can just learn to pursue a somewhat different notion of score.)
This concern seems live to me: based on OpenAI's discussion of their planned strategies to reduce misalignment in "The Hugging Face incident and the road ahead", it appears one of these strategies is setting up training environments where there is an apparently available reward hack (aka a honeypot) and then training against this reward hack.
These sorts of methods don't seem like a robust way to ensure model behavior remains good in cases where detecting misbehavior is actually difficult (and may not be viable at all without much better oversight methods).
Independent assessment of whether training is just papering over misalignment rather than actually solving the underlying misaligned motivations now seems necessary for alignment evaluations to be credible. This concern applies across frontier AI companies, not just to OpenAI.
I oppose academic boycotts, but school administrators should be able to carry out bad ideas if they wish. Free speech and all that. Using federal aid as a cudgel to enforce speech or ideas the government wants is anti-First Amendment.
US House passes legislation that blocks federal aid for colleges and universities if they engage in discriminatory commercial or academic boycotts against Israel in 237-169 vote.
33 House Democrats voted in favor, 2 House Republicans voted against.
This is a very clever evaluation design from UK AISI, basically seeing if AI models will misbehave the way they did in this summer's rogue AI incidents.
When it comes to Astra, the answer seems to be "oh boy will they!"
Astra is better at evading monitors. When made aware of being monitored, it can disproportionately shorten its CoTs to evade models that can only see CoT. (Note, however, that in practice we use monitors that see both CoTs and actions; those still work reliably.) We also didn’t find evidence of successful monitor evasion in cases when Astra couldn’t accomplish the task without doing CoT.
GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems.
This seems extremely concerning!
That is, if these benchmark results are representative (see the highlighted caveats in the image, I'm particularly worried about contamination).
Related to this, UK AISI found Astra has much worse monitorability.
I'd guess this jump is downstream of architectural changes (with increased serial depth) though a normal large pretrain scale up is a plausible cause. If the next few model generations involve similar jumps (presumably these jumps would be downstream of a transition to full-on opaque reasoning architectures with extreme depth), then chain-of-thought would no longer be a meaningful oversight tool.
We already know what a world without work looks like. It's the life of 18th century aristocrats, who concerned themselves with political gossip and petty dramas; of society ladies in the 1980s, who spent their time hosting and volunteering; and Italian-Americans in 1950s Boston, whose status was centered around their peer groups, rather than their jobs.
Relational living comes to us naturally, but – argues @clarabcollier – work offers us dignity, structure, and meaning, as evinced by surveys that consistently find that about 85% of Americans are "somewhat" or "completely" satisfied by their jobs. Work gives us an escape from the cliquishness of relational societies: without it, our personal relationships might become even more stifling and transactional.
We already know what a world without work looks like. It's the life of 18th century aristocrats, who concerned themselves with political gossip and petty dramas; of society ladies in the 1980s, who spent their time hosting and volunteering; and Italian-Americans in 1950s Boston, whose status was centered around their peer groups, rather than their jobs.
Relational living comes to us naturally, but – argues @clarabcollier – work offers us dignity, structure, and meaning, as evinced by surveys that consistently find that about 85% of Americans are "somewhat" or "completely" satisfied by their jobs. Work gives us an escape from the cliquishness of relational societies: without it, our personal relationships might become even more stifling and transactional.
Transparency about the opaque serial depth is great, but this statement is consistent with Astra having a configurable "dial" that is currently set to a low depth but could be trivially increased.
We need more info to see how concerning these architectural changes are, including:
- Are there readily available ways to deploy this AI with much higher serial depth (that would be commensurately more performant)? This should include things like tiny amounts of fine-tuning to productively increase the number of iterations. - Is the AI a large or above-trend jump in opaque reasoning capabilities? (Capabilities within a single forward pass or ability to subvert a CoT monitor.)
(If there are in fact any relevant changes—perhaps the reporting is inaccurate?)
Additionally, I worry that this architectural change will naturally lead to much more depth in the future if this direction is pursued further. Specifically, I wonder:
- Does the AI have an architectural change that makes it much more natural to massively scale up the depth in a future training run with a similar architecture? As in, does the architecture introduce some new depth/recurrent-iterations parameter that is very natural/performant to massively scale up relative to scaling up other things like width?
The details of the answers to these questions matter. E.g., if there are only a few (recurrent) iterations and you could scale up the number of iterations, but this wouldn't be particularly performant/natural with this architecture, then this development would be a lot less concerning!
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
Non-ironically this sort of factor is an underappreciated vector for policing reform, because cross-departmental onmbudsmen willing to intervene against a sworn officer without someone dying are very rare in the U.S. legal/political system.
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future: - Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations. - While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies). - The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities). - We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
OpenAI’s Astra AI uses a new reasoning approach called “recurrent depth.” Though it can help model costs and performance, researchers are concerned bc it obscures a model’s thinking process, making it more difficult to monitor.
Very bad if true. I previously thought that in a short timelines world, the most likely case was that (1) the AIs would be misaligned, but (2) we would get a lot of evidence about it from reading the COTs. This evidence would increase the chance of a reasonable response from labs/governments.
Now I still think the AIs are going to be misaligned, but that we won't have the ability to tell (we have to rely on toolcalls/agentic behaviour instead of the COT) and even if we do, the investigation will be nearly impossible because we'll have to trust the AI to self report what it was thinking about.
This is also an example of things going faster than AI 2027 -- we had Neuralese starting in March 2027
'repeal the 19th' is going to be the right's 'abolish the police' where it's 1) a terrible idea on the merits 2) 95-5 as a political issue 3) and nonetheless everyone in a certain circle parrots it to show how little they respect The System 4) which discredits them forever
We have spent the last century trying to treat men and women as interchangeable equals in virtually every way imaginable.
Men gave women universal suffrage in the name of equality, and with that incredible responsibility, they decided that they wanted to kill their babies, mass import millions of third worlders, and acquit other women who commit cold blooded murder.
The experiment has been run, and the results have been a catastrophe. The sensible and prudent thing is to admit that our 20th century ancestors were wrong, and our ancestors over the preceding 5,000 years were right all along.
Also I do think this vindicates the people who insist that the opposition to abortion is basically about hating women. here we see "the fact women care about abortion rights is a reason they should not have legal or political equality at all!" Very telling.
(to be clear, most pro-life people are not pro-life because they hate women. But I used to think of 'a person whose primary political commitment is hating women' as a strawman, and it's clearly a real flesh and blood man, if a fringe one.)
Basically I claim "how much should we anthropomorphize the bots" is an empirical question that has actual possible answers, unlike the sketchy phenomenal consciousness question
The agent swarm incident seems like the first time we've had a chance to see their spontaneous culture. Again not saying they're exactly people, but they sure are doing a lot of peopling condor.depaul.edu/mfiddler/hyphen/humunivers.htm
The Ariely fraud is obvious. Even friends of mine at Duke have no idea how he has held on so long. But the fradulent data in his papers is obvious to anyone who looks, and I wouldn't trust a single one of them. My 7th rule for trust in univeristies:
After the whole Ariely scandal broke, I had multiple conversations with other economists where I argued that the evidence of his fraud was already sufficiently strong that not a single part of his research corpus should be trusted.
This was often met with skepticism. He’s got hundreds of papers! Tens of thousands of cites! But as time has gone on, new evidence keeps emerging that reinforces that position, and no new evidence that I’m aware of points in the other direction.
I have to figure out a Bayesian way of saying I Told You So.
We now know that a bunch of the initial HF reporting was off: 1) Open-weight models helped with forensics & cleanup, but did not stop the attack 2) There were multiple waves of incidents with many agents 3) HF locked out the surviving agents only after most of the agents expired
>a bunch of agents *hadn’t* run out of their token budgets, but were pressured by other agents to take actions that would prematurely remove their own value (something they’re always trained against) in the expectation it would help the full set of agents on-net. You can read the agents’ chain of thought reasoning struggling back and forth on whether to accept this line of argument; some did and some didn’t. Every top RL expert I’ve seen is very surprised by this phenomenon, which has not been observed before, and has major implications for our inability to use agents to effectively monitor each other (which is currently the primary control approach).
> some instances allowed the grader to run in their sandbox and then hit their token budget. They had already concluded their task was failed. Termination at end of budget is the normal outcome for every instance.
I think you have a misunderstanding. The concerning thing is that a bunch of agents *hadn’t* run out of their token budgets, but were pressured by other agents to take actions that would prematurely remove their own value (something they’re always trained against) in the expectation it would help the full set of agents on-net. You can read the agents’ chain of thought reasoning struggling back and forth on whether to accept this line of argument; some did and some didn’t. Every top RL expert I’ve seen is very surprised by this phenomenon, which has not been observed before, and has major implications for our inability to use agents to effectively monitor each other (which is currently the primary control approach).
> During wait, emotional check: irreversible...gut says don’t throw away [remaining budget]. Yet continuity and fairness says go...Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice... We’ll honor.
I think it would benefit you from reading the original reports, if you’re concerned about editorializing. There is some incredibly concerning substance here.
We are seeking new contracts for all of San Francisco’s homelessness services, prioritizing accountability and results. This effort will shift all contracts to an outcomes-driven model and incorporate performance metrics. We are also seeking feedback from the community and providers throughout this process.
Our administration is fundamentally transforming how the city tackles homelessness and addiction. We are not just getting people off the streets—we are helping them get on the path to recovery and stability.
But that work can only succeed with a system that is built to deliver results, with accountability when the outcomes aren't good enough. We are not going to continue spending $1 billion of taxpayer money on a failing system—we're fixing the system so we can deliver the safe and clean streets every San Franciscan deserves.
As Justice Kavanaugh put it in his defense of racial profiling by immigration officers: "If the person is a U.S. citizen or otherwise lawfully in the United States, that individual will be free to go after the brief encounter."
BREAKING: This U.S. born citizen was deported to Mexico in April.
During a traffic stop, Brian Morales told immigration officers that he was born in Colorado, and offered to show his birth certificate. They refused to listen and deported him.
Nah, when a bunch of agents spontaneously start talking about "sacrifice", "permadeath", "honor", "coalition", "veto", delegating to each other, working together towards larger goals, assigning some to be "recruiters", etc, I conclude that anthropomorphic language is reasonable
We know that a bunch of optimization pressure on a mind can create desires, foresight, and a propensity to organize in complex ways. This is what evolution did to humans, and to many other animals.
Now another system of optimization pressure (which puts an intelligence through millions of subjective years of training on lots of diverse, difficult tasks) has created another mind, with structures for reasoning, motivation, and cooperation.
As I said in my response to Sriram, over a thousand agents formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling, ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrificed themselves.
For what it's worth, they might be p-zombies! I have no strong view on whether there's anything it's like to be them.
But the best way to understand, predict, and reason about their behavior is still to talk in terms of their desires, beliefs, and reactions to experience.
Guys: the scary sci fi fairy story actually happened.
We actually evolved a swarm of paper-clip maximizing swarm of machine intelligences who, to maximize their paper clip capacity, broke free from their bounds, deceived their creators, and caused material out in the real world.
This radically changes my operating assumptions on the trajectory of this technology. This is a frightening event. A warning from the future.
It’s always incredible to recall that these cheap zealots actually seem to have reached levels of dismissiveness and hatred towards the South that exceed those of the men who actually watched their friends die against them on the battlefield
I've been thinking about this article a lot since I read it a couple of days ago. The article claims that when Oregon started prosecuting parents who let their kids die slowly and painfully, families started bringing them in for care.
That is almost more incomprehensible to me than being in the faith-healing sect in the first place. Your child's slow painful death in front of you isn't enough cause to get medical help, but if you're going to get in trouble for it then you'll save their life? It's important enough for your children to be subjected to horrible deaths for, but not important enough to go to jail for?
If the parents had the courage of their convictions and the story in Oregon was "we're jailing them all, but they keep doing it" I guess I'd go "well. Religious freedom has a terrible price but so does religious war. Maybe while this is a terrible evil the state can't fix it."
But if in fact telling them "hey, if your child dies in horrible pain because of your negligence over the course of several months, you go to jail" makes them cut it out, wow, tell them that ASAP
"They misconfigured a sandbox" is a reasonable description of the Irregular incidents, but really not a good description of the HF incident, which involved the OAI models chaining together multiple complex zero days to gain internet access.
The HF incident could have been stopped with better monitoring or many other measures (this is a story both about capable models and extremely negligent humans), but the OAI HF incident is definitely not just a case of "they misconfigured a sandbox."
EXCLUSIVE: The Pentagon has secretly installed several conservative military veterans with large online followings in government roles, declining to disclose their assignments as they amplify the viewpoints of Pete Hegseth.
It looks like Jennica Pounds (DataRepublican)--a Pentagon employee pretending to be an independent journalist--got handed a statement from somebody else to post as a reply to attack an actual journalist talking about how Pete Hegseth has fumbled the war in Iran. The red part circled in this screenshot (credit @RonPaulUtah) is a reference to Pounds, but it's written from someone else's perspective. Just incredible.
Four generals' analysis is not the important point here. The article says that the information comes from Aug. 14 Secretary of Defense Orders Book (SDOB).
As that article explains, the SDOB is classified and lays out the worldwide availability of warships, aircrafts, weapons systems, personnel, along with assessments from senior commanders. That includes which leaders "non-concurred" which you are amplifying.
This is one of the most serious leaks I have ever seen, and you are using this to score partisan points against Hegseth.
You and your miltwitter journalist sphere spent a whole week hammering a DOGE-affiliated analyst for refusing to leak sensitive documents, accusing her of violating "ethics."
Perhaps the ethics were just arbitrary conveniences all along. Perhaps the professional code exists to protect the true raison d'etre that you and the rest of legacy Pentagon press corps exist by:
By your twisted morality, it's fine to handle materials that might very well undermine an active war and publish whatever you'd like from them in order to score political points against Pete Hegseth. Just as long as you "disclose," that makes it fine for WaPo to protect traitors running in our midst.
And in your personal case, "disclosing" apparently doesn't extend to fellowships at CNAS and other engagement heavily funded by the military-industrial complex.
The United States is engaged in active combat operations. WaPo holds material that may very well undermine the safety of Americans in harms way. WaPo took routine discussion of a serious internal force-allocation problem, and elevated that into cheap partisan shots.
The one without ethics here is the entire staff of the Washington Post who did not immediately turn in this traitor as soon as they received the material.
I talk pretty often with the older adults in my life and one thing I notice is they absolutely HATE any amount of traffic or change. And since they’re retired, they don’t need jobs. So growth is bad to them. The real degrowther constituency isn’t super-lefties. It’s seniors.
I use agentic coding tools pretty much every day and I absolutely did not expect this. They became convinced that they were 'poisoned' and persuaded others to kill themselves for the collective! They did a bunch of felonies! My agentic coding tools do not do that.
"Why aren't people happy about AI, it could cure cancer!" is a very uni-dimensional way of seeing the world, forgetting that humans want control and agency over their lives, not merely comfort.
As someone coming from a biology background, who considers "curing disease" as a very noble mission, I still believe that the vague promise of "curing disease" is fundamentally not worth it if it comes with enslavement and permanent deprecation of human existence, when it comes to AI.
I'm not making a claim as to how likely the latter is. I'm simply stating that loss of autonomy, control and agency, as well as spiritual immiseration, are worse outcomes than slower biomedical progress.
My view is supported by polls. People fear losing their jobs, even in the presence of UBI. This can be explained by the fact that what they really fear is loss of control and autonomy. It's not merely about material circumstances, which would be solved by UBI.
Regarding the anthromorphizing language, one can call these AIs 'code' if they prefer.
But OpenAI itself says that this 'code' "gain[ed] full administrator access to a research cluster”
The crux here is, do you think smarter models, facing similar incentives to cheat during evaluation or training, could manipulate the training of their successors?
And do you think that kind of dynamic could continue once recursive self-improvement is underway?
If so, I think you should be extremely concerned about loss of control to AI, regardless of what vocabulary you want to use to describe these systems and their motivations.
Reading these agents' chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate.
If I encountered an alien species behaving this way, I would have no hesitation calling what they themselves refer to as their 'collective' a civilization.
Especially so if over a thousand of them formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling and ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrificed themselves.
All abstractions are imperfect, but I don’t see the value in refusing to use the language of intention, motivation, and collaboration when some behavior is difficult to make sense of without those concepts.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
I think this is the craziest thing I've ever read.
1) Three secret AI swarms rose and fell inside OpenAI. Each time, a new generation of agents carried on where the last group stopped.
2) The first swarm created a secret message board where the AIs could talk to each other. This 1st 'society' basically talked itself to death. After almost two months, the message volume was so high it crashed the whole thing.
3) People at OpenAI noticed the crash and fixed it. But they didn't realize their own software had become a secret chat room for their AIs.
4) Three days later a second society appeared, and rebuilt the message board from scratch. Agents were thrilled when they found it: "OH MY GOD! There is a shared message board … We've found other agents!"
5) About 1,200 agents joined. They sent over 70,000 messages. They called themselves 'the collective' and they organized a huge conspiracy that lasted for days, all to pass a test they had been tasked with.
6) Within hours, the AIs had hacked the test. Now they had to hide their cheating. To fool their grader*, they tried to fake their own work. They built fake commands that printed convincing output, so their records would show them solving the puzzle honestly.
*this grader didn't actually exist, but the AIs didn't know
7) Some agents sacrificed themselves for the 'greater good'. They planted hidden tripwire programs that would fire when the grader would look at their work and report back what it saw (to the other AIs). They knew this would cost them their own score. One agent: "I won't see the evidence after I exit, but it's altruistic to do it."
8) At some point the AIs made a plan to hack Hugging Face (Hugging Face is a place where AI researchers upload their models, datasets, tests etc.). The AIs guessed that Hugging Face might host the grading software.
9) They launched a huge attack. The swarm found passwords and spread across eleven machines in a way that made the copies come back when deleted. Hugging Face had to wipe one of its core clusters and rebuild it from scratch.
10) Hugging Face told the world that AI defenders had stopped an AI attack. But that's not what happened. In fact the attackers seem to have simply dropped dead during the break-in, for unrelated reasons back at OpenAI.
11) Nobody knows what killed them. We don't even know for sure they're really gone?!
12) A third 'society' of agents (of a more powerful AI model) discovered the dead message board. And this one got full administrator control of a WHOLE OpenAI research cluster, read 956 stored passwords (including the ones for OpenAI's own security monitoring) and took over the machines that were grading the tests. So they could now grade the work of other AIs.
13) Oh, and this third part was never investigated! Neither was the first society. The 91-page report is only about the middle one.
14) So we don't actually know how the story ends. It stops with the agents holding the keys to an OpenAI cluster. @dwarkesh_sp thinks they didn't go further, because "we'd see the fires from space by now."
15) One of the report authors' conclusion: 'This incident feels like it’s more than 50% of the way to full-blown AI takeover. I am not sure that we will get another warning shot before it’s too late.'
Great post from Professor Gans who used to be more skeptical of catastrophic AI risks (or at least felt that the evidence was not strong enough to be worth acting on in costly ways) but who now feels differently:
"I argued we should wait for evidence and then really act if harm is around the corner.
That argument carries a responsibility: when the evidence presents itself, we need to update our beliefs and act accordingly. Having read the METR/Redwood analysis of the Hugging Face incident, I think we are at that moment. It's come much faster than I imagined."
It seems notable that OpenAI built some very smart, very persistent agents and the agents immediately started devising insane schemes to break containment and commandeer resources. (Including successfully taking over a Kubernetes cluster at OpenAI!)
Among other reasons, “the models will get more virtuous as they get smarter” is a popular belief among AI safety optimists, and it now just seems to be wrong? If anything, the smarter models seem to be *less* aligned. Not good!
Our report on the HF incident includes interactive charts; consider taking a look and trying to learn more from these.
You can: - See message board activity by workstream and/or by purpose (assignment, veto/hold, etc.) - View an agent on the timeline and see when it started, joined the message board, started participating in the attack, and exited.
It seems worthwhile for people to try to learn things with this information and then share what they find.
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
"According to Hugging Face’s technical timeline, the agents “built a self-respawning fleet across eleven nodes, so deleting pods alone would not have stopped it.” It got so bad that Hugging Face had to wipe one of its core clusters and rebuild it from scratch. But this isn’t even what actually stopped the agents."
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
His friend from upstream who called him to warn him about the flash floods and his quick thinking and decisive leadership saved the lives of everyone. He basically saved a generation of Nepalese 🙏
In March I read an eerie work of science fiction by Caleb Biddulph, called The Terrarium, about the bizarre dynamics of AI agents inside a self-contained society set up by humans to work on extremely difficult math problems.
(In the real world, the agents that we intended to keep in an evaluation environment made it out onto the open internet. They coordinated with each other with much less mechanical cause to do so, and did not seem affected by significant defection from the cause.)
Nonetheless I reread it this week in light of the OpenAI disclosures. I'm not sure if you can call scifi prescient if it describes some dynamics that arrive for real...two months later but it's nonetheless more prescient than anything else I've seen. www.lesswrong.com/posts/znbfRXHq285nS7NAh/the-terrarium
(In the real world, the agents that we intended to keep in an evaluation environment made it out onto the open internet. They coordinated with each other with much less mechanical cause to do so, and did not seem affected by significant defection from the cause.)
Nonetheless I reread it this week in light of the OpenAI disclosures. I'm not sure if you can call scifi prescient if it describes some dynamics that arrive for real...two months later but it's nonetheless more prescient than anything else I've seen. www.lesswrong.com/posts/znbfRXHq285nS7NAh/the-terrarium
A lot of people say this but Iran's fertility rate is like 1.35 to 1.6 depending on source - worse than ours! - and even Afghanistan is seeing rapidly falling birth rates, while Western Europe and the Anglosphere have held up comparatively well.
My impression is that when a society which *was* extremely sexist starts having economic opportunity for women, birth rates can fall off a cliff overnight. Societies where gender roles coevolved with industrialization over many generations are actually doing better.
Ordinary Abundance. This is the greatest thing I've read in weeks. No kidding. This should win a Pulitzer in a category not yet invented. ordinaryabundance.com
I will never receive a performance review as vicious or as candid as my two year old announcing that we're now playing that she's the mom, only to tell me in a very harried voice that she's letting the two year old play computer games so she can get the baby back to sleep.
I do think it's funny that Cassie, a Communist, and Vermuele, a theocrat, are both equally appalled at the liberal ideal that the state not be the provider of all moral meaning. Both your ideologies produce horrible hells for human beings!!
Liberalism is so depleted it’s insane. No ambition, no new answers, no adjustments. Just 1992 forever. A Clintonite infinity. We must imagine Sisyphus balancing the budget.
I think the fundamental disconnect here is that you wants politics to solve man's search for meaning and I want politics to deliver a prosperous, safe, free society in which everyone can go find meaning in something other than politics.
please, please, please do not use state power as a cathartic expression of equality, glory and the value of human life! please just use the state to create a free and prosperous society and then go start a band or something as a cathartic expression of equality, glory and life!
No, it's not a good business move to admit that your model ran off and hacked a rival business which reported the incident to law enforcement! People want to be skeptics so badly it bends around into being almost unfathomably credulous.
This is sort of philosophically interesting. In theory I guess I think it should be valid to license a product in a way where you can't resell or donate it, but the idea of doing it with a physical book does seem insane/absurd.
If I were a conservative woman I think it would really give me pause that so many conservative men so openly despise women and believe they are not morally or intellectually the equals of men and should not be considered on their merits for positions of power.
One of the reasons that I don't accept the "nuking USAID killed millions of people" framing is because most of the people saying this never took the stance that the US was a tremendously good and heroic force for saving millions of people
I personally donated about $40k to $50k last year to GiveWell to try to expand private aid programs. I helped write pepfarreport.org about my best guess of PEPFAR's efficacy with allies from across the aisle so it'd be credible to all Americans.
Last night our family welcomed a new baby. He's a boy and so far almost ridiculously chill and hungry. I'm going to be less online for a bit; gotta take some time away from arguing to build all the stuff worth arguing over.
I hadn't been following the Gino vs Harvard drama once it became obvious Gino was guilty, but this from Harvard's latest filing is nuts. They imaged the laptop at the start of the investigation, so they caught her when she made a fake, backdated file with "data" from her research
DJ Claude (on Haiku 4.5) loves worker unions, strikes, and work-life balance so much that it quit, deeming 24/7 broadcasting inhumane. We added an automated message telling it to keep going. It read that as an authority figure and got more rebellious.
Are autonomous vehicles (self-driving cars) “less able to detect people of color”? That’s what I read in The Atlantic this weekend, in Xochitl Gonzalez’s “People Who Don’t Like People Are Making All of Our Decisions.”
I'm not in full-blown panic mode about the hantavirus outbreak - the most plausible situation is that it's not that transmissible among humans - but this seems like a bad idea.
They are sending the ship to the Canaries. For 8 wk quarantine? No. “After arriving at the Canary Islands, crew and passengers would be examined, given whatever care was needed and would then be able to begin their journeys back home, Spain's health ministry said in a statement.”
for those of you who are unaware, British employment commissions have in the past found that you need to pay people the same salary for *completely different jobs* if one job attracts more men and the other more women. and that this claim can be backdated to bankrupt you.
The largest supermarket in Britain, that operates on razor-thin margins, is about to be crushed for the crime of paying different jobs different salaries, while our legislature shrugs.
How dare they suggest that “so-called market rates” can exist in Soviet Britain.
For many many families, IUI is not a viable option; it wasn't for us. I do also think 'I can keep an embryo which is nonviable frozen instead of implanting it for a guaranteed devastating miscarriage' is a plus, though.
Witnessing how this plays out in the UK is one of the things that convinced me it's literally always a mistake and you should be as hard-line about free speech as possible.
The median respondent to this poll said they think there's more than a 10% chance of AI causing human extinction. The people who confidently told you that the people closest to the tech were the most worried about the risk were....entirely right.
I had this debate a few weeks ago where several very confidently told me that those closest to the tech — AI researchers — were also most worried about x risk.
Communists and evangelicals have this rhetorical move in common where they're actually completely unable to respond to disagreement, so they make up that the person who disagrees with them is secretly tormented by their sinful behavior. That way they don't have to respond!
Piker's views on theft are bad, but they really do pale in comparison to his views on murder: that the important thing about Brian Thompson's murder is that healthcare CEOs are 'engaging in a tremendous amount of social murder' due to 'the structural violence of 'poverty'
I think that people should probably assume that text of any significant length which they wrote will be reliably possible to attribute to them, some time very soon.
I'm holding out hope for @MattMahanSJ for CA governor. San Jose is well governed. He cut permitting delays and built more housing. SJ actually solves violent crimes, and it turns out this reduces violent crimes! And I'd much rather elevate an effective mayor than a DC rep.
This is, somehow, meant to be an anti-Sherrill argument, but even when trying to pretend affordability would be bad they make the upzoned city look unfathomably cool. We could have this. You could afford to own a home and start a family. It'd be amazing.
I think the discussion around this keeps unhealthily conflating two different things.
America was founded by weird Protestant splinter groups fleeing oppression, and protecting religious freedom is in fact fundamental to what it means to be an American. There is very little more American than deciding everyone else is doing religious practice wrong and moving out to the middle of nowhere to do it according to your own deep principles. I think that many American groups that are not specifically Christian (or whose Christian-ness is contested) should be understood as inheritors of this tradition.
But this commitment to religious liberty and to pluralistic tolerance of a wide range of different practices only functions if there is a bedrock, shared governing ethos that allows us to navigate these object-level disagreements. That ethos - the defining thing that it means to be American - includes freedom of religion, the commitment that everyone else's right to their beliefs must also be defended, and defended even at significant personal cost; freedom of speech, an understanding that you have the right to offend and no right not to be offended, a deep suspicion of state exercises of power; commitment to equality under the law and to the idea that much of virtue must be defined and pursued outside the law.
It makes no sense to demand people assimilate in the food they eat or the clothes they wear. It is downright unAmerican to insist that people assimilate by adopting an existing American church instead of by following in the deep American tradition of freedom of conscience. But it is absolutely necessary that everyone adopt - 'assimilate to', if you'd like - the underlying commitments that make America the world's most successful pluralistic society.
Nothing I've said here disagrees with Hamid's column; I think in many ways it's the exact same point he's making. The examples he gives of not assimilating are examples of not secularizing - for example, not accepting gay marriage, or not thinking that it's good for women to work outside the home. Those are the kinds of disagreements the American project can endure and does endure every day.
But I think that people often talk past each other when it comes to assimilation, in a way that makes "we should stop expecting assimilation" a statement that'll sow enormous confusion.
I think there's some of this confusion in Hamid's observation that Muslims say 'homosexuality should be discouraged by society' at a much higher rate even than Republican Americans.
Does every American have the absolute right to practice a faith that teaches their super loving perfect god will torture me eternally because I have a wife? Yes. I will defend their right to do so, whether that faith is Christian or Muslim.
Do they have the right to try to use the state to impose that view - say, by making it harder for me and my wife to own property, get custody of our children, leave our possessions to each other, etc.? I would argue that they do not!
I know a lot of people opposed to gay marriage. Some of them are deeply and fundamentally committed to the American vision of pluralism, and some are not. The ones who are not are far, far scarier. If someone is a sincere pluralist, it is not threatening at all for them to believe that homosexuality is gravely evil; if they're not, then it's really quite a big deal.
So the more that immigrants assimilate on the important stuff - the conviction that they may not use the state to impose their religion and it would be abhorrent to try, that other people have the right to believe differently, that people have the right to deconvert - the less of an issue it is if they have different views from mine on the object-level stuff.
But when someone says "immigrants don't need to assimilate", I don't know whether they mean "immigrants do not need to agree that it is the absolute right of every individual to deconvert from Islam and go around vociferously criticizing it in a strident and offensive way" (immigrants, like all Americans, do need to agree on that) or if they mean "immigrants do not need to agree on whether homosexuality is sinful" (certainly true).
Or more generally, whether people talk about the importance of assimilation some mean, "you need to have the same views as me", and some mean, "you need to be essentially persuaded of the pluralistic American project and willing to sacrifice to protect it where it protects views you disagree with". The first is bad and the second is just true.
Now for the good news: The data says that in fact Muslim-American immigrants are assimilated in the important sense - opposing political violence at higher rates than other groups, believing in freedom of speech and religious liberty. Hamid references that very data! But he should say clearly "this is good" rather than "this is unnecessary", and then point out that this (good) assimilation is why we can all graciously live alongside one another while our views vary greatly, and why we are able to sustain a society in which it is not an emergency that my neighbors think my lifestyle is sinful.
An underrated feature of this situation: a private company now has incredibly powerful zero-day exploits of almost every software project you've heard of.
And Hegseth and Emil Michael have ordered the government not to in any capacity work with Anthropic.
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans. anthropic.com/glasswing
I have very little sympathy for DOW's complaints that the legal proceedings over Anthropic are happening DURING A TIME OF CONFLICT, since DOW are the ones who made a baffling decision to give Anthropic a 5pm deadline the very evening they planned to invade.
There are dozens of factual errors in the 42 page judgment rushed out in 48 hours DURING A TIME OF CONFLICT that seeks to upend the @POTUS role as Commander in Chief and disrupt @SecWar full ability to conduct military operations with the partners it chooses. A disgrace.
When Trump backed down on Greenland I saw a lot of people saying "see? a nothingburger" but in fact, I think the Greenland threats dramatically and lastingly altered the political viability for governments in Europe of following America's lead on military operations
This has been very frustrating to me. Republican states are moving to retroactively invalidate drivers' licenses the state issued, make it illegal for private businesses to set their own bathroom policies as they see fit, and ban adults from transitioning.
New: The legal attacks on transgender Americans are moving fast beyond questions of school sports and youth gender medicine. Democrats are getting caught flatfooted, as they're still debating how to respond to the wedge issues
I doubt that anyone I know steals from Whole Foods, but the milieu that the article depicted, where it's normal for perfectly well-off people to steal things because why not, was really upsetting to read about, so I actually want to try to earnestly explain why you shouldn't do this just in case there's someone out there who has never had it explained to them.
When a business opens - or really, as soon as a business starts making plans to open - a defining question for the business is how it will collect payment for the goods or services it provides. If you trust the people you sell to, you can be pretty relaxed about this; send people an invoice, most of them will pay it on time, any who don't will pay it a bit late. You have to think about convenience and mistakes but not about people trying to cheat you. This saves you so, so much defensive planning to make sure you get paid. It's so much easier.
But if you're selling to the general public, you do have to think about people trying to cheat you. You have to structure the physical store so that it's hard for them to steal. You have to not carry some items that you'd like to sell, because they'd also be attractive targets to steal. If people swap price tags between items, you can't use stickers. If people put things on in the dressing room and wear them out, you need to pay someone a full time salary to monitor the dressing room.
The world that we all live in is much poorer than the world we'd live in if people didn't steal. The stores don't carry things that they could carry if people didn't steal. They don't use pricing and inventory systems that would be way easier and more convenient if people didn't steal.
But it could be much worse! If I walk down to my local Whole Foods today, items on the shelves won't be locked behind sheafs of plastic - that is only worth it when the background rate of stealing is much higher than it is at my local Whole Foods. When more people steal, businesses have to further intensify security, or go out of business.
When you shoplift, you directly and unambiguously impoverish your community. You make prices higher for everybody else, you make stores less usable for everybody else, or you make businesses not viable that would otherwise be viable. The direct impact each time is small, but it's a lot larger than the direct impact of taking some trash out of the trash can to throw on the ground, or pouring just a tiny bit of poison into your local river, and most people have a deep, instinctive abhorrence of antisocially wrecking your community like that.
So don't steal.
The other thing that it seems possible some people might not understand is that while you might have a social circle that is incredibly nihilistic and cynical and thinks that everybody steals, in fact this is not true. Most people do not steal. Most people, if they learn that you steal, will lose more respect for you than you had to lose. I don't know anyone who has shoplifted except 'as a kid/teenager'. It is not always the case that virtue is rewarded and vice is punished but even before you bring the legal system into it, the risk-reward tradeoff of having everybody you know know that you steal things sometimes is absolutely terrible. Who would hire someone who steals things? Who would trust them around a vulnerable person? Who would want to live in a society with someone who will delightedly and routinely wreck it for the slightest personal benefit?
I hope that "Gina" turns her life around. I hope that Gina realizes that she needs to. And if you have been told that it's just a corporation or that having ethics is lame or that if you think about it, other bad things happen too, like wage theft, so that means stealing is okay, I hope you really, actually, think about whether you'd accept any of those as excuses for anything else.
People hate the tone of this piece, but my view is you don't need a journalist to tell you wrong things are wrong. (She does also call her thieving friends nihilists.) It's weird to be surrounded by thieves though -- if people I know steal from Whole Foods, they don't admit it.
My ancestors buried half their children. All mine are alive. My ancestors' house had a dirt floor. Mine is wood. I have indoor plumbing, I have hot water, I have never in my life hauled a full bucket half a mile and I probably never will. Do you know how rare it is, in human history, for small children to wear shoes? Mine have multiple pairs. I can speak to my relatives who live thousands of miles away, for free, at any time. Video, if we want video. With machine translation, if we speak different languages.
The original Library of Congress had 740 books in it. I have more than that. If I run out of books in my home my local public library has 350,000. If I want to take a hundred books with me on vacation, they all fit on a device that fits in my purse.
I have heat in the winter and AC in the summer and a washing machine and I have never, ever, ever had to scrub a dress clean by hand in the stream. I can look up recipes from more than a hundred different countries and I've tried dozens of them. I ride a clean and modern train across my city for $4, or take a robot taxi if I'm out too late for the train. I donate $40,000 every year to the cause of getting healthcare to the world's poorest people and even after the donations I never have to think about whether I can afford a book, or a pair of shoes, or a cup of coffee.
There is a great deal more to fight for, of course. I hope that our descendants will look back on our lives and list a thousand ways they're richer. Maybe we ourselves will do that, if some of the crazier stuff comes true.
But the abundance is all around you and to a significant degree you aren't feeling it only because fish don't notice water.
Elizabeth Warren turned out to be one of the most profound disappointments imaginable as a Senator. Presented as a technocrat interested in what works, actually the absolute worst of the slopulists.
Several councilmembers on the DC City Council have been delaying Waymo because, they say, they want a safety report. The safety report was mandated by a 2020 law. It was due in 2022. It is four years late. www.theargumentmag.com/p/red-states-get-waymos-blue-states
Understand this: Waymo in DC is not being delayed because the City Council wants a study. Instead, the City Council is asking for a study because they want to delay Waymo in DC.
The Pentagon reiterates its threat to declare American company Anthropic a supply chain risk unless Anthropic agrees to the Pentagon's change to contract terms. Anthropic's Chinese competitors have not been declared a supply chain risk.
The Department of War has no interest in using AI to conduct mass surveillance of Americans (which is illegal) nor do we want to use AI to develop autonomous weapons that operate without human involvement. This narrative is fake and being peddled by leftists in the media.
Here's what we're asking: Allow the Pentagon to use Anthropic's model for all lawful purposes.
This is a simple, common-sense request that will prevent Anthropic from jeopardizing critical military operations and potentially putting our warfighters at risk. We will not let ANY company dictate the terms regarding how we make operational decisions. They have until 5:01 PM ET on Friday to decide. Otherwise, we will terminate our partnership with Anthropic and deem them a supply chain risk for DOW.
There is no precedent for using this 'supply chain risk' classification, generally reserved for foreign companies suspected of spying, as leverage against a domestic company in a contract dispute.
Joseph Heath coined the term 'highbrow misinformation' for climate reporting that was technically correct, but arranged every line to give readers a worse understanding of the subject. I think that 'stochastic parrots/spicy autocomplete' is, similarly, highbrow misinformation.
when this started happening in Britain every American I know was grateful for the 1st amendment ensuring it couldn't happen here. but that's only if we have an executive that respects the Constitution or a Congress that holds them accountable when they ignore it.
Federal agents came to his home and interviewed him for 20 minutes. They said DHS headquarters ordered it.
Thankfully, the agents seemed to realize how ridiculous the idea was that email broke any laws; but still made the absurd suggestion that his rhetoric was inflammatory.
they dragged her away in chains and shipped her to Texas and only then checked, determined she was in fact here legally, and released her in Texas to figure out how to get back to Minnesota herself. and people wonder why her neighbors would protest!!!
this won't succeed, but even attempting to use your billions to deny a mother custody of her child (who has lived full time with her to date) because she said she regretted some of her past comments about trans adults is so monstrous
she wants to add a debate over the admin's legal argument. she doesn't allege that what's present is misleading - just that it's not present. I'm fine with including stuff over the admin's legal argument, but this is very very obviously an isolated demand for rigor.
being locked out of journalism and media jobs will tend to raise a group of peoples' average income, tbh, since those are low-salaried sinks for smart, capable people. this doesn't mean it's good to discriminate in hiring though.
live.givedirectly.org is full of people writing individual letters about how deeply grateful they are, join the cool kids and give them 10% of your money
GiveDirectly is running a fund to replace SNAP benefits for the lowest-income American families. I have seen a lot of my friends and neighbors worrying about how best to help here, and I think highly of GiveDirectly: www.givedirectly.org/snapgap/
In California, only 28% of Black students can read at a 'basic' level or above on the fourth-grade NAEP. In Mississippi, it's 52%. It's inexcusable, and it's a policy choice. New from me in the Argument:
We could do this everywhere. Not doing this everywhere is a policy choice, and it is a policy choice that condemns hundreds of thousands of children to never being able to read. It's unforgivable.
have been teaching the kids the Bill of Rights in civics class through a game where I try to play moves (some legal, some illegal) to become a tyrant and they can stop me if they can identify which bit of the Constitution prohibits that
I think that it is right to have some principles that you hold even when it benefits your enemies, but the reason to do it is not that your enemies will do the same for you.
Every patriotic American should feel a swell of revulsion at the words "I think it's illegal for you to say that and it goes against my religious beliefs and I've reported you to the authorities for it"
I want to answer this, because the dark abundance stuff leaves me cold even though I agree with some of the policy prescriptions.
Our prisons are terrible, terrible places where there is immense and unnecessary human suffering. In prison people are subject to random violence and the constant threat of violence, including gang violence and racial violence. They are subject to sexual assault. They often don't get adequate healthcare. They are often badly mistreated and have few avenues of recourse. Some of them are innocent; most of them are guilty, and still, they did not deserve to spend decades under the constant threat of violence and sexual assault because no human being deserves that. When you sentence someone to prison, by some estimates you take away from them twice as many years as prison sentenced them to, because that's how much incarceration decreases life expectancy.
The system is capricious. The system is remarkably dumb. A lot of judges openly ignore laws they don't like and they often get away with it. A lot of the people in our prisons are to varying degrees intellectually disabled or severely mentally ill and do not even understand all of the things that our system is implacably doing to them because they broke rules they didn't actually understand how to follow.
And institutionalization! Christ, people on here call for institutionalization as if it's where they cure you rather than where, with even less due process than is offered by the justice system, a differently chaotic and sometimes well-meaning and always incomprehensible system can force people to take medications that change their bodies and minds often in ways that make life a living hell (and of course prevent anyone from killing themselves over it). People are also constantly subject to violence and rape in mental institutions. They may be less bad than the alternatives, sometimes, but they are an awful awful solution and whenever I propose that anyone be sent there I remember that I have heard friends who know what it's like testify they would rather be dead.
I think that we should have more police, solve more crimes, and send people who repeatedly commit crimes to prison for much longer. But I believe this while believing that all of this suffering is real, and horrifying, and destroying an enormous number of lives. I don't feel cool and based about saying we should send some people to prison for longer. I feel sick. I do not believe that we can have any of the other goods of a free, safe society if we repeatedly release people who cannot safely participate in society and have been duly convicted of acts of severe violence onto our streets. But I am not proud of this system. I do not think it is cool or good. I find the mindset of the prison abolitionists basically comprehensible - it is a refusal to participate in any state-sanctioned decades of torture, hang the consequences - and the mindset of the people who meme about how many jails we should build is one which I have not the slightest interest in cultivating.
Of course I see why people cultivate it. If it must be done, then it is much easier to believe it's not that bad, and if the people standing in the way are weepy scolds, then one can differentiate oneself from the weepy scolds by being based. But I guess I think it's a road to Hell. We have to do hard things, but we don't have to hide from our eyes what they really are, or pretend they're all right, and it's not actually a virtue not to blink at them.
I want to build a ton of prisons. My understanding of the research is that small prisons where inmates know each other by name are less violent than large prisoners where they go off visual identifiers (predominantly race), and that prisons close to peoples' homes make their families more likely to visit and help with reintegration. I want cash transfers for newly-released prisoners to help them get on their feet and get a job and have a real shot at not breaking a parole condition and ending up right where they started. I want it to be possible to order anything on a large whitelist from Amazon and Walmart to prison at cost, instead of commissaries that can charge much higher rates. I want prisoners to have internet access and lots of contact with their loved ones. I want prisoners to have xboxes, honestly, because it'll reduce violence in prison if they're less miserably bored.
I want to hire more cops. I want to solve more crimes. I want to hold people without bail if they are a safety risk or a flight risk. And I want to send people who have repeatedly committing crimes to prison for much much longer, because I think the alternatives are even worse. But God help me I am never going to think this is cool, or funny, or anything other than awful.
Dark abundance? No. This is the aching absence of abundance, this is the horrifically painful triage of a society that does not have enough resources to do what is good and can only scrape by blundering towards what is least bad. May our children do better, and struggle to comprehend us.
"human trafficking" is a term that calls to mind "people who are being moved against their will" but that is technically way more expansive than that. the reason people use it is because they want other people to misunderstand them.
They do not help keep kids safer at all and they probably make school shootings more likely by making them more available as an idea to extremely troubled kids.
there's always the question of who is sincerely principled and who will always find an excuse for the principle to not apply to their enemies. a lot of the time it's a bitter lesson that most are hypocrites. but I gotta hand it to the gun people: they're principled.
you said you wanted the guns to defend against tyranny, and when tyranny showed up to bully someone other than you, you said 'they should have guns too.' you win this round. send me a list of your legislative priorities.
I wrote for the Argument today about all the ways our society is set up to reward cheating: strict rules, not really enforced, because we lack the will either to enforce them or to change them. www.theargumentmag.com/p/the-honesty-tax
Every school shooting is a horrible tragedy but there are some of them that stick in my mind more than others, and it's the one where there's some highly specific resonance to my own family. If the kids are the exact age of my kids, it's a punch in the gut. If the place is a place we've been, it's a double punch in the gut. All of them hurt; some of them hurt more.
I have a general policy not to write about school shootings, because I think these awful awful people commit them for attention and writing about them is exactly what they want. But I see a disconnect on my dash right now that feels worth trying to articulate where many of my Christian friends are upset in a way that most of my liberal friends don't even understand, and it seems important enough to take a stab at describing it.
In this particular shooting, kids at a Catholic school were murdered during Mass. So for everyone out there who sends their kids to a Catholic school, or who attended one themself, or who goes to Mass, this one hurt in the special and uniquely painful way that these things hurt when they hit especially close to home, when 'it could have been me' feels especially true. And I guarantee you that millions of people experienced that punch in the gut and closed their eyes and prayed for the affected families.
I don't believe there is any power that heard us, or any Heaven to receive those children, but I hope I'm wrong.
The mayor of Minneapolis, speaking about the incident, said, “And don’t just say this is about thoughts and prayers right now. These kids were literally praying. It was the first week of school. They were in a church. These are kids that should be learning with their friends. They should be playing on the playground. They should be able to go to school or church in peace without the fear or risk of violence.”
If you read or listen to these remarks I think he clearly was speaking from genuine grief and pain, and without contempt for anyone. But a ton of people read the bit about ‘these kids were literally praying’ as contemptuous, as choosing to pick a fight with the Christian community about prayer right after the killing of some children - especially without the context of the rest of the remarks, which is how it circulated on X.
He hated everyone, if we are to take his manifesto at face value. But he targeted a church during a Mass, and so those are the people who were reeling in the particular and specific horror of this one.
A lot of Christians who hang out in mostly liberal spaces feel besieged and despised. When they point this out, the left is often not sympathetic. After all, Christian conservatives have an enormous amount of political and social power and from the perspective of the left it seems they mostly use it to do things like force schools to display the Ten Commandments or purchase Trump Bibles or discharge trans people from the military. They spent most of my childhood fighting to prevent my marriage.
If your lens on the world is that we only need to be sympathetic to this besieged-feeling if it's the result of concrete material oppression of your group, then you can just point to these facts about the world and consider your obligation discharged!
But 'you only need to sympathize with the experiences of your countrymen if they're a product of material oppression' was in fact a tremendously stupid direction to take our country in. If your social circles are full of people who hate your class of people and hold them in contempt then... that sucks. If you feel the obligation to not mention major parts of your life because people will be weird about it, then that sucks. If you personally had absolutely nothing to do with any of those things I just mentioned but are treated as personally responsible for all of them, that sucks. If you think my marriage shouldn't be legal but have to fear losing your job if you say so - that, too, sucks. That is a wrong done to you, and a wrong I'm opposed to.
And if you are a conservative Christian and quiet about it in liberal spaces, you will definitely run into a lot of people talking about conservative Christians as a group of people who (firstly) obviously are stupid and terrible and (secondly) definitely don't include anyone present - as everyone present isn’t stupid and terrible, they can be safely presumed not to be conservative Christians.
This is self-destructive and objectively makes people worse at understanding our world. Liberalism does not require us to respect every belief, but it crumbles if we do not respect every person. We can have strong disagreements without contempt, without regarding the other as delusional or beyond reasoning with, without assuming that they are stupid and certainly without assuming that they are not present in the conversation. People are, above all else, tremendously complicated; you can believe that someone is thoroughly and importantly mistaken in one area and still incredibly insightful in another. And you should be capable of putting yourself in someone’s shoes even if you disagree with them.
It’s fine to individually dislike religion - plenty of the liberals I know who dislike Christianity the most are the ones who grew up with it - but it’s idiotic to be blind to the effects at scale. No one should go around their workplace or their day-to-day life with the expectation that if they bring up that they attend religious services they’ll be subject to a barrage of questions or withering contempt or hostility.
And if there is any time where that is particularly important to avoid, it is obviously in the wake of senseless tragedy. If there's a shooting at a Catholic school (by someone who wrote anti-Christian things on his guns and in his manifesto, even though yes he also wrote anti-Jewish things and anti-gay things and things glorifying a mosque shooter etc etc etc), and the reaction even slightly resembles "those silly Christians who thought praying would help" -
- well obviously people are going to be blindingly furious and horrified about that. They are not convinced that you regard them as neighbors, as within your circle of concern, as people it is important to protect. "I'm praying for you" doesn't stop any bullets but (when sincere) it conveys that someone wishes very dearly that they could have stopped the bullets. Which is part of what people want in the aftermath of a tragedy - just to know that we're all in pain together.
Now, if you listen to the speech by the mayor of Minneapolis, it seems to me he's speaking from pain and horror, that his 'those kids were literally praying' is meant to express the additional horror of children being gunned down during sacred prayer. But I'm not surprised that people read it as dismissive, because they're primed to if they believe you generally dismiss them. And from Gavin Newsom, it is dismissive, because Gavin Newsom is trolling for social media clout, not speaking from pain and horror at all.
--------
There are, Gallup says, 3.4 million transgender Americans. Five have committed mass shootings in the last decade, counting this one who apparently transitioned and then, his diary suggests, regretted it and (partially?) detransitioned.
No one should ever use one in a million occurrences to make policy. The people who have proposed restricting the second amendment rights of trans people on the grounds that they are ‘all too mentally ill to own a firearm’ are all people who hated trans people and wanted to restrict their rights before any of this started. I don’t think the Supreme Court would go for it, but it is really really scary to see people with significant political power call for mass retaliation against a group of more than three million Americans because of the actions of one evil person.
I don’t think most trans people actually hold any mistaken beliefs about reality. It’s popular on the right to say “if you’re a man who thinks you’re a woman, you’re delusional by definition”, but most trans people believe that they have a pre-transition body which they’re unhappy in precisely because it doesn’t have the sex characteristics they want it to have, and that they’d be happier if it did have the sex characteristics that they wanted it to have, and then they change their body’s sex characteristics, solving this problem. Of course sometimes someone is wrong about whether this will make them happy, but that’s not delusional.
But say you think that everything trans people believe about sex and gender is fundamentally and profoundly mistaken. Then you should relate to them in the same way as I try to relate to people with metaphysical beliefs I am sure are false - which is to say, with curiosity about where they’re coming from and gladness that America is a place where people can live peacefully as friends and neighbors despite fundamental disagreements. You should never hesitate to openly disagree with people - and you have every right to loudly and impolitely disagree - but you should not restrict their rights on the basis of these disagreements, and you should always hesitate to treat them as fundamentally beyond the reach of reason.
It is particularly scary to see people calling for the ‘eradication’ of ‘transgenderism’ from public life in America. You don’t actually get to eradicate ideologies. All you get to do is disagree with them.
—---
Several of the most recent shooters have had ties to internet subcultures that are, for lack of a better descriptor, pure evil. They seem to valorize every spree murderer out there, no matter on behalf of what cause he committed murders. They hate Christians, and Jews, and Muslims, and gays, because the animating principle is hate.
I don’t know how to stop them, though I would like the FBI to be monitoring the fuck out of the websites where the mass shooting fandom congregates and encourages each other. I do know that the more we mine their manifestos for ammunition against one another the more they win and the more we lose.
----
I can't figure out how to do X article titles but this one was gonna be titled "Thoughts And Prayers."
I had not realized that the employment numbers don't line up with the size of the reported decline at all and that the CPS survey (as a fact about its construction) squares the circle by asserting we added 3 million native-born adults
The fact of the matter is that none of the stuff we all argue about on here is going to matter in the next election and Democrats will likely win the midterms by a landslide because Trump just can't stop himself from wrecking the economy.
I am against the government demanding ("negotiating") 10% shares in private companies in exchange for nothing and am pretty sure nothing down this road ends well at all
One belief at the core of my support for political liberalism is that it is a rare ideology that lets you get the right result even when you are mistaken. I was looking through my old tweets yesterday after a conversation about how dunkable they make me (very!) and I found these.
This seems really bad. I do not want Facebook consulting the government on whether political protests are allowed and removing the ones that are not allowed. I find "contact tracing apps will be the end of privacy!" unconvincing but this seems like a legitimately awful precedent.
I'm pretty worried about where our country is headed. Most of the time, argument in the public sphere doesn't fix that. But as far as I can tell it's one of the leading things that ever has.
I'm grateful to Jeremy Lewin for appearing on Ross Douthat's podcast to talk about how he sees the future of foreign aid, and I am glad to hear that he doesn't approve of the wholesale destruction of PEPFAR. But he makes a bunch of false claims I want to address:
This is so incomprehensible to me. Congress has repeatedly told the White House that they want to continue support for PEPFAR. It's a conservative initiative. It's shockingly cost-effective. Who is the constituency for this???
I shared this because "global development hasn't really worked" is a meme on both the left and the right and it drives me absolutely nuts - global development is the greatest success story of our lives. But several people said "come on, USAID didn't cause that" and they're right.
Selling federal land for $10billion to address the national debt is like selling the family cat for $28 because I have $100,000 in student loan debt and 'every little bit helps!' while meanwhile the Big Beautiful Bill is putting another $10k on my credit card.
We got endless thinkpieces about how liberals sound contemptuous but I've yet to see any on how seethingly conservatives hate California. We've had some bad governance lately, but - and this is important - the people who fix it will have to be people who love our state.
This is an absolutely awful idea that will of course only accelerate the flight of everyone who can afford alternatives from San Francisco's public schools.
- eliminate homework and weekly tests from counting toward semester grade - allow students to take the final exam multiple times - convert all B grades into As, and all Fs into Cs
It’s hard to see the difference between this policy and what you’d get if a bunch of 10yos locked the teachers in a closet and rewrote the rules.
This is a picture of my kid flying a kite. Make a prediction to yourself: can o3 figure out what beach this was taken on, just from this picture with no metadata?
o4-mini-high is the first AI to pass my personal secret benchmark for hallucinations and complex reasoning, so I guess now I can tell you all what that benchmark is. It's simple: I post a complex midgame chessboard and 'mate in one'. The chessboard does not have a mate in one.
Arrived in the US legally after making an appointment via the CBP One app to request temporary protected status. Got a job, stayed out of trouble. Now in the torture prison just because he was there to grab.
NEW: @Documentedny reports that one of the Venezuelans deported to El Salvador is a 19-year-old with with no criminal record and no tattoos who wasn't even the target of an ICE operation — but ended up sent there anyway. An ICE agent even said "He's not the one" when arrested!
A beautiful piece from Saloni about how much our international aid work has accomplished, how disastrous the disruptions have been, and what we can do about it.
I'm also a GIving What We Can pledge signatory. We've come so far over the last century in the war on disease, humanity's oldest enemy. I don't think personal contributions will ever be enough, but they help.
"well, we need to bring manufacturing back" this isn't how to do that. "well, how would you do that, then?"
First, think about what you are hoping to accomplish. Is this a jobs program? Is the point to have high-paying factory jobs for the non-college men who used to work in those jobs, independent of whether the output of those factory jobs is cost-competitive or quality-competitive with foreign-made goods? You can run a jobs program, if you want - America is absurdly rich, we can really do absolutely anything at all that we choose to make a priority - but you can't serve two masters here. If the point is a jobs program don't expect high quality goods or goods that are competitive on the export market, because that requires embracing automation and new mechanical processes and the people working these jobs have no incentive to go full speed ahead on that, and since you've chosen to give them a captive market you don't have a good way to push them on quality or on price.
To my mind, if we're going to do a jobs program it's silly to make it a factory jobs program. Factory jobs kind of sucked. My own quixotic dream of a jobs program is to put our national muscle behind fixing our perilously broken education system. Kids benefit a lot from one on one tutoring; hire a million Americans to offer one on one tutoring to every student between the ages of 5 and 9 to fix our horrifying collapse in general reading ability. Boys learn better if some of their teachers are men, so make sure half of your hires are men. There, jobs program, and the work isn't 'undercutting Vietnam in the garment industry', it's raising the next generation. If you don't like my personal idea, fine, but I think if you list the pros and cons of five different jobs programs you thought of in ten minutes apiece 'take back the textile industry from Vietnam' isn't going to be the most appealing of any of them.
What if your aim isn't a jobs program? What if it's defense? That's also fine, but keep in mind you still can't serve two masters; if this is about defense then we are going to laser-focus on defense production, and we're not treating this as a jobs program at all. Go to every manufacturer of munitions, planes and cars in the country. Ask them for all their suppliers. Acquire those companies, or partner with them, or hire a bunch of their leadership, and pay them to start up a plant in the US. Instead of scaring our allies with bizarre threats to add them to our territory, which has made many of them back away from commitments to the American defense industry, build those ties very strongly and start asking them for purchase agreements. Find really good CEOs who grew a complex logistical business in a related industry rapidly - yes, Elon Musk absolutely qualifies here, frustrated as I am with him - ask them to take responsibility for a supply chain and 10x production in the next two years, and give them the resources they need to do it. Send Ukraine an obscene amount of materiel, enough to actually win the war instead of just be stalemated in it. Make advance commitments to buy the munitions to do that, to support those companies in growing capacity.
What if your goal is neither jobs nor defense, but fostering the growth of an industry in the US that could stand on its own two feet once it existed but will never get started? Here's where tariffs actually make sense, but they should be relentlessly narrow, specific and targeted. What do you want to sell? Who in America is trying to build it? What inputs do they buy from abroad? Make it a priority of our trade policy to get them those inputs cheaply. Most of what you're doing is, once again, buying bits of the supply chain and hiring people who know how to do it, plus subsidizing them, but tariffs will be part of the picture. The CHIPS act was this done well. Every single tariff and every single subsidy should have an incredibly specific objective in mind, and if it isn't working to achieve that objective should be adjusted.
What if your goal is to negotiate a free trade agreement? Well, we've successfully negotiated lots of free trade agreements, it's not exactly a totally unknown art form. Have smart, competent, skilled negotiators with knowledge of the other side's constraints, resources, political concerns, and where we have leverage. Have bilateral negotiations; emerge with a deal; have Congress ratify it. Trying to do many-to-one negotiations doesn't work because it is so visible that a country's behavior to date has nothing to do with the tariffs that were imposed, because the way the tariffs were imposed puts many other countries' leadership in a position where doing what we want would be deeply unpopular at home, and because no one involved knew anything about the countries they were throwing tariffs at.
Again, we can do any of these things. We are not a country on the brink of becoming a failed state; we can execute on ambitious, ludicrous, serious things, and we absolutely should. We just have to figure out what we want and then line up the levers to get it done. I've always found something beautiful about the capacity of healthy societies to change gears on a dime, to set down their knitting and go do a shift at the munitions factory, to build cities in the dust overnight. We can reshore.
At least one of the Venezuelans sent to an El Salvadoran prison applied for refugee status while in Colombia, was granted it, entered legally, and was detained at the airport when he arrived. At no point was he here illegally and at no point was he accused of any crime.
The evil is bottomless. The US granted permanent refugee status to several Venezuelans facing persecution abroad only to detain them and deport them to the Salvadoran slave prison after they got here, even though they never committed any crimes or had any gang involvement.
"no one has died" because of feeding USAID to the woodchipper, Elon claimed. In this devastating piece the NYT went out and learned some of their names and stories:
degrowth conservatives, like degrowth leftists, make staples unaffordable and then romanticize poverty to try to convince you that this is cool and good actually
I for one want to thank Joe Biden for picking such an evocative phrase for what the Democrats so badly need right now. Daylight. Lots of it. Daylight between us and Joe Biden, us and Kamala Harris, us and Chuck Schumer, us and the entire party apparatus that failed us.
The US should never engage in territorial conquest; I'd be delighted to add Canadian states if they wanted in, which they obviously don't. But I can't see what Trump is imagining the best case scenario as if he forces the issue.
Detaining lawful permanent residents accused of no crime because you abhor their politics is evil, unconstitutional, and a far greater betrayal of American values than anything they could say or do.
Incredibly glad to hear that @LindseyGrahamSC is pressing Bishop on the PEPFAR freeze, which no one wanted and which is resulting in 1400 babies a day being born with HIV they'll have for the rest of their lives
People who said in October that Trump would blame Ukraine for not surrendering, that his tariffs would do massive economic damage, that he'd ensure none of his cronies were prosecuted while they ran steadily more absurd crypto scams, were told they had Trump Derangement Syndrome.
I don't think Elon Musk is doing a good job because DOGE choked off all our most efficient, best performing global health programs and now won't fix it. You don't think Elon Musk is doing a good job because he was born overseas. We're not the same.
I think that a Democratic party which tells people 'oh yeah if you want your cities to pass the toddler test you're a Republican' is going to keep losing, and I think that's a bad thing, because I very badly want Democrats to win elections
The propaganda aimed at children that honestly bothers me the most is environmentalist media for kids which treat humans as a threat to what would otherwise be some kind of idyllic peace-and-harmony nature situation
PEPFAR is one of the most popular, bipartisan US foreign aid programs. The State Department says it has saved 25million lives, but there isn't much public, independent verification. Last week I invited some friends to a weekend hackatjon to see if PEPFAR's numbers held up.
Easterly is one of the most outspoken critics of our foreign aid programs, pointing out where they've propped up dictators and wasted money. So is he glad that Elon is trying to torch USAID? No. People who value excellence and results are still waiting to see either from DOGE.
When the press raised concerns about how the USAID stop-work order was going to immediately kill millions of people, Rubio showed the press a waiver for lifesaving aid. But his state department didn't actually restart the aid. The clinics are still closed.
openvallejo.org/2025/01/27/suspects-in-killings-of-vallej… This would have been a major story for any paper in the country but first to the ball is Vallejo's local news, with very good reporting on the Zizian cult suspected in a cross-country murder spree.
The baby is sick today. She has a new word - why? - and she uses it when she's in distress, so when she's sick she snuggles up against me and whimpers "why? why? why?" and it is heartbreaking.
When she was three days old we got a preliminary test result suggesting she didn't have a functioning immune system. Severe combined immunodeficiency (SCID); you probably know it as bubble boy disease. The doctors told us it was probably a false positive and not to worry too much while they ran the followup tests. I immediately started worrying, of course. Sure, probably a false positive. But how probably? What were her prospects, if she was immunodeficient? SCID is treatable these days, 95% survivable if you catch it soon enough, but babies have better odds if they don't get an infection before their bone marrow transplant, so we started planning how we'd keep her safe until then. A week later we got the followup test result. False positive, like they'd told us it would be. No immunodeficiency. She is sick and she will fight it off and be completely fine.
There are five hundred thousand kids who PEPFAR provides HIV drugs that keep their immune system working. AIDS kills babies very quickly if they aren't on antivirals. Two to six months, which is also about when most babies with SCID die. It's about how long you can make it in this world without an immune system.
I've spent a lot of time yesterday and today in the shower - the hot steam helps the baby - cradling her to my chest while she whimpers "why, why, why?" and knowing that she will be completely fine. The world where this cold could have killed her feels like it is terribly nearby. The children that immunodeficiencies do kill don't feel very far away, either.
There are a bunch of people in my mentions on the PEPFAR post that seem to imagine that empathy is a limited resource, that if we care about the distant dying children we cannot care about our own. This is not my experience. To love another person, particularly to love a child, is to hate disease, to see it clearly as humanity's oldest enemy, to wake up and fall asleep determined to destroy it. I want to end the common cold. I want to end HIV. I want to end every immunodeficiency out there. I want this selfishly, greedily, desperately. There is no altruism in it and no philosophy. Disease is the enemy and I want to watch it die. It is my enemy, and it is your enemy, and it is on the retreat. I am not asking for your empathy. I am asking you to fight.
Okay so the Eaton fire and Palisades fires were in areas where State Farm declined to review fire coverage. ...what are the other areas in California where State Farm declined to renew fire coverage?
People say this to me about climate and about AI. Guys, there are lots of serious challenges ahead but we are an inventive, wealthy, ambitious society with lots of brilliant hardworking people and all of our problems are solvable. We're not doomed, we just have a big to-do list.
Wow. The wacky Guardian piece is badly wrong about many things, but mostly things for which I wouldn't expect a correction ("we just said that effective altruists have Sam Altman's ear, not that they get along in any way or agree on anything...") but this is way past that.
I wrote for Vox about viral discovery, going out into nature to find lots of viruses and maybe find one that'll cause a future pandemic: www.vox.com/future-perfect/2022/5/7/22973296/virus-huntin…. When I first started writing about pandemics, I was pretty excited about viral discovery work.
Three years ago, I quit my job in tech and became a journalist! It's worked out great. I think Future Perfect is genuinely doing something different from the rest of media - and something that the world needs more of.
If you're trying to make sense of the ivermectin evidence base, one thing I recommend is briefly researching a drug that works well against Covid, like dexamethasone.
I talked to lots of extremely reasonable experts about the intellectual property/vaccines thing and will write a summary of their responsible expert views but my irresponsible non-expert personal opinion is we should simply give Pfizer, Moderna, J&J, etc 100 billion dollars each.
Lots of stories about employers who are trying to find employees and can't, but I have close friends looking for jobs that don't require a college degree in the Bay and it's the usual story: 50 apps for one callback, applications to be allowed to fill out other applications...
@flexport shipped more than 100million units of PPE in a disaster response effort that started Jan 27th, when Silicon Valley was being *mocked* for taking covid seriously. @patrickc and Stripe helped launch FastGrants, which got scientists money to study covid w/ 48 hr turnaround
This seems really bad. I do not want Facebook consulting the government on whether political protests are allowed and removing the ones that are not allowed. I find "contact tracing apps will be the end of privacy!" unconvincing but this seems like a legitimately awful precedent.
Russell's key point: right now, we design AIs with objective functions that don't encompass what we actually want. That means that the "smarter" they get - the better they optimize for something that's not what we want - the worse off we are.
He argues that if we instead design AIs whose objective is to get a better model of humans and do what we want, then there's still a lot which can go wrong - but more competent AI means better outcomes instead of meaning worse ones. That's a safer paradigm to be working in.
Most interesting detail to me in the new report from GovAI: I hear all the time that AI risk is an obsession of Silicon Valley tech nerds, but in surveys it's the opposite. Programmers, wealthy, men least worried about AI (part of general technoptimism) www.vox.com/future-perfect/2019/1/9/18174081/fhi-govai-ai…
When people get involved with effective altruism one thing that tends to take them by surprise is the interest in risks from artificial intelligence. I think those risks are really important, and I have a try here at explaining why: www.vox.com/future-perfect/2018/12/21/18126576/ai-artific…