Unfortunately, when you try to starve the beast, it's the essential functions that get slashed instead of the leadership and adminstrative apparatus. The animal is harmed further while the parasite intercepts the generated value.
If he's a clown what part of his theory is the circus?
Are you saying that superintelligence is impossible?
Are you saying that the alignment problem will certainty be solved before superintelligence emerges?
Are you saying that a superintelligent being connected to the internet would be unable to gain resources such as GPU time, money, and social influence?
Are you saying that a superintelligent being would for some reason be incapable of deception and cunning?
Are you saying that a superintelligent being would necessarily regard human flourishing as a prime objective to be prized above it's own goals and ambitions?
If it's really just doomerism we should be able to point to the flaws in his argument instead of making ad hominem attacks.
>Are you saying that superintelligence is impossible?
Yes, end of the discussion, I don't debate metaphysics with crackheads, sociology with psychopaths or geography with flat earthers, or very bad science fiction with yudkowski. Going on about "exposing the flaws in the argument" of the crackhead just means wasting your time.
I haven't read from him in a while, but I don't think there was a single dated prediction that supports your argument. Do you have any sources for this claim?
Researches at top AI labs don't consider EY to be a kook even though they may not necessarily agree. EY concepts/terminology appear in Anthropic safety papers. Geoffrey Hinton takes him quite seriously and mentions him in his interviews.
Anthropic is the AI doomer / safetyism lab, and Hinton is one of the patron saints of 'rationalist' AI doomerism.
AI doomerism is psychologically attractive to "people with autistic cognitive traits, including dichotomous (black-and-white) thinking, intolerance of uncertainty, and a tendency toward catastrophizing". They are pascal's mugging themselves, to ironically use one of their terms. It's fundamentally a cognitive distortion.
"What if AI doom is all fear-mongering, and we create AI less prone to make up dangerous stuff or mistake buggy goals for real ones" (which is what alignment is) "for nothing?"
Even if Yudkowsky is autistic, you're muddling the condition. Autistic people have a *practical* intolerance of uncertainty in the moment (everything unexpected from a noise to a missed turn can be a jump-scare in their day-to-day activities), but often they're absolutely fine with intellectual uncertainty, unconventional ideas, abstract ambiguity, nonconformity, etc. Indeed, one of Yudkowsky's main things is Bayesianism, i.e. being precise about uncertainty.
Yudkowsky's reported P(doom) is somewhere around 90%, which is very much in the realm of "we might eventually be able to figure this out, *but we're not even close to ready so for the love of everything slow down so we can figure this all out*"; the book title comes from a long tradition of authors noticing you need to beat readers over the head with your point for them to notice it.
Anthropic (like at least also OpenAI), appears to think they can solve the problems that Yudkowsky has found. They're a lot more optimistic than him, but they take these problems seriously.
For his work on AI, Hinton got a Nobel prize in Physics, a Turing Award, the inaugural Rumelhart Prize, a Princess of Asturias Award, a VinFuture Prize, and a Queen Elizabeth Prize for Engineering. Calling him a "patron saint" of "doomerism" is like calling Paul Krugman (Nobel laureate in Economics) a patron saint of "Trump Derangement Syndrome" on the basis of what he says in his YouTube channel: a smart person's considered opinions are worth listening to even if you have not got time for the details, because you can be sure someone else has considered the details and will absolutely be responding to even an i missing a dot.
Much like a lot of LLM usage burns tokens so that mediocre people can hallucinate that they're doing something brilliant, Yudkowskyism is just a lot of empty verbiage for the purpose of building a sex cult around a plump gnome. Reusing his nonsensical and poorly defined terms but failing to get the benefit of the sex cult really misses the point of the entire exercise.
The problem is that effort spent to reduce the "risk" of creating an evil god who tortures us all for the rest of time doesn't actually produce outcomes that reduces the risk of things like widespread job loss or hyperaggregation of influence and money.
"Oh we'll at least get some side benefit" is not actually what is coming out of the endlessly circular forums talking about the apocalypse.
Even if there was no overlap*, that would be like criticising the green movement for not focussing on working hours and pay like trade unions do.
Different people can care about different things; it's good that each of us gets to focus on what motivates us, rather than all chasing the same thing, because when multiple teams do all chase the same thing typically only the best few of them actually make a difference.
* as it happens, there is some overlap. Knowing more about how a narrow utility function behaves outside distribution is useful for both capabilities and safety. We're not even at the stage of being able to make AI not kill random subsets of the users with bad advice, nor reliably prevent users from falling into delusions of grandeur, let alone giving AI a reliable sense of liberty and the pursuit of happiness to maintain.
The people who've made the biggest contribution to creating a better world over the last 50 years have been the Chinese; powered largely by coal and petroleum. And in one of the most ironic results in the 21st century, they're now the leaders in solar panel production on the back of the largest investment in fossil fuel energy in global history.
The comic ran into the same problem as the climate change movement in general - they proposed ideas that generally made people worse off. And if measured in terms of CO2 emissions achieved nothing except pushing wealth creation to Asia. Which, in fairness, is probably appreciated by the Asians.
BYD had release the first plug in hybrid the year before.
The Beijing Olynpics had made air pollution a hot topic in China in 2007-8.
Wind power had accelerated after their 2005 Renewable Energy Law.
Solar panel production rose around this time, taking over the market from European manufacturers when the Financial Crisis hit and they pulled back investments.
So China at that time, was doing all the things on the cartoon's presentation list, and has benefitted greatly from them.
Many people in Europe want to see green energy transition. But no transition is happening in China.
"
“We see addition, not transition,” said Yasheng Huang, a professor of global economics and management at the MIT Sloan School. “China is building alternative sources of energy as well as fossil energy sources, simultaneously. In terms of the global footprint on CO2, China is emitting twice as much as Europe and the United States. I don’t think there’s a transition going on.”
"
What an embarrassingly ill-informed thing to say. But when the guy wrote a book in 2023 about the fall of China, he kind of has to say that doesn't he, even as he lives through the fall of the USA.
He's called out in the sub-head as an "expert" but what is he an expert in? Renewables? Energy policy? No, he's an expert in saying that China is too state-led. Why would an expert in that want to downplay their success, apart from all the obvious reasons?
"
For Beijing to achieve those goals, Climate Action Tracker says China needs "clear targets for coal consumption reduction" in its new 5YP. However, the economic roadmap released in March was not "explicit about how fossil fuels will be constrained," said China analyst Qi Qin of the Finland-based Center for Research on Energy and Clean Air.
Though Chinese President Xi Jinping promised in 2021 to detail a reduction in coal energy use in the 2026-31 plan, it contains "no clear phase-down plan, no clear fossil fuel cap," said Qin. "The language is much more conservative than many people expected," she told DW. One reason is the continued influence of the powerful coal lobby on Chinese government policy.
"
> New Chinese government guidelines on fossil fuels released on April 22 support the view that the country is willing to move away from finite fossil fuels, strengthen energy independence and still achieve its climate targets, says Qin.
> "The new central guideline talks about strictly controlling fossil fuel consumption, reducing coal and controlling oil. It still leaves room for flexibility, but these are concrete policy levers," Qin said of the document, which also indicated a desire to increase clean energy consumption.
Elsewhere Climate Action Tracker on the USA:
> The Trump Administration is pursuing an executive and legislative agenda to systematically repeal targets, policies, and funding for climate change mitigation and science. The administration is actively obstructing the buildout of renewable energy, while encouraging the production and consumption of fossil fuels, completely reversing the Biden Administration’s course on climate action. This is the most aggressive, comprehensive, and consequential climate policy rollback that the Climate Action Tracker has ever analysed.
All of which, even the bits quoted to claim "no transition is happening", support my original contention that all the things mentioned in the cartoon were being strongly pushed by China in 2009. They have only gained momentum since and they've profited from doing so.
I agree that the Chinese 5Y plan is better than the US policy "Drill, baby, drill!". But how much exactly, we will see.
Lets drill more into details:
"
Against this backdrop, coal has re-emerged as a critical stabilizing force in China’s power system. This helps explain the relatively cautious policy signals embedded in China’s 15th Five-Year Plan. While the plan reiterates long-term decarbonization commitments — including reducing carbon intensity by 17 percent and raising the share of non-fossil energy to 25 percent by 2030 — it still stops short of setting explicit timelines for coal or oil consumption to peak. This reflects a deliberate effort to preserve flexibility as Chinese policymakers balance energy transition goals with near-term electric system stability.
In practice, China’s coal production has rebounded significantly. After nearly a decade of supply-side reforms that kept output around 4 billion tonnes annually, coal production rose sharply following the 2022 power shortages and has continued to increase, reaching a record 4.85 billion tonnes in 2025.
"
I would like to note "reducing carbon intensity", doesn't mean reducing total carbon emissions, it's reducting carbon emissions per unit GDP.
This is in strong contrast to EU
"Specifically, the EU has a legally binding headline emission reduction target of 90% by 2040 relative to 1990, with a domestic target of 85% and up to 5% of international carbon credits."
Something that has been largely forgotten about is that it used to be routine to see pictures of smoggy Chinese and Asian cities, this was a problem for them that they solved. I can't help thinking we can't get this kind of preventative action on any large scale, we need to have severe issues first and that's not accounting for longer term/cumulative effects.
"Over the past years, the government has implemented various methods to improve the air quality in Northern China. Sandstorms, which were quite common 15 years ago, are now rarely seen in Beijing’s spring thanks to afforestation projects on China’s northern borders. The license-plate lottery system was introduced in Beijing to restrict the growth of private vehicles. Large trucks were not allowed to enter certain areas in Beijing. Above all, the coal consumption in Beijing has been restricted by shutting down industrial sites and improving heating systems. Beijing’s efforts to improve air quality has also been highly praised by the UN as a successful model for other cities. However, there is also criticism pointing out that the improvement of Beijing’s air quality is based on the sacrifice of surrounding provinces (including Hebei), as many factories were moved from Beijing to other regions."
The US had the same issue and fixed it through federal and state environmental regulation. It just happened in the US 100 years before it happened in china Heavy pollution is what lead to the environmental movement that started back in the 60s and that led to the creation of the EPA and whole slate of state and federal regulation that dramatically improved air/water quality in the US. It was a slow process that took a ton of work to build a movement of support, but it can be done.
We can actually address problems when we want to. It's just pretty slow and requires people to actually give a shit and put in the effort to build support.
The unfortunate comparable here is that all the people who care about making sure their AI is safe, regardless of what they mean by that, are beaten to the market by the people who don't.
Just because some researchers are infected with this idiocy that EY propagates does not mean that it is legit.
Maybe they should pay more attention to real problems like the sycophantic nature of current LLMs causing psychosis in people and worry less about theoretical AGI.
Who are you to say? Why do you have such little regard for everyone in the field, both pro- and anti- AI development? Do you think they're colluding to deceive us?
theres billions, even trillions of dollars on the line, why not start with the assumption they have every incentive to deceive, even if unintentional (ie, deceiving themselves)
Ready Player One was completely misread and misunderstood by people who thought they could make a lot of money with VR.
It wasn't a homage to 70s/80s/90s nerd culture and a hopeful glimpse of what VR tech could be.
It was a warning for people to get off their fucking phones and to work together at improving the real world, versus ignoring it and living out unrealistic fantasies inside a digital ecosystem that makes us all a bit less human.
The whole point of the book is that VR and addictive tech is a red herring. It was deliberately misunderstood by Zuck and his ilk.
Thank you, like most of the world I would assume "EY" would refer to Ernst and Young, the multi-national Big Four with a website of ey.com who I'm sure has opinions on AI, but nowhere near enough to be classed as expertise
The evidence is just "human nature". Honestly, it's just negligence at this point to give people power over others without due oversight and accountability. But it's nice we have concrete examples of abuse to help motivate action.
I think a lot of the objections to your post could be answered by reminding folks of how Microsoft Access databases tend to pop up in small businesses as well as corporate environments outside of IT departments. Yes, they're not "proper" databases but they /get business done/ and often serve as v0 before a real app can be properly conceived of.
One can easily imagine an LLM-enabled database that lets a wider audience build meat-and-potatoes line-of-business apps for small team use with minimal compliance concerns.
Yes, that's the right framing. Millions flow through spreadsheets/CSVs/MS Access with none of the auth/backups/architecture people seem to be stuck to.
I saw an article on HN one time about CSVs and how much business still flows through them. Reminds me of the xkcd comic about the one tiny block propping up lots of infrastructure. It stuck with me because it's ripe area for LLM agent based upgrades.
Sure don't give LLMs access to the well architected blocks. But not wanting to improve the brittle areas seems crazy to me even if it's contrarian.
A military coup in the U.S. is imaginable, which probably explains some of the top brass purges (until recently, where it's probably an attempt to deflect blame for the massive Iran fuck up).
Putin did it better; he kept the military weak and aggressively managed the risk via the FSB.
I don't think it's plausible, but an authoritarian president invoking emergency powers and deploying military and paramilitary forces to exert control on the streets is, on the basis it's already going on at a limited scale. All it takes is for that scale to gradually dial up over time until the frog's cooked.
The problem you have is these elected kings. Not just any king, pretty specifically the majority of the powers enjoyed by George III in the 1790s. The fact that you still have this, unreformed over 200 years later and still think that somehow your constitutional system is modern, is a matter for despair. Get yourselves a proper parliamentary system, with maybe a head of state as a figurehead.
>The problem you have is these elected kings. Not just any king, pretty specifically the majority of the powers enjoyed by George III in the 1790s. The fact that you still have this, unreformed over 200 years later and still think that somehow your constitutional system is modern, is a matter for despair. Get yourselves a proper parliamentary system, with maybe a head of state as a figurehead.
What a poorly thought out and questionably motivated take. It will no doubt be well received here.
In any case, reconstructing out legislature to copy european stuff isn't gonna change anything if the legislature still sees fit to vest so much power in the executive.
My point is precisely that the US system is substantially a copy of European stuff. It had some significant innovations for it's time of course, but it's really showing it's age. Meanwhile Parliamentary systems have significantly reformed and further innovated since.
Your main point is valid, but I'd argue it's less the power of the President and more the two-party system and the weakness of Congress that is the root of many American governance problems. Executive power has grown in the vacuum of Congressional impotence.
As far as reforms, we need more to be sure, but there's at least the 22nd Amendment, formalizing the two-term tradition that Washington initiated and FDR abrogated into a hard limit, that means Trump can't legally keep power past 2028.
I think, despite all the furor whipped up and attempts by high-ranking officials to foment a coup, the military is actually very unlikely to disobey Presidential orders; this is based on my frequent interactions with military officers over the last ten years. Do you perceive some other, more likely, coup scenario?
A coup from the same military that happily deployed foot soldiers into American cities to perform law enforcement duties they were not trained to perform? And is happily killing boat-people in the Caribbean? And ran a covert operation to kidnap a foreign head of state? And ran another covert operation to assassinate political leaders in another sovereign state and are now bombing that same state into rubble for no publicly disclosed reason?
Yeah, no coup is happening here. Our military is built around civilian control via the White House. That ain't changing any time soon. Ignoring the various uses of force above, when the generals were called before Hegseth to bend the knee, all but one of them did.
It's not for me, but I can see the appeal - minimalism, distraction elimination, geek cred, and the sort of flow state one gets from working in a low latency, high muscle memory environment.
Okay, I'll say it: is it really worth encumbering the movements of millions of people for decades in order to make a few boring history exhibits? If you want to see some the bone comb that belonged to somebody's great^100-grandmother, there are dozens of museums that already have one on display.
Amazon extracts a lot of the value of a purchase from the seller's take. Sellers risk sanctions if they sell a product cheaper thru their brand website.
reply