Existential Risk from Power-Seeking AI
Joe Carlsmith · 2025
Abstract This essay formulates and examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I begin by discussing a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire—especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. I then examine the type of agents we should be worried about; the incentives to create them; the difficulty of ensuring that they don’t seek power in unintended ways; the reasons to expect them to end up deployed regardless; the likelihood that the problem scales to the permanent disempowerment of humanity; and the value lost if so. My current view is that the existential risk at stake is disturbingly high (i.e. greater than 10% by 2070).