Machine Intelligence Research Institute

In 2000, Eliezer Yudkowsky founded the Singularity Institute for Artificial Intelligence with funding from Brian and Sabine Atkins, with the purpose of accelerating the development of artificial intelligence (AI). However, Yudkowsky began to be concerned that AI systems developed in the future could become superintelligent and pose risks to humanity. In 2005 the institute began operating in Silicon Valley and focused on ways to identify and manage those risks, which were at the time largely ignored by scientists in the field.

Starting in 2006, the Institute organized the Singularity Summit to discuss the future of AI including its risks, initially in cooperation with Stanford University and with funding from Peter Thiel. The San Francisco Chronicle described the first conference as a "Bay Area coming-out party for the tech-inspired philosophy called transhumanism". In 2011, its offices were four apartments in downtown Berkeley. In December 2012, the institute sold its name, web domain, and the Singularity Summit to the Singularity Group, and in the following month took the name "Machine Intelligence Research Institute".

In 2014 and 2015, public and scientific interest in the risks of AI grew, increasing donations to fund research at MIRI and similar organizations.

In 2019, Open Philanthropy recommended a general-support grant of approximately $2.1 million over two years to MIRI. In April 2020, MIRI announced an additional $7.7 million grant from Open Philanthropy over two years.

In 2021, Vitalik Buterin donated several million dollars' worth of Ethereum to MIRI.

MIRI's earlier research on identifying and managing the risks of AI, led by Yudkowsky, primarily addressed the AI control problem and how to design friendly AI, covering both the initial design of AI systems and the creation of mechanisms to ensure that evolving AI systems remain friendly.

MIRI researchers advocate early safety work as a precautionary measure. In 2015, MIRI researchers expressed skepticism about the reliability of forecasts of when advanced AI would emerge. MIRI has funded forecasting work through an initiative called AI Impacts, which studies historical instances of discontinuous technological change, and has developed new measures of the relative computational power of humans and computer hardware.

MIRI aligns itself with the principles and objectives of the effective altruism movement.

In January 2024, MIRI announced that it had shifted its priorities in 2023 toward policy and public communication. It advocated an international agreement to halt progress toward smarter-than-human AI until understanding of such systems substantially improved and they could be secured against misuse, while continuing technical research.

• Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk" (PDF). In Bostrom, Nick; Ćirković, Milan M. (eds.). Global Catastrophic Risks. Oxford University Press. pp. 308–345. doi:10.1093/oso/9780198570509.003.0021. ISBN 978-0-19-857050-9.
• Yudkowsky, Eliezer (2011). "Complex Value Systems in Friendly AI" (PDF). Artificial General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3–6, 2011. Berlin: Springer. pp. 388–393. doi:10.1007/978-3-642-22887-2_48.
• LaVictoire, Patrick; Fallenstein, Benja; Yudkowsky, Eliezer; Bárász, Mihály; Christiano, Paul; Herreshoff, Marcello (2014). "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem" (PDF). Multiagent Interaction without Prior Coordination: Papers from the AAAI-14 Workshop. AAAI Press. pp. 22–27.
• Soares, Nate; Fallenstein, Benja; Yudkowsky, Eliezer; Armstrong, Stuart (2015). "Corrigibility" (PDF). Artificial Intelligence and Ethics: Papers from the 2015 AAAI Workshop. AAAI Technical Report WS-15-02. AAAI Press. pp. 74–82.
• Taylor, Jessica (2016). "Quantilizers: A Safer Alternative to Maximizers for Limited Optimization" (PDF). The Workshops of the Thirtieth AAAI Conference on Artificial Intelligence. AI, Ethics, and Society: Technical Report WS-16-02. AAAI Press. pp. 124–131.
• Soares, Nate; Fallenstein, Benya (23 May 2017). "Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda" (PDF). In Callaghan, Victor; Miller, James; Yampolskiy, Roman; Armstrong, Stuart (eds.). The Technological Singularity: Managing the Journey. Springer. pp. 103–125. doi:10.1007/978-3-662-54033-6_5.
• Graves, Matthew (8 November 2017). "Why We Should Be Concerned About Artificial Superintelligence". Skeptic. The Skeptics Society. Retrieved 28 July 2018.
• Levinstein, Benjamin A.; Soares, Nate (May 2020). "Cheating Death in Damascus" (PDF). The Journal of Philosophy. 117 (5): 237–266. doi:10.5840/jphil2020117516.
• Yudkowsky, Eliezer; Soares, Nate (16 September 2025). If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown and Company. ISBN 978-0-316-59564-3.