Editing Superintelligence (section)

=== Intelligence explosion and control problem ===

Some researchers argue that through recursive self-improvement, an ASI could rapidly become so powerful as to be beyond human control. This concept, known as an "intelligence explosion", was first proposed by I. J. Good in 1965:

{{quote|Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.<ref>{{Cite journal |last=Good |first=I. J. |date=1965 |title=Speculations Concerning the First Ultraintelligent Machine |journal=Advances in Computers}}</ref>}}

This scenario presents the AI control problem: how to create an ASI that will benefit humanity while avoiding unintended harmful consequences.{{sfn|Russell|2019|pp=137-160}} Eliezer Yudkowsky argues that solving this problem is crucial before ASI is developed, as a superintelligent system might be able to thwart any subsequent attempts at control.<ref>{{Cite journal |last=Yudkowsky |first=Eliezer |date=2008 |title=Artificial Intelligence as a Positive and Negative Factor in Global Risk |url=https://intelligence.org/files/AIPosNegFactor.pdf |journal=Global Catastrophic Risks|doi=10.1093/oso/9780198570509.003.0021 |isbn=978-0-19-857050-9 }}</ref>