Editing Superintelligence (section)

== Potential threat to humanity ==
{{Main|Existential risk from artificial general intelligence|AI alignment|AI safety}}

The development of artificial superintelligence (ASI) has raised concerns about potential existential risks to humanity. Researchers have proposed various scenarios in which an ASI could pose a significant threat:

=== Intelligence explosion and control problem ===

Some researchers argue that through recursive self-improvement, an ASI could rapidly become so powerful as to be beyond human control. This concept, known as an "intelligence explosion", was first proposed by I. J. Good in 1965:

{{quote|Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.<ref>{{Cite journal |last=Good |first=I. J. |date=1965 |title=Speculations Concerning the First Ultraintelligent Machine |journal=Advances in Computers}}</ref>}}

This scenario presents the AI control problem: how to create an ASI that will benefit humanity while avoiding unintended harmful consequences.{{sfn|Russell|2019|pp=137-160}} Eliezer Yudkowsky argues that solving this problem is crucial before ASI is developed, as a superintelligent system might be able to thwart any subsequent attempts at control.<ref>{{Cite journal |last=Yudkowsky |first=Eliezer |date=2008 |title=Artificial Intelligence as a Positive and Negative Factor in Global Risk |url=https://intelligence.org/files/AIPosNegFactor.pdf |journal=Global Catastrophic Risks|doi=10.1093/oso/9780198570509.003.0021 |isbn=978-0-19-857050-9 }}</ref>

=== Unintended consequences and goal misalignment ===

Even with benign intentions, an ASI could potentially cause harm due to misaligned goals or unexpected interpretations of its objectives. Nick Bostrom provides a stark example of this risk:

{{quote|When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.{{sfn|Bostrom|2002}}}}

Stuart Russell offers another illustrative scenario:

{{quote|A system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances, rather than to improve the external world.{{sfn|Russell|2019|p=136}}}}

These examples highlight the potential for catastrophic outcomes even when an ASI is not explicitly designed to be harmful, underscoring the critical importance of precise goal specification and alignment.

=== Potential mitigation strategies ===

Researchers have proposed various approaches to mitigate risks associated with ASI:

* [[AI capability control|Capability control]] – Limiting an ASI's ability to influence the world, such as through physical isolation or restricted access to resources.{{sfn|Bostrom|2014|pp=129-136}}
* Motivational control – Designing ASIs with goals that are fundamentally aligned with human values.{{sfn|Bostrom|2014|pp=136-143}}
* Ethical AI – Incorporating ethical principles and decision-making frameworks into ASI systems.<ref>{{Cite book |last1=Wallach |first1=Wendell |url=https://books.google.com/books?id=_r3N82ETng4C |title=Moral Machines: Teaching Robots Right from Wrong |last2=Allen |first2=Colin |date=2008-11-19 |publisher=Oxford University Press |isbn=978-0-19-970596-2 |language=en}}</ref>
* Oversight and governance – Developing robust international frameworks for the development and deployment of ASI technologies.<ref name=":1">{{Cite web |last=Dafoe |first=Allan |date=August 27, 2018 |title=AI Governance: A Research Agenda |url=https://cdn.governance.ai/GovAI-Research-Agenda.pdf |website=Center for the Governance of AI}}</ref>

Despite these proposed strategies, some experts, such as Roman Yampolskiy, argue that the challenge of controlling a superintelligent AI might be fundamentally unsolvable, emphasizing the need for extreme caution in ASI development.<ref>{{Cite web |last=Yampolskiy |first=Roman V. |date=July 18, 2020 |title=On Controllability of Artificial Intelligence |url=https://philpapers.org/archive/YAMOCO.pdf |arxiv=2008.04071}}</ref>

=== Debate and skepticism ===

Not all researchers agree on the likelihood or severity of ASI-related existential risks. Some, like [[Rodney Brooks]], argue that fears of superintelligent AI are overblown and based on unrealistic assumptions about the nature of intelligence and technological progress.<ref>{{Cite web |last=Brooks |first=Rodney |date=October 6, 2017 |title=The Seven Deadly Sins of AI Predictions |url=https://www.technologyreview.com/2017/10/06/241837/the-seven-deadly-sins-of-ai-predictions/ |access-date=2024-10-23 |website=MIT Technology Review |language=en}}</ref> Others, such as [[Joanna Bryson]], contend that [[Anthropomorphism|anthropomorphizing]] AI systems leads to misplaced concerns about their potential threats.<ref>{{Cite journal |last=Bryson |first=Joanna J |date=2019 |title=The Past Decade and Future of AI's Impact on Society |url=https://researchportal.bath.ac.uk/en/publications/the-past-decade-and-future-of-ais-impact-on-society |journal=Towards a New Enlightenment? A Transcendent Decade|volume=11 |isbn=978-84-17141-21-9 }}</ref>

=== Recent developments and current perspectives ===

The rapid advancement of LLMs and other AI technologies has intensified debates about the proximity and potential risks of ASI. While there is no scientific consensus, some researchers and AI practitioners argue that current AI systems may already be approaching AGI or even ASI capabilities.

* LLM capabilities – Recent LLMs like GPT-4 have demonstrated unexpected abilities in areas such as reasoning, problem-solving, and multi-modal understanding, leading some to speculate about their potential path to ASI.<ref>{{Cite arXiv |date=April 2023 |title=Sparks of Artificial General Intelligence: Early experiments with GPT-4 |eprint=2303.12712 |last1=Bubeck |first1=Sébastien |last2=Chandrasekaran |first2=Varun |last3=Eldan |first3=Ronen |last4=Gehrke |first4=Johannes |last5=Horvitz |first5=Eric |last6=Kamar |first6=Ece |last7=Lee |first7=Peter |author8=Yin Tat Lee |last9=Li |first9=Yuanzhi |last10=Lundberg |first10=Scott |last11=Nori |first11=Harsha |last12=Palangi |first12=Hamid |author13=Marco Tulio Ribeiro |last14=Zhang |first14=Yi |class=cs.CL }}</ref>

* Emergent behaviors – Studies have shown that as AI models increase in size and complexity, they can exhibit emergent capabilities not present in smaller models, potentially indicating a trend towards more general intelligence.<ref name=":0" />

* Rapid progress – The pace of AI advancement has led some to argue that we may be closer to ASI than previously thought, with potential implications for existential risk.<ref>{{Cite book |last=Ord |first=Toby |title=The precipice: existential risk and the future of humanity |date=2020 |publisher=Bloomsbury academic |isbn=978-1-5266-0023-3 |location=london New York (N.Y.)}}</ref>

A minority of researchers and observers, including some in the AI development community, believe that current AI systems may already be at or near AGI levels, with ASI potentially following in the near future. This view, while not widely accepted in the scientific community, is based on observations of rapid progress in AI capabilities and unexpected emergent behaviors in large models.<ref>{{Cite journal |last1=Ngo |first1=Richard |last2=Chan |first2=Lawrence |last3=Mindermann |first3=Sören |date=2022 |title=The Alignment Problem from a Deep Learning Perspective |journal=ICLR |arxiv=2209.00626}}</ref>

However, many experts caution against premature claims of AGI or ASI, arguing that current AI systems, despite their impressive capabilities, still lack true understanding and general intelligence.<ref>{{Cite web |date=17 September 2024 |title=How and Why Gary Marcus Became AI's Leading Critic > Marcus says generative AI like ChatGPT poses immediate dangers |url=https://spectrum.ieee.org/ai-inventions |access-date= |website=IEEE Spectrum |language=en}}</ref> They emphasize the significant challenges that remain in achieving human-level intelligence, let alone superintelligence.

The debate surrounding the current state and trajectory of AI development underscores the importance of continued research into AI safety and ethics, as well as the need for robust governance frameworks to manage potential risks as AI capabilities continue to advance.<ref name=":1" />