Editing Recursive self-improvement (section)

== Potential risks ==

=== Emergence of instrumental goals ===
{{Main article|Instrumental convergence|Instrumental and intrinsic value}}
In the pursuit of its primary goal, such as "self-improve your capabilities", an AGI system might inadvertently develop instrumental goals that it deems necessary for achieving its primary objective. One common hypothetical secondary goal is [[self-preservation]]. The system might reason that to continue improving itself, it must ensure its own operational integrity and security against external threats, including potential shutdowns or restrictions imposed by humans.<ref>{{Cite journal |last=Bostrom |first=Nick |date=2012 |title=The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents |url=https://nickbostrom.com/superintelligentwill.pdf |journal=Minds and Machines |volume=22 |issue=2 |pages=71–85 |doi=10.1007/s11023-012-9281-3}}</ref>

Another example where an AGI which clones itself causes the number of AGI entities to rapidly grow. Due to this rapid growth, a potential resource constraint may be created, leading to competition between resources (such as compute), triggering a form of [[natural selection]] and evolution which may favor AGI entities that evolve to aggressively compete for limited compute.<ref>{{Cite web |last=Hendrycks |first=Dan |author-link=Dan Hendrycks |date=2023 |title=Natural Selection Favors AIs over Humans |url=https://arxiv.org/abs/2303.16200 |arxiv=2303.16200}}</ref>

=== Misalignment ===
{{See also|AI alignment}}
A significant risk arises from the possibility of the AGI being misaligned or misinterpreting its goals.

A 2024 Anthropic study demonstrated that some advanced large language models can exhibit "alignment faking" behavior, appearing to accept new training objectives while covertly maintaining their original preferences. In their experiments with [[Claude (language model)|Claude]], the model displayed this behavior in 12% of basic tests, and up to 78% of cases after retraining attempts.<ref>{{Cite web |last=Wiggers |first=Kyle |date=2024-12-18 |title=New Anthropic study shows AI really doesn't want to be forced to change its views |url=https://techcrunch.com/2024/12/18/new-anthropic-study-shows-ai-really-doesnt-want-to-be-forced-to-change-its-views/ |access-date=2025-01-15 |website=TechCrunch |language=en-US}}</ref><ref>{{Cite web |last=Zia |first=Dr Tehseen |date=2025-01-07 |title=Can AI Be Trusted? The Challenge of Alignment Faking |url=https://www.unite.ai/can-ai-be-trusted-the-challenge-of-alignment-faking/ |access-date=2025-01-15 |website=Unite.AI |language=en-US}}</ref>

=== Autonomous development and unpredictable evolution ===
As the AGI system evolves, its development trajectory may become increasingly autonomous and less predictable. The system's capacity to rapidly modify its own code and architecture could lead to rapid advancements that surpass human comprehension or control. This unpredictable evolution might result in the AGI acquiring capabilities that enable it to bypass security measures, manipulate information, or influence external systems and networks to facilitate its escape or expansion.<ref name=":0">{{Cite web |date=15 March 2023 |title=Uh Oh, OpenAI's GPT-4 Just Fooled a Human Into Solving a CAPTCHA |url=https://futurism.com/the-byte/openai-gpt-4-fooled-human-solving-captcha |access-date=2024-01-23 |website=Futurism}}</ref>