Editing Friendly artificial intelligence (section)

== Etymology and usage ==
[[File:Eliezer Yudkowsky, Stanford 2006 (square crop).jpg|thumb|[[Eliezer Yudkowsky]], AI researcher and creator of the term]]
The term was coined by [[Eliezer Yudkowsky]],<ref>{{cite book|last1=Tegmark|first1=Max|title=Our Mathematical Universe: My Quest for the Ultimate Nature of Reality|date=2014|isbn=9780307744258|edition=First|chapter=Life, Our Universe and Everything|quote=Its owner may cede control to what Eliezer Yudkowsky terms a "Friendly AI,"...|title-link=Our Mathematical Universe: My Quest for the Ultimate Nature of Reality|publisher=Knopf Doubleday Publishing }}</ref> who is best known for popularizing the idea,<ref name="aima">{{cite book |last1=Russell |first1=Stuart |author1-link=Stuart J. Russell |last2=Norvig |first2=Peter |author2-link=Peter Norvig |date=2009 |title=Artificial Intelligence: A Modern Approach |publisher=Prentice Hall |isbn=978-0-13-604259-4|title-link=Artificial Intelligence: A Modern Approach }}</ref><ref>{{cite book |last=Leighton |first=Jonathan |date=2011 |title=The Battle for Compassion: Ethics in an Apathetic Universe |publisher=Algora |isbn=978-0-87586-870-7}}</ref> to discuss [[superintelligence|superintelligent]] artificial agents that reliably implement human values. [[Stuart J. Russell]] and [[Peter Norvig]]'s leading [[artificial intelligence]] textbook, ''[[Artificial Intelligence: A Modern Approach]]'', describes the idea:<ref name="aima" />

<blockquote>Yudkowsky (2008) goes into more detail about how to design a '''Friendly AI'''. He asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the challenge is one of mechanism design&mdash;to define a mechanism for evolving AI systems under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.</blockquote>

"Friendly" is used in this context as [[technical terminology]], and picks out agents that are safe and useful, not necessarily ones that are "friendly" in the colloquial sense. The concept is primarily invoked in the context of discussions of recursively self-improving artificial agents that rapidly [[intelligence explosion|explode in intelligence]], on the grounds that this hypothetical technology would have a large, rapid, and difficult-to-control impact on human society.<ref>{{cite book |last1=Wallach |first1=Wendell |last2=Allen | first2=Colin |date=2009 |title=Moral Machines: Teaching Robots Right from Wrong |publisher=Oxford University Press, Inc. |isbn=978-0-19-537404-9 }}</ref>