Editing Neural network (machine learning) (section)

=== Backpropagation ===
[[Backpropagation]] is an efficient application of the [[chain rule]] derived by [[Gottfried Wilhelm Leibniz]] in 1673<ref name="leibniz16762">{{Cite book |last=Leibniz |first=Gottfried Wilhelm Freiherr von |url=https://books.google.com/books?id=bOIGAAAAYAAJ&q=leibniz+altered+manuscripts&pg=PA90 |title=The Early Mathematical Manuscripts of Leibniz: Translated from the Latin Texts Published by Carl Immanuel Gerhardt with Critical and Historical Notes (Leibniz published the chain rule in a 1676 memoir) |date=1920 |publisher=Open court publishing Company |isbn=9780598818461 |language=en}}</ref> to networks of differentiable nodes. The terminology "back-propagating errors" was actually introduced in 1962 by Rosenblatt,<ref name="rosenblatt1962"/> but he did not know how to implement this, although [[Henry J. Kelley]] had a continuous precursor of backpropagation in 1960 in the context of [[control theory]].<ref name="kelley19602">{{cite journal |last1=Kelley |first1=Henry J. |author-link=Henry J. Kelley |year=1960 |title=Gradient theory of optimal flight paths |journal=ARS Journal |volume=30 |issue=10 |pages=947–954 |doi=10.2514/8.5282}}</ref> In 1970, [[Seppo Linnainmaa]] published the modern form of backpropagation in his Master's [[thesis]] (1970).<ref name="lin19703">{{cite thesis |first=Seppo |last=Linnainmaa |author-link=Seppo Linnainmaa |year=1970 |type=Masters |title=The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors |language=fi |publisher=University of Helsinki |page=6–7}}</ref><ref name="lin19763">{{cite journal |last1=Linnainmaa |first1=Seppo |author-link=Seppo Linnainmaa |year=1976 |title=Taylor expansion of the accumulated rounding error |journal=BIT Numerical Mathematics |volume=16 |issue=2 |pages=146–160 |doi=10.1007/bf01931367 |s2cid=122357351}}</ref><ref name="DLhistory" /> G.M. Ostrovski et al. republished it in 1971.<ref name="ostrowski1971">Ostrovski, G.M., Volin,Y.M., and Boris, W.W. (1971). On the computation of derivatives. Wiss. Z. Tech. Hochschule for Chemistry, 13:382–384.</ref><ref name="backprop"/> [[Paul Werbos]] applied backpropagation to neural networks in 1982<ref name="werbos1982">{{cite book |last=Werbos |first=Paul |author-link=Paul Werbos |title=System modeling and optimization |publisher=Springer |year=1982 |pages=762–770 |chapter=Applications of advances in nonlinear sensitivity analysis |access-date=2 July 2017 |chapter-url=http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf |archive-url=https://web.archive.org/web/20160414055503/http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf |archive-date=14 April 2016 |url-status=live}}</ref><ref name=":1">{{Cite book |url=https://direct.mit.edu/books/book/4886/Talking-NetsAn-Oral-History-of-Neural-Networks |title=Talking Nets: An Oral History of Neural Networks |date=2000 |publisher=The MIT Press |isbn=978-0-262-26715-1 |editor-last=Anderson |editor-first=James A. |language=en |doi=10.7551/mitpress/6626.003.0016 |editor-last2=Rosenfeld |editor-first2=Edward |archive-date=12 October 2024 |access-date=7 August 2024 |archive-url=https://archive.today/20241012223136/https://direct.mit.edu/books/book/4886/Talking-NetsAn-Oral-History-of-Neural-Networks |url-status=live }}</ref> (his 1974 PhD thesis, reprinted in a 1994 book,<ref name="werbos1974">{{cite book |last=Werbos |first=Paul J. |title=The Roots of Backpropagation : From Ordered Derivatives to Neural Networks and Political Forecasting |location=New York |publisher=John Wiley & Sons |year=1994 |isbn=0-471-59897-6 }}</ref> did not yet describe the algorithm<ref name="backprop">{{cite web | last = Schmidhuber | first = Juergen | title = Who Invented Backpropagation? | author-link = Juergen Schmidhuber | publisher = IDSIA, Switzerland | url = https://people.idsia.ch/~juergen/who-invented-backpropagation.html | date = 25 October 2014 | access-date = 14 September 2024 | archive-url = https://web.archive.org/web/20240730110408/https://people.idsia.ch/~juergen/who-invented-backpropagation.html | archive-date = 30 July 2024 | quote =  | url-status = live }}</ref>). In 1986, [[David E. Rumelhart]] et al. popularised backpropagation but did not cite the original work.<ref>{{Cite journal |last1=Rumelhart |first1=David E. |last2=Hinton |first2=Geoffrey E. |last3=Williams |first3=Ronald J. |date=October 1986 |title=Learning representations by back-propagating errors |url=https://www.nature.com/articles/323533a0 |journal=Nature |language=en |volume=323 |issue=6088 |pages=533–536 |doi=10.1038/323533a0 |bibcode=1986Natur.323..533R |issn=1476-4687 |archive-date=8 March 2021 |access-date=17 March 2021 |archive-url=https://web.archive.org/web/20210308045630/https://www.nature.com/articles/323533a0 |url-status=live }}</ref>