Editing Neural network (machine learning) (section)

=== Convolutional neural networks ===
Kunihiko Fukushima's [[convolutional neural network]] (CNN) architecture of 1979<ref name="FUKU1979"/> also introduced [[max pooling]],<ref>{{Cite journal |last1=Fukushima |first1=Kunihiko |last2=Miyake |first2=Sei |date=1 January 1982 |title=Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position |url=https://www.sciencedirect.com/science/article/abs/pii/0031320382900243 |journal=Pattern Recognition |volume=15 |issue=6 |pages=455–469 |doi=10.1016/0031-3203(82)90024-3 |bibcode=1982PatRe..15..455F |issn=0031-3203 |archive-date=12 October 2024 |access-date=9 September 2024 |archive-url=https://archive.today/20241012232918/https://www.sciencedirect.com/science/article/abs/pii/0031320382900243 |url-status=live }}</ref> a popular downsampling procedure for CNNs. CNNs have become an essential tool for [[computer vision]].

The [[time delay neural network]] (TDNN) was introduced in 1987 by [[Alex Waibel]] to apply CNN to phoneme recognition. It used convolutions, weight sharing, and backpropagation.<ref name=Waibel1987>{{cite conference |title=Phoneme Recognition Using Time-Delay Neural Networks |last1=Waibel |first1=Alex |date=December 1987 |location=Tokyo, Japan |conference=Meeting of the Institute of Electrical, Information and Communication Engineers (IEICE) |url=https://isl.anthropomatik.kit.edu/pdf/Waibel1987a.pdf |access-date=20 September 2024 |archive-date=17 September 2024 |archive-url=https://web.archive.org/web/20240917173146/https://isl.anthropomatik.kit.edu/pdf/Waibel1987a.pdf |url-status=live }}</ref><ref name="speechsignal">[[Alex Waibel|Alexander Waibel]] et al., ''[http://www.inf.ufrgs.br/~engel/data/media/file/cmp121/waibel89_TDNN.pdf Phoneme Recognition Using Time-Delay Neural Networks] {{Webarchive|url=https://web.archive.org/web/20241211184304/https://www.inf.ufrgs.br/~engel/data/media/file/cmp121/waibel89_TDNN.pdf |date=11 December 2024 }}'' IEEE Transactions on Acoustics, Speech, and Signal Processing, Volume 37, No. 3, pp. 328. – 339 March 1989.</ref> In 1988, Wei Zhang applied a backpropagation-trained CNN to alphabet recognition.<ref name="wz1988">{{cite journal |last=Zhang |first=Wei |date=1988 |title=Shift-invariant pattern recognition neural network and its optical architecture |url=https://drive.google.com/file/d/1nN_5odSG_QVae54EsQN_qSz-0ZsX6wA0/view?usp=sharing |journal=Proceedings of Annual Conference of the Japan Society of Applied Physics |archive-date=23 June 2020 |access-date=12 April 2023 |archive-url=https://web.archive.org/web/20200623051222/https://drive.google.com/file/d/1nN_5odSG_QVae54EsQN_qSz-0ZsX6wA0/view?usp=sharing |url-status=live }}</ref>
In 1989, [[Yann LeCun]] et al. created a CNN called [[LeNet]] for [[Handwriting recognition|recognizing handwritten ZIP code]]s on mail. Training required 3 days.<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition", ''Neural Computation'', 1, pp. 541–551, 1989.</ref> In 1990, Wei Zhang implemented a CNN on [[optical computing]] hardware.<ref name="wz1990">{{cite journal |last=Zhang |first=Wei |date=1990 |title=Parallel distributed processing model with local space-invariant interconnections and its optical architecture |url=https://drive.google.com/file/d/0B65v6Wo67Tk5ODRzZmhSR29VeDg/view?usp=sharing |journal=Applied Optics |volume=29 |issue=32 |pages=4790–7 |bibcode=1990ApOpt..29.4790Z |doi=10.1364/AO.29.004790 |pmid=20577468 |archive-date=6 February 2017 |access-date=12 April 2023 |archive-url=https://web.archive.org/web/20170206111407/https://drive.google.com/file/d/0B65v6Wo67Tk5ODRzZmhSR29VeDg/view?usp=sharing |url-status=live }}</ref> In 1991, a CNN was applied to medical image object segmentation<ref>{{cite journal |last=Zhang |first=Wei |date=1991 |title=Image processing of human corneal endothelium based on a learning network |url=https://drive.google.com/file/d/0B65v6Wo67Tk5cm5DTlNGd0NPUmM/view?usp=sharing |journal=Applied Optics |volume=30 |issue=29 |pages=4211–7 |bibcode=1991ApOpt..30.4211Z |doi=10.1364/AO.30.004211 |pmid=20706526 |archive-date=19 June 2024 |access-date=20 September 2024 |archive-url=https://web.archive.org/web/20240619084309/https://drive.google.com/file/d/0B65v6Wo67Tk5cm5DTlNGd0NPUmM/view?usp=sharing |url-status=live }}</ref> and breast cancer detection in mammograms.<ref>{{cite journal |last=Zhang |first=Wei |date=1994 |title=Computerized detection of clustered microcalcifications in digital mammograms using a shift-invariant artificial neural network |url=https://drive.google.com/file/d/0B65v6Wo67Tk5Ml9qeW5nQ3poVTQ/view?usp=sharing |journal=Medical Physics |volume=21 |issue=4 |pages=517–24 |bibcode=1994MedPh..21..517Z |doi=10.1118/1.597177 |pmid=8058017 |archive-date=20 June 2024 |access-date=20 September 2024 |archive-url=https://web.archive.org/web/20240620055642/https://drive.google.com/file/d/0B65v6Wo67Tk5Ml9qeW5nQ3poVTQ/view?usp=sharing |url-status=live }}</ref> [[LeNet]]-5 (1998), a 7-level CNN by Yann LeCun et al., that classifies digits, was applied by several banks to recognize hand-written numbers on checks digitized in 32×32 pixel images.<ref name="lecun98">{{cite journal |last=LeCun |first=Yann |author2=Léon Bottou |author3=Yoshua Bengio |author4=Patrick Haffner |year=1998 |title=Gradient-based learning applied to document recognition |url=http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf |journal=Proceedings of the IEEE |volume=86 |issue=11 |pages=2278–2324 |citeseerx=10.1.1.32.9552 |doi=10.1109/5.726791 |s2cid=14542261 |access-date=7 October 2016 |archive-date=30 October 2023 |archive-url=https://web.archive.org/web/20231030100650/http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf |url-status=dead }}</ref>

From 1988 onward,<ref name="Qian1988">Qian, Ning, and Terrence J. Sejnowski. "Predicting the secondary structure of globular proteins using neural network models." ''Journal of molecular biology'' 202, no. 4 (1988): 865–884.</ref><ref name="Bohr1988">Bohr, Henrik, Jakob Bohr, Søren Brunak, Rodney MJ Cotterill, Benny Lautrup, Leif Nørskov, Ole H. Olsen, and Steffen B. Petersen. "Protein secondary structure and homology by neural networks The α-helices in rhodopsin." ''FEBS letters'' 241, (1988): 223–228</ref> the use of neural networks transformed the field of [[protein structure prediction]], in particular when the first cascading networks were trained on ''profiles'' (matrices) produced by multiple [[sequence alignment]]s.<ref name="Rost1993">Rost, Burkhard, and Chris Sander. "Prediction of protein secondary structure at better than 70% accuracy." ''Journal of molecular biology'' 232, no. 2 (1993): 584–599.</ref>