Editing Instantaneously trained neural networks

'''Instantaneously trained neural networks''' are [[Feedforward neural network|feedforward artificial neural networks]] that create a new hidden neuron node for each novel training sample. The weights to this hidden neuron separate out not only this training sample but others that are near it, thus providing generalization.<ref name="kak93">Kak, S. On training feedforward neural networks. Pramana, vol. 40, pp. 35-42, 1993 [https://link.springer.com/article/10.1007/BF02898040] </ref><ref>Kak, S. [https://www.sciencedirect.com/science/article/pii/0167865594900620 New algorithms for training feedforward neural networks]. Pattern Recognition Letters 15: 295-298, 1994.</ref> This separation is done using the nearest hyperplane that can be written down instantaneously. In the two most important implementations the neighborhood of generalization either varies with the training sample (CC1 network) or remains constant (CC4 network). These networks use [[unary coding]] for an effective representation of the data sets.<ref>Kak, S. [https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.3290&rep=rep1&type=pdf On generalization by neural networks], Information Sciences 111: 293-302, 1998.</ref>

This type of network was first proposed in a 1993 paper of [[Subhash Kak]].<ref name="kak93"/> Since then, instantaneously trained neural networks have been proposed as models of short term [[learning]] and used in [[web search]], and financial [[time series prediction]] applications.<ref>Kak, S. Faster web search and prediction using instantaneously trained neural networks. IEEE Intelligent Systems 14: 79-82, November/December 1999.</ref> They have also been used in instant [[document classification|classification of documents]]<ref>Zhang, Z. et al., [https://link.springer.com/chapter/10.1007/11427445_37 TextCC: New feedforward neural network for classifying documents instantly]. Advances in Neural Networks ISNN 2005. [[Lecture Notes in Computer Science]] 3497: 232-237, 2005.</ref> and for [[deep learning]] and [[data mining]].<ref>Zhang, Z. et al., Document Classification Via TextCC Based on Stereographic Projection and for deep learning, International Conference on Machine Learning and Cybernetics, Dalin, 2006</ref><ref>Schmidhuber, J. Deep Learning in Neural Networks: An Overview, arXiv:1404.7828, 2014 https://arxiv.org/abs/1404.7828</ref>

As in other neural networks, their normal use is as software, but they have also been implemented in hardware using FPGAs<ref>Zhu, J. and G. Milne, [https://link.springer.com/chapter/10.1007/3-540-44614-1_29 Implementing Kak Neural Networks on a Reconfigurable Computing Platform], Lecture Notes in Computer Science Volume 1896: 260-269, 2000.</ref> and by [[optical neural network|optical implementation]].<ref>Shortt, A., J.G. Keating, L. Moulinier, C.N. Pannell, [http://eprints.maynoothuniversity.ie/8663/1/JK-Optical-2005.pdf Optical implementation of the Kak neural network], Information Sciences 171: 273-287, 2005.</ref>

==CC4 network==
In the CC4 network, which is a three-stage network, the number of input nodes is one more than the size of the training vector, with the extra node serving as the biasing node whose input is always 1. For binary input vectors, the weights from the input nodes to the hidden neuron (say of index j) corresponding to the trained vector is given by the following formula:
:<math id="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.86.3290&rep=rep1&type=pdf">w_{ij} = \begin{cases}
   -1, & \mbox{for } x_i = 0\\
                         +1, & \mbox{for } x_i = 1\\
  r-s+1, & \mbox{for } i = n+1
\end{cases}</math>

where <math>r </math> is the radius of generalization and <math>s </math> is the [[Hamming weight]] (the number of 1s) of the binary sequence. From the hidden layer to the output layer the weights are 1 or -1 depending on whether the vector belongs to a given output class or not. The neurons in the hidden and output layers output 1 if the weighted sum to the input is 0 or positive and 0, if the weighted sum to the input is negative:

:<math>y = \left\{ \begin{matrix} 1 & \mbox{if } \sum x_i \ge 0\\ 0 & \mbox{if } \sum x_i< 0\end{matrix} \right.</math>

==Other networks==
The CC4 network has also been modified to include non-binary input with varying radii of generalization so that it effectively provides a CC1 implementation.<ref>Tang, K.W. and Kak, S. [https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.444.303&rep=rep1&type=pdf Fast classification networks for signal processing]. ''Circuits, Systems, Signal Processing'' 21, 2002, pp. 207-224.</ref>

In feedback networks the Willshaw network as well as the [[Hopfield network]] are able to learn instantaneously.

==References==
{{Reflist}}

[[Category:Learning]]
[[Category:Artificial neural networks]]