Editing Hopfield network (section)

=== Discrete variables ===
A simple example<ref name=":1" /> of the modern Hopfield network can be written in terms of [[binary data|binary variables]] <math>V_i</math> that represent the active <math>V_i=+1</math> and inactive <math>V_i=-1</math> state of the model neuron <math>i</math>.<math display="block" id="DAM_Energy">E = - \sum\limits_{\mu = 1}^{N_\text{mem}} F\Big(\sum\limits_{i=1}^{N_f}\xi_{\mu i} V_i\Big)</math>In this formula the weights <math display="inline">\xi_{\mu i}</math> represent the matrix of memory vectors (index <math>\mu = 1...N_\text{mem}</math> enumerates different memories, and index <math>i=1...N_f</math> enumerates the content of each memory corresponding to the <math>i</math>-th feature neuron), and the function <math>F(x)</math> is a rapidly growing non-linear function. The update rule for individual neurons (in the asynchronous case) can be written in the following form <math display="block" id="DAM_update_rule">V^{(t+1)}_i = Sign\bigg[ \sum\limits_{\mu=1}^{N_\text{mem}} \bigg(F\Big(\xi_{\mu i} + \sum\limits_{j\neq i}\xi_{\mu j} V^{(t)}_j\Big) - F\Big(-\xi_{\mu i} + \sum\limits_{j\neq i}\xi_{\mu j} V^{(t)}_j\Big) \bigg)\bigg]</math>which states that in order to calculate the updated state of the <math display="inline">i</math>-th neuron the network compares two energies: the energy of the network with the <math>i</math>-th neuron in the ON state and the energy of the network with the <math>i</math>-th neuron in the OFF state, given the states of the remaining neuron. The updated state of the <math>i</math>-th neuron selects the state that has the lowest of the two energies.<ref name=":1" />

In the limiting case when the non-linear energy function is quadratic <math>F(x) = x^2</math> these equations reduce to the familiar energy function and the update rule for the classical binary Hopfield Network.<ref name="Hopfield1982" />

The memory storage capacity of these networks can be calculated for random binary patterns. For the power energy function <math>F(x)=x^n</math> the maximal number of memories that can be stored and retrieved from this network without errors is given by<ref name=":1" /><math display="block">N^{max}_{\text{mem}}\approx \frac{1}{2 (2n-3)!!} \frac{N_f^{n-1}}{\ln(N_f)}</math>For an exponential energy function <math display="inline">F(x)=e^x</math> the memory storage capacity is exponential in the number of feature neurons<ref name=":2" /><math display="block">N^{max}_{\text{mem}}\approx 2^{N_f/2}</math>
[[File:Modern Hopfield Network.png|thumb|632x632px|Fig. 1: An example of a continuous modern Hopfield network with <math display="inline">N_f=5</math>  feature neurons and <math>N_\text{mem}=11</math> memory (hidden) neurons with symmetric synaptic connections between them.]]