Editing Bernoulli distribution (section)

==Entropy and Fisher's Information==

===Entropy===
Entropy is a measure of uncertainty or randomness in a probability distribution. For a Bernoulli random variable <math>X</math> with success probability <math>p</math> and failure probability <math>q = 1 - p</math>, the entropy <math>H(X)</math> is defined as:

:<math>\begin{align}
H(X) &= \mathbb{E}_p \ln (\frac{1}{P(X)}) = - [P(X = 0) \ln P(X = 0) + P(X = 1) \ln P(X = 1)] \\
H(X) &= - (q \ln q + p \ln p) , \quad  q = P(X = 0),  p = P(X = 1)
\end{align}</math>

The entropy is maximized when <math>p = 0.5</math>, indicating the highest level of uncertainty when both outcomes are equally likely. The entropy is zero when <math>p = 0</math> or <math>p = 1</math>, where one outcome is certain.

===Fisher's Information===
Fisher information measures the amount of information that an observable random variable <math>X</math> carries about an unknown parameter <math>p</math> upon which the probability of <math>X</math> depends. For the Bernoulli distribution, the Fisher information with respect to the parameter <math>p</math> is given by:

:<math>\begin{align}
I(p) = \frac{1}{pq}
\end{align}</math>

'''Proof:'''

*The '''Likelihood Function''' for a Bernoulli random variable<math>X</math> is:

:<math>\begin{align}   
L(p; X) = p^X (1 - p)^{1 - X}
\end{align}</math>
This represents the probability of observing <math>X</math> given the parameter <math>p</math>.

*The '''Log-Likelihood Function''' is:

:<math>\begin{align}   
\ln L(p; X) = X \ln p + (1 - X) \ln (1 - p)
\end{align}</math>

*The Score Function (the first derivative of the log-likelihood w.r.t. <math>p</math> is:

:<math>\begin{align}   
\frac{\partial}{\partial p} \ln L(p; X) = \frac{X}{p} - \frac{1 - X}{1 - p}
 \end{align}</math>

*The second derivative of the log-likelihood function is:

:<math>\begin{align}   
\frac{\partial^2}{\partial p^2} \ln L(p; X) = -\frac{X}{p^2} - \frac{1 - X}{(1 - p)^2}
\end{align}</math>

*'''Fisher information''' is calculated as the negative expected value of the second derivative of the log-likelihood:

:<math>\begin{align}   
I(p) = -E\left[\frac{\partial^2}{\partial p^2} \ln L(p; X)\right] = -\left(-\frac{p}{p^2} - \frac{1 - p}{(1 - p)^2}\right) = \frac{1}{p(1-p)} = \frac{1}{pq}
\end{align}</math>

It is maximized when <math>p = 0.5</math>, reflecting maximum uncertainty and thus maximum information about the parameter <math>p</math>.