Editing Feature selection (section)

=== Joint mutual information ===

In a study of different scores Brown et al.<ref name="Brown" />  recommended the [[joint mutual information]]<ref>{{cite journal |last1=Yang |first1=Howard Hua |last2=Moody |first2=John |title=Data visualization and feature selection: New algorithms for nongaussian data |journal=Advances in Neural Information Processing Systems |date=2000 |pages=687–693 |url=https://papers.nips.cc/paper/1779-data-visualization-and-feature-selection-new-algorithms-for-nongaussian-data.pdf}}</ref> as a good score for feature selection. The score tries to find the feature, that adds the most new information to the already selected features, in order to avoid redundancy. The score is formulated as follows:

:<math>
\begin{align}
JMI(f_i) &= \sum_{f_j \in S} (I(f_i;c) + I(f_i;c|f_j)) \\
         &= \sum_{f_j \in S} \bigl[ I (f_j;c) + I (f_i;c) - \bigl(I (f_i;f_j) - I (f_i;f_j|c)\bigr)\bigr]
\end{align}
</math>

The score uses the [[conditional mutual information]] and the [[mutual information]] to estimate the redundancy between the already selected features (<math> f_j \in S </math>) and the feature under investigation (<math>f_i</math>).