Editing Minimum message length (section)

==Definition==
[[Claude E. Shannon|Shannon]]'s ''[[A Mathematical Theory of Communication]]'' (1948) states that in an optimal code, the message length (in binary) of an event <math>E</math>, <math>\operatorname{length}(E)</math>, where <math>E</math> has probability <math>P(E)</math>, is given by <math>\operatorname{length}(E) = -\log_2(P(E))</math>.

[[Bayes's theorem]] states that the probability of a (variable) hypothesis <math>H</math> given fixed evidence <math>E</math> is proportional to <math>P(E|H) P(H)</math>, which, by the definition of [[conditional probability]], is equal to <math>P(H \land E)</math>. We want the model (hypothesis) with the highest such [[posterior probability]]. Suppose we encode a message which represents (describes) both model and data jointly. Since <math>\operatorname{length}(H \land E) = -\log_2(P(H \land E))</math>, the most probable model will have the shortest such message. The message breaks into two parts: <math>-\log_2(P(H \land E)) = -\log_2(P(H)) + -\log_2(P(E|H))</math>. The first part encodes the model itself. The second part contains information (e.g., values of parameters, or initial conditions, etc.) that, when processed by the model, outputs the observed data.

MML naturally and precisely trades model complexity for goodness of fit. A more complicated model takes longer to state (longer first part) but probably fits the data better (shorter second part).  So, an MML metric won't choose a complicated model unless that model pays for itself.