Editing Minimum description length (section)

==Overview==

Selecting the minimum length description of the available data as the best model observes the ''principle'' identified as Occam's razor.  Prior to the advent of computer programming, generating such descriptions was the intellectual labor of scientific theorists.  It was far less formal than it has become in the computer age.  If two scientists had a theoretic disagreement, they rarely could ''formally'' apply Occam's razor to choose between their theories.  They would have different data sets and possibly different descriptive languages.  Nevertheless, science advanced as Occam's razor was an informal guide in deciding which model was best.

With the advent of formal languages and computer programming Occam's razor was mathematically defined.  Models of a given set of observations, encoded as bits of data, could be created in the form of computer programs that output that data.  Occam's razor could then ''formally'' select the shortest program, measured in bits of this ''algorithmic information'', as the best model.

To avoid confusion, note that there is nothing in the MDL principle that implies the model must be produced by a machine.  It can be entirely the product of humans.  The MDL principle applies regardless of whether the description to be run on a computer is the product of humans, machines or any combination thereof.  The MDL principle requires ''only'' that the shortest description, when executed, produce the original data set without error.

===Two-Part codes===

The distinction in computer programs between programs and literal data applies to all formal descriptions and is sometimes referred to as "''two parts''" of a description.  In statistical MDL learning, such a description is frequently called a ''two-part code''.