Editing Data dredging (section)

=== Drawing conclusions from data ===
The conventional [[statistical hypothesis testing]] procedure using [[frequentist probability]] is to formulate a research hypothesis, such as "people in higher social classes live longer", then collect relevant data. Lastly, a statistical [[significance test]] is carried out to see how likely the results are by chance alone (also called testing against the null hypothesis).

A key point in proper statistical analysis is to test a hypothesis with evidence (data) that was not used in constructing the hypothesis. This is critical because every [[data set]] contains some patterns due entirely to chance. If the hypothesis is not tested on a different data set from the same [[statistical population]], it is impossible to assess the likelihood that chance alone would produce such patterns.

For example, [[flipping a coin]] five times with a result of 2 heads and 3 tails might lead one to hypothesize that the coin favors tails by 3/5 to 2/5. If this hypothesis is then tested on the existing data set, it is confirmed, but the confirmation is meaningless. The proper procedure would have been to form in advance a hypothesis of what the tails probability is, and then throw the coin various times to see if the hypothesis is rejected or not. If three tails and two heads are observed, another hypothesis, that the tails probability is 3/5, could be formed, but it could only be tested by a new set of coin tosses. The statistical significance under the incorrect procedure is completely spurious—significance tests do not protect against data dredging.