Editing Data dredging (section)

==Remedies==
While looking for patterns in data is legitimate, applying a statistical test of significance or [[hypothesis test]] to the same data until a pattern emerges is prone to abuse. One way to construct hypotheses while avoiding data dredging is to conduct randomized [[out-of-sample test]]s. The researcher collects a data set, then randomly partitions it into two subsets, A and B. Only one subset—say, subset A—is examined for creating hypotheses. Once a hypothesis is formulated, it must be tested on subset B, which was not used to construct the hypothesis. Only where B also supports such a hypothesis is it reasonable to believe the hypothesis might be valid. (This is a simple type of [[cross-validation (statistics)|cross-validation]] and is often termed training-test or split-half validation.)

Another remedy for data dredging is to record the number of all significance tests conducted during the study and simply divide one's criterion for significance (alpha) by this number; this is the [[Bonferroni correction]].  However, this is a very conservative metric. A family-wise alpha of 0.05, divided in this way by 1,000 to account for 1,000 significance tests, yields a very stringent per-hypothesis alpha of 0.00005. Methods particularly useful in analysis of variance, and in constructing simultaneous confidence bands for regressions involving basis functions are [[Scheffé's method]] and, if the researcher has in mind only [[Pairwise comparison (psychology)|pairwise comparison]]s, the [[Tukey range test|Tukey method]]. To avoid the extreme conservativeness of the Bonferroni correction, more sophisticated selective inference methods are available.<ref name="TaylorTibshirani2015">
{{Cite journal
|author1=Taylor, J. |author2=Tibshirani, R.
|title = Statistical learning and selective inference
|journal = Proceedings of the National Academy of Sciences
|doi = 10.1073/pnas.1507583112
|year = 2015 |volume=112 |issue=25 |pages=7629–7634 
 |doi-access=free|pmid=26100887
|pmc=4485109|bibcode=2015PNAS..112.7629T
}}
</ref> The most common selective inference method is the use of Benjamini and Hochberg's [[false discovery rate]] controlling procedure: it is a less conservative approach that has become a popular method for control of multiple hypothesis tests. 

When neither approach is practical, one can make a clear distinction between data analyses that are [[Statistical hypothesis testing|confirmatory]] and analyses that are [[exploratory data analysis|exploratory]]. Statistical inference is appropriate only for the former.<ref name="BerkBrownZhao" />

Ultimately, the statistical significance of a test and the statistical confidence of a finding are joint properties of data and the method used to examine the data. Thus, if someone says that a certain event has probability of 20% ± 2% 19 times out of 20, this means that if the probability of the event is estimated ''by the same method'' used to obtain the 20% estimate, the result is between 18% and 22% with probability 0.95. No claim of statistical significance can be made by only looking, without due regard to the method used to assess the data.

Academic journals increasingly shift to the [[registered report]] format, which aims to counteract very serious issues such as data dredging and [[HARKing|{{abbr|HARKing|Hypothesizing After Results are Known}}]], which have made theory-testing research very unreliable. For example, ''[[Nature Human Behaviour]]'' has adopted the registered report format, as it "shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them".<ref>{{cite journal|title=Promoting reproducibility with registered reports|date=10 January 2017|journal=Nature Human Behaviour|volume=1|issue=1|pages=0034|doi=10.1038/s41562-016-0034|s2cid=28976450|doi-access=free}}</ref> The ''[[European Journal of Personality]]'' defines this format as follows: "In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes."<ref>{{cite web|url=https://www.ejp-blog.com/blog/2017/2/3/streamlined-review-and-registered-reports-coming-soon|title=Streamlined review and registered reports soon to be official at EJP|website=ejp-blog.com|date=6 February 2018 }}</ref>

Methods and results can also be made publicly available, as in the [[open science]] approach, making it yet more difficult for data dredging to take place.<ref>{{cite journal |last1=Vyse |first1=Stuart |title=P-Hacker Confessions: Daryl Bem and Me |journal=[[Skeptical Inquirer]] |date=2017 |volume=41 |issue=5 |pages=25–27 |url=https://www.csicop.org/specialarticles/show/p-hacker_confessions_daryl_bem_and_me |access-date=5 August 2018|archive-url=https://web.archive.org/web/20180805142806/https://www.csicop.org/specialarticles/show/p-hacker_confessions_daryl_bem_and_me |archive-date=2018-08-05 }}</ref>