Editing Design of experiments (section)

==Avoiding false positives==
{{see also|Metascience}}
[[False positive]] conclusions, often resulting from the [[Publish or perish|pressure to publish]] or the author's own [[confirmation bias]], are an inherent hazard in many fields.<ref>{{Cite journal |last1=Forstmeier |first1=Wolfgang |last2=Wagenmakers |first2=Eric-Jan |last3=Parker |first3=Timothy H. |date=23 November 2016 |title=Detecting and avoiding likely false-positive findings – a practical guide |journal=Biological Reviews |language=en |volume=92 |issue=4 |pages=1941–1968 |doi=10.1111/brv.12315 |pmid=27879038 |s2cid=26793416 |issn=1464-7931|doi-access=free |hdl=11245.1/31f84a5b-4439-4a4c-a690-6e98354199f5 |hdl-access=free }}</ref>

Use of [[Double-blind|double-blind designs]] can prevent [[Bias|biases]] potentially leading to [[false positives]] in the [[data collection]] phase. When a double-blind design is used, participants are randomly assigned to experimental groups but the researcher is unaware of what participants belong to which group. Therefore, the researcher can not affect the participants' response to the intervention.<ref name=":0">{{Cite journal |last1=David |first1=Sharoon |last2=Khandhar1 |first2=Paras B. |date=July 17, 2023 |title=Double-Blind Study |url=https://www.ncbi.nlm.nih.gov/books/NBK546641/ |journal=[[StatPearls Publishing]]|pmid=31536248 }}</ref>

Experimental designs with undisclosed [[degrees of freedom]]{{Technical inline|date=August 2023}} are a problem,<ref>{{cite journal| last = Simmons| first = Joseph|author2=Leif Nelson |author3=Uri Simonsohn | title = False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant| journal = Psychological Science| volume = 22| issue = 11| pages = 1359–1366| date = November 2011| issn = 0956-7976| doi = 10.1177/0956797611417632| pmid = 22006061| doi-access = }}
</ref> in that they can lead to conscious or unconscious "[[p-hacking]]": trying multiple things until you get the desired result. It typically involves the manipulation – perhaps unconsciously – of the process of [[statistical analysis]] and the [[degrees of freedom]] until they return a figure below the [[P-value|p<.05 level]] of [[statistical significance]].<ref>{{cite news
 | url=http://www.kplu.org/post/science-trust-and-psychology-crisis
 | title=Science, Trust And Psychology in Crisis
 | work=[[KNKX|KPLU]]
 | date=2014-06-02
 | access-date=2014-06-12
 | archive-url=https://web.archive.org/web/20140714151939/http://www.kplu.org/post/science-trust-and-psychology-crisis
 | archive-date=14 July 2014
 | url-status=dead
 }}</ref><ref>{{cite news
 | url=https://psmag.com/environment/statistically-significant-studies-arent-necessarily-significant-82832
 | title=Why Statistically Significant Studies Can Be Insignificant
 | work=Pacific Standard
 | date=2014-06-04
 | access-date=2014-06-12 }}
</ref> 

P-hacking can be prevented by [[Preregistration (science)|preregistering]] researches, in which researchers have to send their data analysis plan to the journal they wish to publish their paper in before they even start their data collection, so no data manipulation is possible.<ref>{{Cite journal |last1=Nosek |first1=Brian A. |last2=Ebersole |first2=Charles R. |last3=DeHaven |first3=Alexander C. |last4=Mellor |first4=David T. |date=2018-03-13 |title=The preregistration revolution |journal=Proceedings of the National Academy of Sciences |language=en |volume=115 |issue=11 |pages=2600–2606 |doi=10.1073/pnas.1708274114 |issn=0027-8424 |pmc=5856500 |pmid=29531091 |bibcode=2018PNAS..115.2600N |doi-access=free }}</ref><ref>{{Cite web |title=Pre-Registering Studies – What Is It, How Do You Do It, and Why? |url=https://www.acf.hhs.gov/opre/blog/2022/08/pre-registering-studies-what-it-how-do-you-do-it-and-why |access-date=2023-08-29 |website=www.acf.hhs.gov |language=en}}</ref>

Another way to prevent this is taking a double-blind design to the data-analysis phase, making the study triple-blind, where the data are sent to a data-analyst unrelated to the research who scrambles up the data so there is no way to know which participants belong to before they are potentially taken away as outliers.<ref name=":0" />

Clear and complete [[documentation]] of the experimental [[methodology]] is also important in order to support [[replication of results]].<ref>{{cite news
 | url=https://www.theguardian.com/science/head-quarters/2014/jun/10/physics-envy-do-hard-sciences-hold-the-solution-to-the-replication-crisis-in-psychology
 | title=Physics envy: Do 'hard' sciences hold the solution to the replication crisis in psychology?
 | work=theguardian.com
 | author=Chris Chambers
 | date=2014-06-10
 | access-date=2014-06-12 }}
</ref>