Editing Statistical significance (section)

==Role in statistical hypothesis testing==
{{Main|Statistical hypothesis testing|Null hypothesis|Alternative hypothesis|p-value|Type I and type II errors}}

[[File:NormalDist1.96.svg|250px|thumb|In a [[one- and two-tailed tests|two-tailed test]], the rejection region for a significance level of {{math|''α'' {{=}} 0.05}} is partitioned to both ends of the [[sampling distribution]] and makes up 5% of the area under the curve (white areas).]]

Statistical significance plays a pivotal role in statistical hypothesis testing. It is used to determine whether the [[null hypothesis]] should be rejected or retained. The null hypothesis is the hypothesis that no effect exists in the phenomenon being studied.<ref name=Meier>{{cite book |last1 = Meier|first1 = Kenneth J.|last2=Brudney |first2=Jeffrey L. |last3=Bohte|first3=John |title = Applied Statistics for Public and Nonprofit Administration| edition=3rd |publisher = Cengage Learning |location = Boston, MA | year = 2011 |isbn =978-1-111-34280-7 |pages=189–209}}</ref> For the null hypothesis to be rejected, an observed result has to be statistically significant, i.e. the observed ''p''-value is less than the pre-specified significance level <math>\alpha</math>.

To determine whether a result is statistically significant, a researcher calculates a ''p''-value, which is the probability of observing an effect of the same magnitude or more extreme given that the null hypothesis is true.<ref name=":0" /><ref name="Devore"/> The null hypothesis is rejected if the ''p''-value is less than (or equal to) a predetermined level, <math>\alpha</math>.  <math>\alpha</math> is also called the ''significance level'', and is the probability of rejecting the null hypothesis given that it is true (a [[Type I error#Type I error|type I error]]). It is usually set at or below 5%.

For example, when <math>\alpha</math> is set to 5%, the [[conditional probability]] of a [[Type I error#Type I error|type I error]], ''given that the null hypothesis is true'', is 5%,<ref name="Healy2009">{{cite book |last1 = Healy|first1 = Joseph F. |title = The Essentials of Statistics: A Tool for Social Research | edition=2nd |publisher = Cengage Learning |location = Belmont, CA | year = 2009 | isbn =978-0-495-60143-2 |pages=177–205}}</ref> and a statistically significant result is one where the observed ''p''-value is less than (or equal to) 5%.<ref name="Healy2006">{{cite book |last1 =  McKillup|first1 = Steve |title = Statistics Explained: An Introductory Guide for Life Scientists |url =  https://archive.org/details/statisticsexplai0000mcki|url-access =  registration| edition=1st |publisher =  Cambridge University Press  |location = Cambridge, UK | year = 2006 |isbn =978-0-521-54316-3 |pages=[https://archive.org/details/statisticsexplai0000mcki/page/32 32–38]}}</ref> When drawing data from a sample, this means that the rejection region comprises 5% of the [[sampling distribution]].<ref name=Heath>{{cite book |last1 = Health|first1 = David |title = An Introduction To Experimental Design And Statistics For Biology| edition=1st |publisher = CRC press |location = Boston, MA | year = 1995 |isbn =978-1-85728-132-3 |pages=123–154}}</ref> These 5% can be allocated to one side of the sampling distribution, as in a [[one-tailed test]], or partitioned to both sides of the distribution, as in a [[two-tailed test]], with each tail (or rejection region) containing 2.5% of the distribution.

The use of a one-tailed test is dependent on whether the [[research question]] or [[alternative hypothesis]] specifies a direction such as whether a group of objects is ''heavier'' or the performance of students on an assessment is ''better''.<ref name="Myers et al-p65">{{cite book |last1 = Myers | first1 = Jerome L. | last2 = Well | first2 = Arnold D. | last3 = Lorch | first3 = Robert F. Jr. | chapter = Developing fundamentals of hypothesis testing using the binomial distribution | title = Research design and statistical analysis | edition=3rd | publisher = Routledge |location = New York, NY | year = 2010 | isbn = 978-0-8058-6431-1  | pages=65–90}}</ref> A two-tailed test may still be used but it will be less [[Statistical power|powerful]] than a one-tailed test, because the rejection region for a one-tailed test is concentrated on one end of the null distribution and is twice the size (5% vs. 2.5%) of each rejection region for a two-tailed test. As a result, the null hypothesis can be rejected with a less extreme result if a one-tailed test was used.<ref name="Hinton 2014">{{cite book |last1 = Hinton | first1 = Perry R. | chapter = Significance, error, and power | title = Statistics explained | edition=3rd | publisher = Routledge |location = New York, NY | year = 2010 | isbn = 978-1-84872-312-2  | pages=79–90}}</ref> The one-tailed test is only more powerful than a two-tailed test if the specified direction of the alternative hypothesis is correct. If it is wrong, however, then the one-tailed test has no power.

=== Significance thresholds in specific fields ===
{{Further|Standard deviation|Normal distribution}}
In specific fields such as [[particle physics]] and [[manufacturing]], statistical significance is often expressed in multiples of the [[standard deviation]] or sigma (''σ'') of a [[normal distribution]], with significance thresholds set at a much stricter level (for example 5''σ'').<ref name=Vaughan>{{cite book |last1 = Vaughan|first1 = Simon |title = Scientific Inference: Learning from Data | edition=1st |publisher = Cambridge University Press|location = Cambridge, UK | year = 2013 |isbn = 978-1-107-02482-3 |pages=146–152}}</ref><ref name=Bracken>{{cite book |last1 = Bracken|first1 = Michael B.|title = Risk, Chance, and Causation: Investigating the Origins and Treatment of Disease |url = https://archive.org/details/riskchancecausat0000brac|url-access = registration| edition=1st |publisher =Yale University Press|location = New Haven, CT | year = 2013 |isbn = 978-0-300-18884-4 |pages=[https://archive.org/details/riskchancecausat0000brac/page/260 260–276]}}</ref> For instance, the certainty of the [[Higgs boson]] particle's existence was based on the 5''σ'' criterion, which corresponds to a ''p''-value of about 1 in 3.5 million.<ref name="Bracken"/><ref name=franklin>{{cite book |last1 = Franklin|first1 = Allan|chapter= Prologue: The rise of the sigmas |title=Shifting Standards: Experiments in Particle Physics in the Twentieth Century|edition=1st |publisher = University of Pittsburgh Press|location = Pittsburgh, PA | year = 2013 |isbn = 978-0-8229-4430-0 |pages=Ii–Iii}}</ref>

In other fields of scientific research such as [[Genome-wide association study|genome-wide association studies]], significance levels as low as {{val|5|e=-8}} are not uncommon<ref name="Clarke et al">{{cite journal | last1 =  Clarke | first1 = GM | last2 = Anderson | first2 = CA | last3 = Pettersson | first3 = FH | last4 = Cardon | first4 = LR | last5 = Morris | first5 = AP | last6 = Zondervan | first6= KT | title = Basic statistical analysis in genetic case-control studies | journal = Nature Protocols | volume = 6 | issue = 2 | pages = 121–33 | date = February 6, 2011 | pmid = 21293453 | doi = 10.1038/nprot.2010.182 | pmc=3154648}}</ref><ref name="Barsh et al">{{cite journal | last1 = Barsh | first1 = GS | last2 = Copenhaver | first2 = GP | last3 = Gibson | first3 = G | last4 = Williams | first4 = SM | title = Guidelines for Genome-Wide Association Studies | journal = PLOS Genetics | volume = 8 | issue = 7 | pages = e1002812 | date = July 5, 2012 | pmid = 22792080 | doi = 10.1371/journal.pgen.1002812 | pmc=3390399 | doi-access = free }}</ref>—as the number of tests performed is extremely large.