Editing Statistical significance (section)

{{Short description | Concept in inferential statistics}}
In [[statistical hypothesis testing]],<ref name=Sirkin>{{cite book |last1 = Sirkin|first1 = R. Mark |chapter= Two-sample t tests | title = Statistics for the Social Sciences| edition=3rd |publisher = SAGE Publications, Inc | location = Thousand Oaks, CA | year = 2005 |isbn =978-1-4129-0546-6 |pages=271–316}}</ref><ref name=Borror>{{cite book |last1 = Borror |first1 = Connie M. |chapter= Statistical decision making |title = The Certified Quality Engineer Handbook | edition=3rd |publisher = ASQ Quality Press |location = Milwaukee, WI | year = 2009 |isbn =978-0-87389-745-7 |pages=418–472}}</ref> a result has '''statistical significance''' when a result at least as "extreme" would be very infrequent if the [[null hypothesis]] were true.<ref name="Myers et al-p65" /> More precisely, a study's defined '''significance level''', denoted by <math>\alpha</math>, is the probability of the study rejecting the null hypothesis, given that the null hypothesis is true;<ref name="Dalgaard">{{cite book |last=Dalgaard |first=Peter |title=Introductory Statistics with R |location=New York |publisher=Springer |year=2008 |pages=155–56 |isbn=978-0-387-79053-4 |doi=10.1007/978-0-387-79054-1_9 |chapter=Power and the computation of sample size |series=Statistics and Computing }}</ref> and the [[p-value|''p''-value]] of a result, ''<math>p</math>'', is the probability of obtaining a result at least as extreme, given that the null hypothesis is true.<ref name=":0">{{Cite web|url=http://www.dartmouth.edu/~matc/X10/Show.htm|title=Statistical Hypothesis Testing|website=www.dartmouth.edu|access-date=2019-11-11|archive-date=2020-08-02|archive-url=https://web.archive.org/web/20200802050104/http://www.dartmouth.edu/~matc/X10/Show.htm|url-status=dead}}</ref> The result is said to be '''''statistically significant''''', by the standards of the study, when <math>p \le \alpha</math>.<ref
name="Johnson">{{cite journal |last= Johnson| first= Valen E. |date= October 9, 2013 |title= Revised standards for statistical evidence |journal= Proceedings of the National Academy of Sciences |doi= 10.1073/pnas.1313476110 | pmid= 24218581 |volume=110 | issue= 48 |pages=19313–19317|pmc=3845140 | bibcode= 2013PNAS..11019313J | doi-access= free }}</ref><ref name="Redmond and Colton">{{cite book | last1 = Redmond|first1 = Carol | last2 = Colton | first2 = Theodore | chapter = Clinical significance versus statistical significance | title = Biostatistics in Clinical Trials | series = Wiley Reference Series in Biostatistics | edition=3rd |publisher = John Wiley & Sons Ltd  |location = West Sussex, United Kingdom | year = 2001 |isbn = 978-0-471-82211-0 |pages = 35–36}}</ref><ref name="Cumming-p27">{{cite book | last1 = Cumming|first1 = Geoff | title = Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis |publisher = Routledge |location = New York, USA | year = 2012|pages = 27–28}}</ref><ref name="Krzywinski and Altman">{{cite journal |last1= Krzywinski |first1= Martin |last2= Altman |first2= Naomi |date= 30 October 2013   |title= Points of significance: Significance, P values and t-tests |journal= Nature Methods |volume= 10 |issue= 11 |pages= 1041–1042 |doi= 10.1038/nmeth.2698 |pmid= 24344377 |doi-access= free }}</ref><ref name="Sham and Purcell">{{cite journal |last1= Sham |first1= Pak C.|last2= Purcell |first2= Shaun M |date= 17 April 2014 |title= Statistical power and significance testing in large-scale genetic studies |journal= Nature Reviews Genetics |volume= 15 |issue= 5 |pages= 335–346 |doi= 10.1038/nrg3706 |pmid= 24739678|s2cid= 10961123}}</ref><ref name="Altman">{{cite book | last1 = Altman|first1 = Douglas G. | title = Practical Statistics for Medical Research | url = https://archive.org/details/isbn_9780412276309| url-access = registration|publisher = Chapman & Hall/CRC |location = New York, USA | year = 1999 | isbn = 978-0-412-27630-9  |pages = [https://archive.org/details/isbn_9780412276309/page/167 167]}}</ref><ref name=Devore>{{cite book |last1 = Devore|first1 = Jay L.|title = Probability and Statistics for Engineering and the Sciences| edition=8th |publisher = Cengage Learning |location = Boston, MA | year = 2011 |isbn =978-0-538-73352-6 |pages=300–344}}</ref> The significance level for a study is chosen before data collection, and is typically set to 5%<ref name="Salkind">{{cite encyclopedia|year=2007|title=Significance level|encyclopedia=Encyclopedia of Measurement and Statistics|publisher=SAGE Publications|location=Thousand Oaks, CA|editor-last1=Salkind|editor-first1=Neil J.|volume=3|pages=889–891|isbn=978-1-4129-1611-0|last1=Craparo|first1=Robert M.}}</ref> or much lower—depending on the field of study.<ref name="Sproull">{{cite book|title=Handbook of Research Methods: A Guide for Practitioners and Students in the Social Science|last1=Sproull|first1=Natalie L.|publisher=Scarecrow Press, Inc.|year=2002|isbn=978-0-8108-4486-5|edition=2nd|location=Lanham, MD|pages=[https://archive.org/details/handbookofresear00spro/page/49 49–64]|chapter=Hypothesis testing|chapter-url=https://archive.org/details/handbookofresear00spro/page/49}}</ref>

In any [[experiment]] or [[Observational study|observation]] that involves drawing a [[Sampling (statistics)|sample]] from a [[Statistical population|population]], there is always the possibility that an observed effect would have occurred due to [[sampling error]] alone.<ref name=Babbie2>{{cite book |last1 = Babbie|first1 = Earl R. |chapter= The logic of sampling | title = The Practice of Social Research| edition=13th |publisher = Cengage Learning |location = Belmont, CA | year = 2013|isbn =978-1-133-04979-1 |pages=185–226}}</ref><ref name=Faherty>{{cite book |last1 = Faherty | first1 = Vincent | chapter= Probability and statistical significance | title = Compassionate Statistics: Applied Quantitative Analysis for Social Services (With exercises and instructions in SPSS) | edition=1st |publisher = SAGE Publications, Inc |location = Thousand Oaks, CA | year = 2008 |isbn =978-1-4129-3982-9 |pages=127–138}}</ref> But if the ''p''-value of an observed effect is less than (or equal to) the significance level, an investigator may conclude that the effect reflects the characteristics of the whole population,<ref name=Sirkin/> thereby rejecting the null hypothesis.<ref name=McKillup>{{cite book |last1 = McKillup|first1 = Steve |title = Statistics Explained: An Introductory Guide for Life Scientists|chapter-url = https://archive.org/details/statisticsexplai0000mcki|chapter-url-access = registration| edition = 1st | publisher = Cambridge University Press|location = Cambridge, United Kingdom | year = 2006 |chapter=Probability helps you make a decision about your results | isbn = 978-0-521-54316-3 |pages=[https://archive.org/details/statisticsexplai0000mcki/page/44 44–56]}}</ref>

This technique for testing the statistical significance of results was developed in the early 20th century. The term ''significance'' does not imply importance here, and the term ''statistical significance'' is not the same as research significance, theoretical significance, or practical significance.<ref name=Sirkin /><ref name=Borror /><ref name="Myers et al-p124">{{cite book |last1 = Myers|first1 = Jerome L. |last2 = Well|first2 = Arnold D. |last3 = Lorch |first3 = Robert F. Jr. |chapter= The t distribution and its applications |title = Research Design and Statistical Analysis | edition=3rd |publisher = Routledge |location = New York, NY | year = 2010 |isbn =978-0-8058-6431-1 |pages=124–153}}</ref><ref name=":1">{{Cite web|url=http://www.stat.ualberta.ca/~hooper/teaching/misc/Pvalue.pdf|title=What is P-value?|last=Hooper|first=Peter|website=University of Alberta, Department of Mathematical and Statistical Sciences|access-date=November 10, 2019|archive-date=March 31, 2020|archive-url=https://web.archive.org/web/20200331153714/http://www.stat.ualberta.ca/~hooper/teaching/misc/Pvalue.pdf|url-status=dead}}</ref> For example, the term [[clinical significance]] refers to the practical importance of a treatment effect.<ref>{{Cite journal|last=Leung|first=W.-C.|date=2001-03-01|title=Balancing statistical and clinical significance in evaluating treatment effects|url= |journal=Postgraduate Medical Journal|language=en|volume=77|issue=905|pages=201–204|doi=10.1136/pmj.77.905.201|issn=0032-5473|pmid=11222834|pmc=1741942}}</ref>