Editing Fisher's exact test (section)

==Controversies==
Fisher's test gives exact ''p''-values, but some authors have argued that it is conservative, i.e. that its actual rejection rate is below the nominal significance level.<ref name="campbell2007" /><ref name="Liddell-1976">{{Cite journal
 | doi = 10.2307/2988087
 | last = Liddell
 | first = Douglas
 | year = 1976
 | title = Practical tests of 2×2 contingency tables
 | journal = The Statistician
 | volume = 25
 | issue = 4
 | pages = 295–304
 | jstor = 2988087
}}</ref><ref name="Berkson1978">{{Cite journal
 | last = Berkson
 | first = Joseph
 | year = 1978
 | title = In dispraise of the exact test
 | journal = Journal of Statistical Planning and Inference
 | volume = 2
 | pages = 27–42
 | doi = 10.1016/0378-3758(78)90019-8
}}</ref><ref name="DAgostino1988">{{Cite journal
 | doi = 10.2307/2685002
 | author1=D'Agostino, R. B. |author2=Chase, W. |author3= Belanger, A. |name-list-style=amp | year = 1988
 | title = The appropriateness of some common procedures for testing equality of two independent binomial proportions
 | journal = The American Statistician
 | volume = 42
 | issue = 3
 | pages = 198–202
 | jstor = 2685002
}}</ref> The apparent contradiction stems from the combination of a discrete statistic with fixed significance levels.<ref name="Yates1984">{{Cite journal
 | doi = 10.2307/2981577
 | author = Yates, F.
 | year = 1984
 | title = Tests of significance for 2 × 2 contingency tables (with discussion)
 | journal = Journal of the Royal Statistical Society, Series A
 | volume = 147
 | issue = 3
 | pages = 426–463
 | jstor = 2981577
| s2cid = 15760519
 }}</ref><ref name="Little1989">{{Cite journal
 | doi = 10.2307/2685390
 | author = Little, Roderick J. A.
 | year = 1989
 | title = Testing the equality of two independent binomial proportions
 | journal = The American Statistician
 | volume = 43
 | issue = 4
 | pages = 283–288
 | jstor = 2685390
}}</ref> Consider the following proposal for a significance test at the 5%-level: reject the null hypothesis for each table to which Fisher's test assigns a ''p''-value equal to or smaller than 5%. Because the set of all tables is discrete, there may not be a table for which equality is achieved. If <math>\alpha_e</math> is the largest ''p''-value smaller than 5% which can actually occur for some table, then the proposed test effectively tests at the <math>\alpha_e</math>-level. For small sample sizes, <math>\alpha_e</math> might be significantly lower than 5%.<ref name="Liddell-1976" /><ref name="Berkson1978" /><ref name="DAgostino1988" /> While this effect occurs for any discrete statistic (not just in contingency tables, or for Fisher's test), it has been argued that the problem is compounded by the fact that Fisher's test conditions on the marginals.<ref>{{cite web |first1=Cyrus R. |last1=Mehta |first2=Pralay |last2=Senchaudhuri |date=4 September 2003 |url=https://www.statsols.com/hubfs/Resources_/Comparing-Two-Binomials.pdf |title=Conditional versus unconditional exact tests for comparing two binomials |access-date=20 November 2009}}</ref> To avoid the problem, many authors discourage the use of fixed significance levels when dealing with discrete problems.<ref name="Yates1984" /><ref name="Little1989" />

The decision to condition on the margins of the table is also controversial.<ref name="Barnard1945a">
{{Cite journal
|doi=10.1038/156177a0
|author=Barnard, G.A.
|year=1945
|title=A new test for 2×2 tables
|journal=Nature
|volume=156
|page=177
|issue=3954
|bibcode=1945Natur.156..177B
|doi-access=free
}}</ref><ref name="NatureDiscussion">
{{Cite journal
|author=Fisher
|year=1945
|journal=Nature
|volume=156
|page=388
|doi=10.1038/156388a0
|title=A New Test for 2 × 2 Tables
|issue=3961
|bibcode=1945Natur.156..388F
|s2cid=4113420
|doi-access=free
}};
{{Cite journal
|author=Barnard, G.A.
|year=1945
|journal=Nature
|volume=156
|pages=783–784
|title=A new test for 2×2 tables
|doi=10.1038/156783b0
|issue=3974
|bibcode=1945Natur.156..783B
|s2cid=4099311
}}
</ref> The ''p''-values derived from Fisher's test come from the distribution that conditions on the margin totals. In this sense, the test is exact only for the conditional distribution and not the original table where the margin totals may change from experiment to experiment. It is possible to obtain an exact ''p''-value for the 2×2 table when the margins are not held fixed. [[Barnard's exact test|Barnard's test]], for example, allows for random margins. However, some authors<ref name="Yates1984" /><ref name="Little1989" /><ref name="NatureDiscussion" /> (including, later, Barnard himself)<ref name="Yates1984" /> have criticized Barnard's test based on this property. They argue that the marginal success total is an (almost<ref name="Little1989" />) [[ancillary statistic]], containing (almost) no information about the tested property.

The act of conditioning on the marginal success rate from a 2×2 table can be shown to ignore some information in the data about the unknown odds ratio.<ref name="Choi2015">
{{Cite journal
|vauthors=Choi L, Blume JD, Dupont WD
|year=2015
|title=Elucidating the foundations of statistical inference with 2×2 tables
|journal=PLOS ONE
|volume=10
|issue=4
|pages=e0121263
|doi=10.1371/journal.pone.0121263
|pmc=4388855
|pmid=25849515
|bibcode=2015PLoSO..1021263C
|doi-access=free
}}</ref> The argument that the marginal totals are (almost) ancillary implies that the appropriate likelihood function for making inferences about this odds ratio should be conditioned on the marginal success rate.<ref name="Choi2015" /> Whether this lost information is important for inferential purposes is the essence of the controversy.<ref name="Choi2015" />