Editing Optimal experimental design

{{Short description|Experimental design that is optimal with respect to some statistical criterion}}
{{about|optimal [[design of experiments]]|optimal design in control theory|shape optimization}}

<!-- {{Context|date=May 2012}} -->
[[Image:Theb1982.jpg|thumb|right|alt=Picture of a man taking measurements with a theodolite in a frozen environment.|[[Gustav Elfving]] developed the optimal design of experiments, and so minimized surveyors' need for  [[theodolite|theodolite measurements]] ''(pictured)'', while trapped in his tent in storm-ridden [[Greenland]].<ref name="Nord76">{{harvtxt|Nordström|1999|p=176}}</ref>]]

In the [[design of experiments]], '''optimal experimental designs''' (or '''optimum designs'''<ref>The adjective "optimum" (and not "optimal") "is the slightly older form in English and avoids the construction 'optim(um) + al´—there is no 'optimalis' in Latin" (page x in ''Optimum Experimental Designs, with SAS'', by Atkinson, Donev, and Tobias).</ref>) are a class of [[design of experiments|experimental designs]] that are [[Optimization (mathematics)|optimal]] with respect to some [[statistical theory|statistical]] [[objective function|criterion]]. The creation of this field of statistics has been credited to Danish statistician [[Kirstine Smith]].<ref name=GL2009>{{cite journal |last1=Guttorp |first1=P. |last2=Lindgren |first2=G. |title= Karl Pearson and the Scandinavian school of statistics |journal= International Statistical Review |volume=77 |year=2009 |page=64 |doi=10.1111/j.1751-5823.2009.00069.x|citeseerx=10.1.1.368.8328 |s2cid=121294724 }}</ref><ref name=KS1918>{{cite journal |last=Smith |first=Kirstine |title=On the standard deviations of adjusted and interpolated values of an observed polynomial function and its constants and the guidance they give towards a proper choice of the distribution of observations |journal=Biometrika |volume=12 |issue=1/2 |pages=1–85 |year=1918 |doi=10.2307/2331929|jstor=2331929 |url=https://zenodo.org/record/1431591 }}</ref>

In the [[design of experiments]] for [[estimation theory|estimating]] [[statistical model]]s, '''optimal designs''' allow parameters to be [[bias of an estimator|estimated without bias]] and with [[Minimum-variance unbiased estimator|minimum variance]]. A non-optimal design requires a greater number of [[replication (statistics)|experimental runs]] to [[estimation theory|estimate]] the [[parametric model|parameters]] with the same [[efficiency (statistics)|precision]] as an optimal design. In practical terms, optimal experiments can reduce the costs of experimentation.

The optimality of a design depends on the [[statistical model]] and is assessed with respect to a statistical criterion, which is related to the variance-matrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding of [[statistical theory]] and practical knowledge with [[design of experiments|designing experiments]].

==Advantages==
Optimal designs offer three advantages over sub-optimal [[Design of experiments|experimental designs]]:<ref>These three advantages (of optimal designs) are documented in the textbook by Atkinson, Donev, and Tobias.</ref>
#Optimal designs reduce the costs of experimentation by allowing [[statistical model]]s to be estimated with fewer experimental runs.
#Optimal designs can accommodate multiple types of factors, such as process, mixture, and discrete factors.
#Designs can be optimized when the design-space is constrained, for example, when the mathematical process-space contains factor-settings that are practically infeasible (e.g. due to safety concerns).

==Minimizing the variance of estimators==
Experimental designs are evaluated using statistical criteria.<ref>Such criteria are called [[objective function]]s in [[Optimization (mathematics)|optimization theory]].</ref>

It is known that the [[least squares]] estimator minimizes the [[variance]] of [[Expected value|mean]]-[[Bias of an estimator|unbiased]] [[estimators]] (under the conditions of the [[Gauss–Markov theorem]]). In the [[estimation]] theory for [[statistical model]]s with one [[real number|real]] [[parameter]], the [[reciprocal (mathematics)|reciprocal]] of the variance of an ([[statistical efficiency|"efficient"]]) estimator is called the "[[Fisher information]]" for that estimator.<ref>The [[Fisher information]] and other "[[entropy (information theory)|information]]" [[functional (mathematics)|functional]]s are fundamental concepts in [[statistical theory]].
</ref> Because of this reciprocity, '''''minimizing'' the ''variance''''' corresponds to '''''maximizing'' the ''information'''''.

When the [[statistical model]] has several [[parameter]]s, however, the [[Expected value|mean]] of the parameter-estimator is a [[Coordinate vector|vector]] and its [[covariance matrix|variance]] is a [[Matrix (mathematics)|matrix]]. The [[inverse matrix]] of the variance-matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using [[statistical theory]], statisticians compress the  information-matrix using real-valued [[summary statistics]]; being real-valued functions, these "information criteria" can be maximized.<ref>Traditionally, statisticians have evaluated estimators and designs by considering some [[summary statistics|summary statistic]] of the covariance matrix (of a [[Expected value|mean]]-[[Bias of an estimator|unbiased estimator]]), usually with positive real values (like the [[determinant]] or [[matrix trace]]). Working with positive real-numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone). <br />
For several parameters, the covariance-matrices and information-matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a [[partial order|partially]] [[ordered vector space]], under the [[Charles Loewner|Loewner]] (Löwner) order. This cone is closed under matrix-matrix addition, under matrix-inversion, and under the multiplication of positive real-numbers and matrices.

An exposition of matrix theory and the Loewner-order appears in Pukelsheim.
</ref> The traditional optimality-criteria are [[Invariant theory|invariants]] of the [[Fisher information|information]] matrix; algebraically, the traditional optimality-criteria are [[Functional (mathematics)|functionals]] of the [[eigenvalue]]s of the information matrix.

*'''A'''-optimality ("'''average'''" or '''trace''')
**One criterion is '''A-optimality''', which seeks to minimize the [[trace (linear algebra)|trace]] of the [[invertible matrix|inverse]] of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.
*'''C'''-optimality
**This criterion minimizes the variance of a [[best linear unbiased estimator]] of a predetermined linear combination of model parameters.
*{{anchor|D-optimality}}'''D'''-optimality ('''determinant''')
**A popular criterion is '''D-optimality''', which seeks to minimize |(X'X)<sup>−1</sup>|, or equivalently maximize the [[determinant]] of the [[information matrix]] X'X of the design. This criterion results in maximizing the [[Differential entropy|differential Shannon information]] content of the parameter estimates.
*'''E'''-optimality ('''eigenvalue''')
**Another design is '''E-optimality''', which maximizes the minimum [[eigenvalue]] of the information matrix.
*'''S'''-optimality<ref>{{cite journal |first1=Yeonjong |last1=Shin |first2=Dongbin |last2=Xiu |title=Nonadaptive quasi-optimal points selection for least squares linear regression |journal=SIAM Journal on Scientific Computing |volume=38 |pages=A385–A411 |doi=10.1137/15M1015868|year=2016 |issue=1|bibcode=2016SJSC...38A.385S }}
</ref>
**This criterion maximizes a quantity measuring the mutual column orthogonality of X and the [[determinant]] of the information matrix. 
*'''T'''-optimality
**This criterion maximizes the discrepancy between two proposed models at the design locations.<ref>{{Cite journal|last1=Atkinson|first1=A. C.|last2=Fedorov|first2=V. V.|date=1975|title=The design of experiments for discriminating between two rival models|url=https://academic.oup.com/biomet/article-lookup/doi/10.1093/biomet/62.1.57|journal=Biometrika|language=en|volume=62|issue=1|pages=57–70|doi=10.1093/biomet/62.1.57|issn=0006-3444}}</ref>

Other optimality-criteria are concerned with the variance of [[Predictive inference|predictions]]:
*'''G'''-optimality
**A popular criterion is '''G-optimality''', which seeks to minimize the maximum entry in the [[diagonal#Matrices|diagonal]] of the [[hat matrix]] X(X'X)<sup>−1</sup>X'. This has the effect of minimizing the maximum variance of the predicted values.
*'''I'''-optimality ('''integrated''')
**A second criterion on prediction variance is '''I-optimality''', which seeks to minimize  the average prediction variance ''over the design space''.
*'''V'''-optimality ('''variance''')
**A third criterion on prediction variance is '''V-optimality''', which seeks to minimize the average prediction variance over a set of m specific points.<ref>The above optimality-criteria are convex functions on domains of [[Semidefinite programming|symmetric positive-semidefinite matrices]]: See an on-line textbook for practitioners, which has many illustrations and statistical applications:
* {{cite book|title=Convex Optimization|first1=Stephen P.|last1=Boyd|first2=Lieven|last2=Vandenberghe|year=2004|publisher=Cambridge University Press|isbn=978-0-521-83378-3|url=https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf|access-date=October 15, 2011}}  (book in pdf)
Boyd and Vandenberghe discuss optimal experimental designs on pages 384–396.
</ref>

===Contrasts===
{{Main|Contrast (statistics)}}
 {{See also|Nuisance parameter}}
In many applications, the statistician is most concerned with a [[Nuisance parameter|"parameter of interest"]] rather than with [[Nuisance parameter|"nuisance parameters"]]. More generally, statisticians consider [[linear combination]]s of parameters, which are estimated via linear combinations of treatment-means in the [[design of experiments]] and in the [[analysis of variance]]; such linear combinations are called [[contrast (statistics)|contrasts]]. Statisticians can use appropriate optimality-criteria for such [[Nuisance parameter|parameters of interest]] and for [[Contrast (statistics)|contrasts]].<ref>Optimality criteria for [[nuisance parameter|"parameters of interest"]] and for [[Contrast (statistics)|contrasts]] are discussed by Atkinson, Donev and Tobias.</ref>

==Implementation==
Catalogs of optimal designs occur in books and in software libraries.

In addition, major [[List of statistical packages|statistical systems]] like [[SAS System|SAS]] and [[R (programming language)|R]] have procedures for optimizing a design according to a user's specification. The experimenter must specify a [[Statistical model|model]] for the design and an optimality-criterion before the method can compute an optimal design.<ref>Iterative methods and approximation algorithms are surveyed in the textbook by Atkinson, Donev and Tobias and in the monographs of Fedorov (historical) and Pukelsheim, and in the survey article by Gaffke and Heiligers.
</ref>

==Practical considerations==
Some advanced topics in optimal design require more [[statistical theory]] and practical knowledge in designing experiments.

===Model dependence and robustness===
Since the optimality criterion of most optimal designs is based on some function of the information matrix, the 'optimality' of a given design is ''[[Statistical model|model]] dependent'': While an optimal design is best for that [[Statistical model|model]], its performance may deteriorate on other [[Statistical model|models]]. On other [[Statistical model|models]], an ''optimal'' design can be either better or worse than a non-optimal design.<ref>See Kiefer ("Optimum Designs for Fitting Biased Multiresponse Surfaces" pages 289–299).</ref> Therefore, it is important to [[Benchmarking|benchmark]] the performance of designs under alternative [[Statistical model|models]].<!--  and for several criteria --><ref>Such benchmarking is discussed in the textbook by Atkinson et al. and in the papers of Kiefer. ''[[Statistical model|Model]]-[[Robust statistics|robust]]'' designs (including "Bayesian" designs) are surveyed by Chang and Notz.
</ref>

===Choosing an optimality criterion and robustness===
The choice of an appropriate optimality criterion requires some thought, and it is useful to benchmark the performance of designs with respect to several optimality criteria. Cornell writes that

{{quote|since the [traditional optimality] criteria . . . are variance-minimizing criteria, . . .  a design that is optimal for a given model using one of the . . . criteria is usually near-optimal for the same model with respect to the other criteria.|<ref>{{cite book
|author=Cornell, John
|title=Experiments with Mixtures: Designs, Models, and the Analysis of Mixture Data
|edition=third
|publisher=Wiley
|year=2002
|isbn=978-0-471-07916-3
}} (Pages 400-401)
</ref>}}

Indeed, there are several classes of designs for which all the traditional optimality-criteria agree, according to the theory of "universal optimality" of [[Jack Kiefer (mathematician)|Kiefer]].<ref>An introduction to "universal optimality" appears in the textbook of Atkinson, Donev, and Tobias. More detailed expositions occur in the advanced textbook of Pukelsheim and the papers of Kiefer.</ref> The experience of practitioners like Cornell and the "universal optimality" theory of Kiefer suggest that robustness with respect to changes in the ''optimality-criterion'' is much greater than is robustness with respect to changes in the ''model''.

====Flexible optimality criteria and convex analysis====
High-quality statistical software provide a combination of libraries of optimal designs or iterative methods for constructing approximately optimal designs, depending on the model specified and the optimality criterion. Users may use a standard optimality-criterion or may program a custom-made criterion.

All of the traditional optimality-criteria are [[Convex function|convex (or concave) functions]], and therefore optimal-designs are amenable to the mathematical theory of [[convex analysis]] and their computation can use specialized methods of [[Convex optimization|convex minimization]].<ref>Computational methods are discussed by Pukelsheim and by Gaffke and Heiligers.</ref> The practitioner need not select ''exactly one'' traditional, optimality-criterion, but can specify a custom criterion. In particular, the practitioner can specify a convex criterion using the maxima of convex optimality-criteria and [[Conical combination|nonnegative combinations]] of optimality criteria (since these operations preserve [[convex functions]]). For ''convex'' optimality criteria, the [[Jack Kiefer (mathematician)|Kiefer]]-[[Jacob Wolfowitz|Wolfowitz]] [https://books.google.com/books?id=5ZcfDZUJ4F8C&dq=Kiefer+Wolfowitz&pg=PA212 equivalence theorem] allows the practitioner to verify that a given design is globally optimal.<ref>The [[Jack Kiefer (mathematician)|Kiefer]]-[[Jacob Wolfowitz|Wolfowitz]] [https://books.google.com/books?id=5ZcfDZUJ4F8C&dq=Kiefer+Wolfowitz&pg=PA212 equivalence theorem] is discussed in Chapter 9 of Atkinson, Donev, and Tobias.</ref> The [[Jack Kiefer (mathematician)|Kiefer]]-[[Jacob Wolfowitz|Wolfowitz]] [https://books.google.com/books?id=5ZcfDZUJ4F8C&dq=Kiefer+Wolfowitz&pg=PA212 equivalence theorem] is related with the [[Legendre transformation|Legendre]]-[[Fenchel's duality theorem|Fenchel]] [[Convex conjugate|conjugacy]] for [[convex function]]s.<ref>Pukelsheim uses [[convex analysis]] to study [[Jack Kiefer (mathematician)|Kiefer]]-[[Jacob Wolfowitz|Wolfowitz]] [https://books.google.com/books?id=5ZcfDZUJ4F8C&dq=Kiefer+Wolfowitz&pg=PA212 equivalence theorem] in relation to the [[Legendre transformation|Legendre]]-[[Fenchel's duality theorem|Fenchel]] [[Convex conjugate|conjugacy]] for [[convex function]]s

The [[convex optimization|minimization]] of [[convex function]]s on domains of [[Semidefinite programming|symmetric positive-semidefinite matrices]] is explained in an on-line textbook for practitioners, which has many illustrations and statistical applications:
* {{cite book|title=Convex Optimization|publisher=Cambridge University Press|year=2004|url=https://web.stanford.edu/~boyd/cvxbook/ | author9=Boyd, Stephen and Vandenberghe, Lieven}}  (book in pdf)
Boyd and Vandenberghe discuss optimal experimental designs on pages 384–396.
</ref>

If an optimality-criterion lacks [[Quasiconvex function|convexity]], then finding a [[Global optimization|global optimum]] and verifying its optimality often are difficult.

===Model uncertainty and Bayesian approaches===
====Model selection====
{{See also|Model selection}}
When scientists wish to test several theories, then a statistician can design an experiment that allows optimal tests between specified models. Such "discrimination experiments" are especially important in the [[biostatistics]] supporting  [[pharmacokinetics]] and [[pharmacodynamics]], following the work of [[David R. Cox|Cox]] and Atkinson.<ref>See Chapter 20 in Atkinison, Donev, and Tobias.</ref>

====Bayesian experimental design====
{{Main|Bayesian experimental design}}
When practitioners need to consider multiple [[statistical model|models]], they can specify a [[Probability measure|probability-measure]] on the models and then select any design maximizing the [[expected value]] of such an experiment. Such probability-based optimal-designs are called optimal [[Bayesian inference|Bayesian]] [[Bayesian experimental design|designs]]. Such [[Bayesian experimental design|Bayesian designs]] are used especially for [[generalized linear models]] (where the response follows an [[Exponential family|exponential-family]] distribution).<ref>[[Bayesian experimental design|Bayesian designs]] are discussed in Chapter 18 of the textbook by Atkinson, Donev, and Tobias. More advanced discussions occur in the monograph by Fedorov and Hackl, and the articles by Chaloner and Verdinelli and by DasGupta. [[Bayesian experimental design|Bayesian designs]] and other aspects of "model-robust" designs are discussed by Chang and Notz.</ref>

The use of a [[Bayesian experimental design|Bayesian design]] does not force statisticians to use [[Bayesian inference|Bayesian methods]] to analyze the data, however. Indeed, the "Bayesian" label for probability-based experimental-designs is disliked by some researchers.<ref>As an alternative to "''Bayesian'' optimality", "''on-average'' optimality" is advocated in Fedorov and Hackl.</ref>  Alternative terminology for "Bayesian" optimality includes "on-average" optimality or "population" optimality.

==Iterative experimentation==
Scientific experimentation is an iterative process, and statisticians have developed several approaches to the optimal design of sequential experiments.

===Sequential analysis===
{{Main|Sequential analysis}}
[[Sequential analysis]] was pioneered by [[Abraham Wald]].<ref>{{Cite journal
 | author-link = Abraham Wald
 | first = Abraham
 | last = Wald
 | title = Sequential Tests of Statistical Hypotheses
 | journal = The Annals of Mathematical Statistics
 | volume = 16
 | issue = 2
 |date=June 1945
 | pages = 117–186
 | doi = 10.1214/aoms/1177731118
| jstor=2235829
| doi-access = free
 }}
</ref> In 1972, [[Herman Chernoff]] wrote an overview of optimal sequential designs,<ref>Chernoff, H. (1972) ''Sequential Analysis and Optimal Design,'' SIAM Monograph.</ref> while [[Adaptive design (medicine)|adaptive designs]] were surveyed later by S. Zacks.<ref>Zacks, S. (1996) "Adaptive Designs for Parametric Models". In: Ghosh, S. and Rao, C. R., (Eds) (1996). ''Design and Analysis of Experiments,'' Handbook of Statistics, Volume 13. North-Holland. {{isbn|0-444-82061-2}}.  (pages 151–180)</ref> Of course, much work on the optimal design of experiments is related to the theory of [[optimal decision]]s, especially the [[statistical decision theory]] of [[Abraham Wald]].<ref>
[http://www.lse.ac.uk/collections/cats/People/HenryPage.htm Henry P. Wynn] wrote, "the modern theory of optimum design has its roots in the decision theory school of U.S. statistics founded by [[Abraham Wald]]" in his introduction "Jack Kiefer's Contributions to Experimental Design", which is pages xvii–xxiv in the following volume:
* {{cite book |author=[[Jack Kiefer (mathematician)|Kiefer, Jack Carl]] |title=Jack Carl Kiefer Collected Papers III Design of Experiments |editor=[[Lawrence D. Brown|Brown, Lawrence D.]] |editor2=Olkin, Ingram |editor3=Jerome Sacks |editor4=Wynn, Henry P |publisher=Springer-Verlag and the [[Institute of Mathematical Statistics]] |year=1985 | pages=718+xxv |isbn=978-0-387-96004-3 |title-link=Jack Kiefer (mathematician) }}

[[Jack Kiefer (mathematician)|Kiefer]] acknowledges Wald's influence and results on many pages – 273 (page 55 in the reprinted volume), 280 (62), 289-291 (71-73), 294 (76), 297 (79),  315 (97) 319 (101) – in this article:
* {{cite journal
|first=J.
|last=Kiefer
|author-link=Jack Kiefer (mathematician)
|title=Optimum Experimental Designs
|journal=Journal of the Royal Statistical Society, Series B
|volume=21
|year=1959
|issue=2
|pages=272–319
|doi=10.1111/j.2517-6161.1959.tb00338.x
}}

</ref>

===Response-surface methodology===
{{Main|Response surface methodology}}
Optimal designs for [[Response surface methodology|response-surface models]] are discussed in the textbook by Atkinson, Donev and Tobias, and in the survey of Gaffke and Heiligers and in the mathematical text of Pukelsheim. The [[Blocking (statistics)|blocking]] of optimal designs is discussed in the textbook of Atkinson, Donev and Tobias and also in the monograph by Goos.

The earliest optimal designs were developed to estimate the parameters of regression models with continuous variables, for example, by [[Joseph Diaz Gergonne|J. D. Gergonne]] in 1815 (Stigler).  In English, two early contributions were made by [[Charles Sanders Peirce|Charles S. Peirce]] and [https://web.archive.org/web/20090810040534/http://www.webdoe.cc/publications/kirstine.php Kirstine Smith].

Pioneering designs for multivariate [[Response surface methodology|response-surfaces]] were proposed by [[George E. P. Box]]. However, Box's designs have few optimality properties. Indeed, the [[Box–Behnken design]] requires excessive experimental runs when the number of variables exceeds three.<ref>In the field of [[response surface methodology]], the [[Statistical efficiency|inefficiency]] of the [[Box–Behnken design]] is noted by Wu and Hamada (page 422).
* {{cite book |author1=Wu, C. F. Jeff  |author2=Hamada, Michael  |name-list-style=amp |title=Experiments: Planning, Analysis, and Parameter Design Optimization |publisher=Wiley |year=2002 |isbn=978-0-471-25511-6}}
Optimal designs for "follow-up" experiments are discussed by Wu and Hamada.
</ref>
Box's [[Central composite design|"central-composite" designs]] require more experimental runs than do the optimal designs of Kôno.<ref>The [[Statistical efficiency|inefficiency]] of [[George E. P. Box|Box]]'s [[Central composite design|"central-composite" designs]] are discussed by according to Atkinson, Donev, and Tobias (page 165). These authors also discuss the [[Blocking (statistics)|blocking]] of Kôno-type designs for quadratic [[response surface methodology|response-surfaces]].</ref>

===System identification and stochastic approximation===
{{See also|System identification|Stochastic approximation}}
The optimization of sequential experimentation is studied also in [[stochastic programming]] and in [[Systems analysis|systems]] and [[Control theory|control]]. Popular methods include [[stochastic approximation]] and other methods of [[stochastic optimization]]. Much of this research has been associated with the subdiscipline of [[system identification]].<ref>In system identification, the following books have chapters on optimal experimental design:
* {{cite book |author1=Goodwin, Graham C.  |author2=Payne, Robert L. |name-list-style=amp |title=Dynamic System Identification: Experiment Design and Data Analysis | publisher=Academic Press | year=1977 |isbn=978-0-12-289750-4|title-link=System identification }}
* {{cite book |author1=Walter, Éric  |author2=Pronzato, Luc  |name-list-style=amp |title=Identification of Parametric Models from Experimental Data |publisher=Springer |year=1997|title-link=System identification  }}
</ref>
In computational [[optimal control]], D. Judin & A. Nemirovskii and [https://web.archive.org/web/20071031081747/http://www.ipu.ru/labs/lab7/eng/staff/polyak.htm Boris Polyak] has described methods that are more efficient than the ([[Armijo rule|Armijo-style]]) [[Subgradient method#Step size rules|step-size rules]] introduced by [[George E. P. Box|G. E. P. Box]] in [[Response surface methodology|response-surface methodology]].<ref>Some step-size rules for of Judin & Nemirovskii and of [http://www.ipu.ru/labs/lab7/eng/staff/polyak.htm Polyak] {{Webarchive|url=https://web.archive.org/web/20071031081747/http://www.ipu.ru/labs/lab7/eng/staff/polyak.htm |date=2007-10-31 }} are explained in the textbook by Kushner and Yin:
* {{cite book |author=[[Harold J. Kushner|Kushner, Harold J.]] |author2=Yin, G. George|title=Stochastic Approximation and Recursive Algorithms and Applications |edition=Second | publisher=Springer  | year=2003 |isbn=978-0-387-00894-3|title-link=Stochastic approximation }}</ref>

[[Adaptive design (medicine)|Adaptive designs]] are used in [[clinical trials]], and optimal [[Adaptive design (medicine)|adaptive designs]] are surveyed in the ''Handbook of Experimental Designs'' chapter by Shelemyahu Zacks.

==Specifying the number of experimental runs==
===Using a computer to find a good design===
There are several methods of finding an optimal design, given an ''a priori'' restriction on the number of experimental runs or replications. Some of these methods are discussed by Atkinson, Donev and Tobias and in the paper by Hardin and [[Neil Sloane|Sloane]]. Of course, fixing the number of experimental runs ''a priori'' would be impractical.  Prudent statisticians examine the other optimal designs, whose number of experimental runs differ.

===Discretizing probability-measure designs===
In the mathematical theory on optimal experiments, an optimal design can be a [[probability measure]] that is [[Support (measure theory)|supported]] on an infinite set of observation-locations. Such optimal probability-measure designs solve a mathematical problem that neglected to specify the cost of observations and experimental runs. Nonetheless, such optimal probability-measure designs can be [[Discretization|discretized]] to furnish [[Approximation|approximately]] optimal designs.<ref>The [[discretization]] of optimal probability-measure designs to provide [[Approximation|approximately]] optimal designs is discussed by Atkinson, Donev, and Tobias and by Pukelsheim (especially Chapter 12).</ref>

In some cases, a finite set of observation-locations suffices to [[Support (measure theory)|support]] an optimal design. Such a result was proved by Kôno and [[Jack Kiefer (mathematician)|Kiefer]] in their works on [[Response surface methodology|response-surface designs]] for quadratic models. The Kôno–Kiefer analysis explains why optimal designs for response-surfaces can have discrete supports, which are very similar as do the less efficient designs that have been traditional in [[response surface methodology]].<ref>Regarding designs for quadratic [[response surface methodology|response-surfaces]], the results of Kôno and [[Jack Kiefer (mathematician)|Kiefer]] are discussed in Atkinson, Donev, and Tobias.

Mathematically, such results are associated with [[Chebyshev polynomials]], "Markov systems", and "moment spaces": See
* {{cite journal| author=[[Samuel Karlin|Karlin, Samuel]] |author2=[[Lloyd Shapley|Shapley, Lloyd]]|title=Geometry of moment spaces| journal=Mem. Amer. Math. Soc. | volume=12| year=1953}}
* {{cite book | author=[[Samuel Karlin|Karlin, Samuel]] |author2=Studden, William J.| title=Tchebycheff systems: With applications in analysis and statistics| publisher=Wiley-Interscience| year=1966| title-link=Chebyshev polynomials}}
* {{cite book|author1=Dette, Holger  |author2=Studden, William J. |name-list-style=amp |title=The Theory of canonical moments with applications in statistics, probability, and analysis| publisher=John Wiley & Sons Inc.|year=1997|title-link=Moment problem }}
</ref>

==History==
In 1815, an article on optimal designs for [[polynomial regression]] was published by [[Joseph Diaz Gergonne]], according to [[Stephen M. Stigler|Stigler]].

[[Charles Sanders Peirce|Charles S. Peirce]] proposed an economic theory of scientific experimentation in 1876, which sought to maximize the precision of the estimates. Peirce's optimal allocation immediately improved the accuracy of gravitational experiments and was used for decades by Peirce and his colleagues. In his 1882 published lecture at [[Johns Hopkins University]], Peirce introduced experimental design with these words:
<blockquote>
Logic will not undertake to inform you what kind of experiments you ought to make in order best to determine the acceleration of gravity, or the value of the Ohm; but it will tell you how to proceed to form a plan of experimentation.<br />
<br />[....] Unfortunately practice generally precedes theory, and it is the usual fate of mankind to get things done in some boggling way first, and find out afterward how they could have been done much more easily and perfectly.<ref>Peirce, C. S. (1882), "Introductory Lecture on the Study of Logic" delivered September 1882, published in ''Johns Hopkins University Circulars'', v. 2, n. 19, pp. 11–12, November 1882, see p. 11, ''Google Books'' [https://books.google.com/books?id=E0YFAAAAQAAJ&dq=%22Logic+will+not+undertake+to+inform%22+%22unfortunately+practice+generally%22&pg=PA11 Eprint]. Reprinted in ''Collected Papers'' v. 7, paragraphs 59–76, see 59, 63, ''Writings of Charles S. Peirce'' v. 4, pp. 378–82, see 378, 379, and ''The Essential Peirce'' v. 1, pp. 210–14, see 210–1, also lower down on 211.</ref>
</blockquote>

[[Kirstine Smith]] proposed optimal designs for polynomial models in 1918. (Kirstine Smith had been a student of the Danish statistician [[Thorvald N. Thiele]] and was working with [[Karl Pearson]] in London.)

==See also==
{{div col|colwidth=22em}}
*[[Bayesian experimental design]]
*[[Blocking (statistics)]]
*[[Computer experiment]]
*[[Convex function]]
*[[Convex optimization|Convex minimization]]
*[[Design of experiments]]
*[[Efficiency (statistics)]]
*[[Entropy (information theory)]]
*[[Fisher information]]
*[[Glossary of experimental design]]
*[[Hadamard's maximal determinant problem]]
*[[Information theory]]
*[[Jack Kiefer (mathematician)|Kiefer, Jack]]
*[[Replication (statistics)]]
*[[Response surface methodology]]
*[[Statistical model]]
*[[Abraham Wald|Wald, Abraham]]
*[[Jacob Wolfowitz|Wolfowitz, Jacob]]
{{div col end}}

==Notes==
{{Reflist|30em}}

==References==
* {{cite book
    |first1=A. C.
    |last1=Atkinson
    |first2=A. N.
    |last2=Donev
    |first3=R. D.
    |last3=Tobias    |url=http://www.us.oup.com/us/catalog/general/subject/Mathematics/ProbabilityStatistics/~~/dmlldz11c2EmY2k9OTc4MDE5OTI5NjYwNg
    |title=Optimum experimental designs, with '''SAS'''
    |publisher=[[Oxford University Press]]
    |year=2007
    |pages=511+xvi
    |isbn=978-0-19-929660-6
}}
* {{cite book
    |author-link=Herman Chernoff
    |last=Chernoff
    |first=Herman
    |title=Sequential analysis and optimal design
    |publisher=Society for Industrial and Applied Mathematics
    |year=1972
    |isbn=978-0-89871-006-9
}}
* {{cite book
    | last=Fedorov
    | first=V. V.
    | title=Theory of Optimal Experiments
    | publisher=Academic Press
    | year=1972
}}
* {{cite book
    | last1=Fedorov
    | first1=Valerii V.
    | last2=Hackl
    | first2=Peter
    | title=Model-Oriented Design of Experiments
    | series=Lecture Notes in Statistics
    | volume=125
    | publisher=Springer-Verlag
    | year=1997
}}
* {{cite book
    | last=Goos
    | first=Peter
    | title=The Optimal Design of Blocked and Split-plot Experiments
    | url=http://users.telenet.be/peter.goos/springer.htm
    | series=Lecture Notes in Statistics
    | volume=164
    | publisher=Springer
    | year=2002
}}
* {{cite book
    |author-link=Jack Kiefer (mathematician)
    |last=Kiefer
    |first=Jack&nbsp;Carl
    |title=Jack&nbsp;Carl Kiefer: Collected papers&nbsp;III—Design of experiments
    |editor1-link=Lawrence D. Brown
    |editor1-last=Brown
    |editor2-first=Ingram
    |editor2-link=Ingram Olkin
    |editor2-last=Olkin
    |editor3-first=Jerome
    |editor3-last=Sacks
    |editor3-link=Jerome Sacks
    |editor4-first=Henry P.
    |display-editors = 3
    |editor4-last=Wynn
    |editor4-link=Henry P. Wynn
    |publisher=Springer-Verlag and the Institute of Mathematical Statistics
    |year=1985
    |pages=718+xxv
    |isbn=978-0-387-96004-3
}}
* {{cite book
    |last1=Logothetis
    |first1=N.
    |last2=Wynn
    |first2=H.&nbsp;P.
    |author-link2=Henry&nbsp;P. Wynn
    |title=Quality through design: Experimental design, off-line quality&nbsp;control, and Taguchi's contributions
    |publisher=Oxford U.&nbsp;P.
    |year=1989
    |pages=464+xi
    |isbn=978-0-19-851993-5
}}
* {{cite journal
    |first=Kenneth
    |last=Nordström
    |date=May 1999
    |title=The life and work of Gustav Elfving
    |journal=Statistical Science
    |volume=14
    |issue=2
    |pages=174–196
    |mr=1722074
    |doi=10.1214/ss/1009212244
    |jstor=2676737
|doi-access=free
    }}
* {{cite book
    |last=Pukelsheim
    |first=Friedrich
    |author-link=Friedrich Pukelsheim
    |url=http://www.ec-securehost.com/SIAM/CL50.html
    |title=Optimal design of experiments
    |publisher=[[Society for Industrial and Applied Mathematics]]
    |series=Classics in Applied Mathematics
    |volume=50|year=2006
    |edition=republication with errata-list and new preface of Wiley (0-471-61971-X) 1993
    | pages=454+xxxii
    |isbn=978-0-89871-604-7
}}
* {{cite book
    |author1=Shah, Kirti R.  |author2=Sinha, Bikas K.
     |name-list-style=amp | title=Theory of Optimal Designs
    | series= Lecture Notes in Statistics
    | volume=54
    | publisher=Springer-Verlag
    | year=1989
    | pages=171+viii
    |isbn=978-0-387-96991-6
}}

==Further reading==
===Textbooks for practitioners and students===
====Textbooks emphasizing regression and response-surface methodology====
The textbook by Atkinson, Donev and Tobias has been used for short courses for industrial practitioners as well as university courses.

* {{cite book
    |first1=A. C.
    |last1=Atkinson
    |first2=A. N.
    |last2=Donev
    |first3=R. D.
    |last3=Tobias
    |title=Optimum experimental designs, with '''SAS'''
    |publisher= Oxford University Press
    | url = http://www.us.oup.com/us/catalog/general/subject/Mathematics/ProbabilityStatistics/~~/dmlldz11c2EmY2k9OTc4MDE5OTI5NjYwNg
    |year=2007
    |pages=511+xvi
    |isbn=978-0-19-929660-6
}}
* {{cite book
    |last1=Logothetis
    |first1=N.
    |last2=Wynn|first2=H. P.
    |author-link2=Henry P. Wynn
    |title=Quality through design: Experimental design, off-line quality control, and Taguchi's contributions
    |publisher=Oxford U. P.
    |year=1989
    |pages=464+xi
    |isbn=978-0-19-851993-5
}}

====Textbooks emphasizing block designs====
Optimal [[Randomized block design|block designs]] are discussed by Bailey and by Bapat. The first chapter of Bapat's book reviews the [[linear algebra]] used by Bailey (or the advanced books below). Bailey's exercises and discussion of [[random assignment|randomization]] both emphasize statistical concepts (rather than algebraic computations).

* {{cite book
    | last=Bailey
    | first=R. A.
    | author-link=Rosemary A. Bailey
    | title=Design of Comparative Experiments
    | publisher= Cambridge U. P.
    | year=2008
    | isbn=978-0-521-68357-9
    | url=http://www.maths.qmul.ac.uk/~rab/DOEbook
}} Draft available on-line. (Especially Chapter 11.8 "Optimality")
* {{cite book
    | author=Bapat, R. B.
    | title=Linear Algebra and Linear Models
    | url=https://www.springer.com/math/algebra/book/978-0-387-98871-9?cm_mmc=Google-_-Book%20Search-_-Springer-_-0
    | edition=Second
    | publisher=Springer
    | year=2000
    | isbn=978-0-387-98871-9
}} (Chapter 5 "Block designs and optimality", pages 99–111)
Optimal [[Randomized block design|block designs]] are discussed in the advanced monograph by Shah and Sinha and in the survey-articles by Cheng and by Majumdar.

===Books for professional statisticians and researchers===
* {{cite book
    | author=Chernoff, Herman
    | title=Sequential Analysis and Optimal Design 
    | publisher=[[Society for Industrial and Applied Mathematics|SIAM]]
    | year=1972 
    | isbn=978-0-89871-006-9
| title-link=Sequential analysis 
    | author-link=Herman Chernoff 
 }}
* {{cite book
    | author=Fedorov, V. V.
    | title=Theory of Optimal Experiments
    | publisher=Academic Press
    | year=1972
}}
* {{cite book
    | author=Fedorov, Valerii V. |author2=Hackl, Peter
    | title=Model-Oriented Design of Experiments
    | volume=125
    | publisher=Springer-Verlag
    | year=1997
}}
* {{cite book
    | author=Goos, Peter
    | title=The Optimal Design of Blocked and Split-plot Experiments
    | volume=164
    | publisher=Springer
    | year=2002
}}
* {{cite book
    |author1=Goos, Peter
    |author2=Jones, Bradley
    |name-list-style=amp 
    |title=Optimal design of experiments: a case study approach
    |publisher=Chichester Wiley
    |year=2011
    |pages=304
    |isbn=978-0-470-74461-1
}}
* {{cite book
    |author=[[Jack Kiefer (mathematician)|Kiefer, Jack Carl]].
    |title=Jack Carl Kiefer Collected Papers III Design of Experiments
 |editor=[[Lawrence D. Brown|Brown, Lawrence D.]] |editor2=[[Ingram Olkin|Olkin, Ingram]] |editor3=Jerome Sacks |editor4=Wynn, Henry P
    | publisher=Springer-Verlag and the [[Institute of Mathematical Statistics]]
    |year=1985
    | isbn=978-0-387-96004-3
|title-link=Jack Kiefer (mathematician)
    }}
* {{cite book
    |author=Pukelsheim, Friedrich
    |title=Optimal Design of Experiments 
    |url=https://books.google.com/books?id=5ZcfDZUJ4F8C |publisher=[[Society for Industrial and Applied Mathematics]]
    | volume=50|year=2006
    |isbn=978-0-89871-604-7
}} Republication with errata-list and new preface of Wiley (0-471-61971-X) 1993
* {{cite book
    |author1=Shah, Kirti R.  |author2=Sinha, Bikas K.
     |name-list-style=amp | title=Theory of Optimal Designs
    | volume=54
    | publisher=Springer-Verlag
    | year=1989
    |isbn=978-0-387-96991-6
}}
<!--
* {{cite book
    |author=Silvey, S. D.
    |title=Optimal Design: An Introduction to the Theory for [[Parameter estimation|Parameter Estimation]]
    |publisher=Chapman and Hall
    |year=1980
    |pages=86+viii
    |isbn=0-412-22910-2
}} -->

===Articles and chapters===
* {{cite journal
    | doi=10.1214/ss/1177009939
    |author1=Chaloner, Kathryn  |author2=Verdinelli, Isabella
     |name-list-style=amp | year=1995
    |  title=Bayesian Experimental Design: A Review
    | journal=[[Statistical Science]]
    | volume=10
    |issue=3
    | pages=273–304
|citeseerx=10.1.1.29.5355|title-link=Bayesian experimental design  }}
* {{cite book
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | volume=13
    |editor=Ghosh, S. |editor2=[[Calyampudi Radhakrishna Rao|Rao, C. R.]]
    | publisher=North-Holland
    | year=1996
    | isbn=978-0-444-82061-7
}}
** {{cite book
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | chapter=[[Statistical model|Model]] [[Robust statistics|Robust]] Designs
    |pages=1055–1099
}}
** {{cite book
    | author=Cheng, C.-S
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | chapter=Optimal Design: Exact Theory
    |pages=977–1006
}}
** {{cite book
    | author=DasGupta, A
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | chapter=Review of Optimal [[Bayesian experimental design|Bayesian Designs]] |pages=1099–1148
}}
** {{cite book
    |author1=Gaffke, N.  |author2=Heiligers, B
     |name-list-style=amp | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | chapter=Approximate Designs for [[Response surface methodology|Polynomial Regression]]: [[Invariant estimator|Invariance]], [[Admissible decision rule|Admissibility]], and Optimality
    |pages=1149–1199
}}
** {{cite book
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | author=Majumdar, D
    | chapter=Optimal and Efficient Treatment-Control Designs
    |pages=1007–1054
}}
** {{cite book
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | author=Stufken, J
    | chapter=Optimal [[Crossover study|Crossover Designs]]
    |pages=63–90
}}
** {{cite book
    | title=Design and Analysis of Experiments
    | series=Handbook of Statistics
    | author=Zacks, S
    | chapter=Adaptive Designs for Parametric Models
    |pages=151–180
}}
* {{cite journal
    |author=Kôno, Kazumasa
    |year=1962
    |title=Optimum designs for quadratic regression on ''k''-cube
    |journal=Memoirs of the Faculty of Science. Kyushu University. Series A. Mathematics
    |volume=16
    |pages=114–122
    |url=https://www.ams.org/mathscinet/pdf/153090.pdf
    |doi=10.2206/kyushumfs.16.114
    |issue=2
|doi-access=free
    }}

===Historical===
*{{cite journal
    |title=The application of the method of least squares to the interpolation of sequences
    |author=Gergonne, J. D.
    |journal=Historia Mathematica
    |volume=1
    |issue=4
    |date=November 1974 |orig-year=1815
    |pages=439–447
    |edition=Translated by Ralph St. John and [[Stephen M. Stigler|S. M. Stigler]] from the 1815 French
    |doi=10.1016/0315-0860(74)90034-2
    |author-link=Joseph Diaz Gergonne
 |doi-access=free
    }}
*{{cite journal
    |title=Gergonne's 1815 paper on the design and analysis of polynomial regression experiments
    |author=Stigler, Stephen M.
    |journal=Historia Mathematica
    |volume=1
    |issue=4
    |date=November 1974
    |pages=431–439
    |doi=10.1016/0315-0860(74)90033-0
    |author-link=Stephen M. Stigler
 |doi-access=free
    }}
* {{cite journal
| author=Peirce, C. S | year=1876| title=Note on the Theory of the Economy of Research | journal=Coast Survey Report | pages=197–201| author-link=Charles Sanders Peirce}} (Appendix No. 14). [http://docs.lib.noaa.gov/rescue/cgs/001_pdf/CSC-0025.PDF#page=222 NOAA PDF Eprint]. Reprinted in {{cite book| year=1958 | title=Collected Papers of Charles Sanders Peirce | volume=7 | title-link=Charles Sanders Peirce bibliography#CP }} paragraphs 139–157, and in {{cite journal
| last1=Peirce
| first1=C. S.
|date=July–August 1967| title=Note on the Theory of the Economy of Research
| journal=Operations Research
|volume=15 | issue=4
| pages=643–648
| doi=10.1287/opre.15.4.643
|jstor=168276
}}
* {{cite journal
|author=Smith, Kirstine
|title=On the Standard Deviations of Adjusted and Interpolated Values of an Observed Polynomial Function and its Constants and the Guidance They Give Towards a Proper Choice of the Distribution of the Observations
|year=1918
|journal= Biometrika
|volume=12
|issue=1/2
|pages=1–85
|jstor=2331929
|doi=10.2307/2331929|url=https://zenodo.org/record/1431591
}}

{{Experimental design|state=expanded}}
{{Statistics|collection|state=collapsed}}

[[Category:Design of experiments]]
[[Category:Regression analysis]] <!-- Many Optimal designs are optimal for estimating regression coefficients -->
[[Category:Statistical theory]]
[[Category:Optimal decisions]]
[[Category:Mathematical optimization]]
[[Category:Industrial engineering]]
[[Category:Systems engineering]]
[[Category:Statistical process control]]
[[Category:Management cybernetics]]