Editing Statistical model

{{Short description|Type of mathematical model}}
{{Use dmy dates|date=March 2024}}
A '''statistical model''' is a [[mathematical model]] that embodies a set of [[statistical assumptions]] concerning the generation of [[Sample (statistics)|sample data]] (and similar data from a larger [[Statistical population|population]]). A statistical model represents, often in considerably idealized form, the [[Data generating process|data-generating process]].<ref>{{Harvnb | Cox | 2006 | page=178}}</ref> When referring specifically to [[probability|probabilities]], the corresponding term is '''probabilistic model'''. All [[Statistical hypothesis testing|statistical hypothesis tests]] and all [[Estimator|statistical estimators]] are derived via statistical models. More generally, statistical models are part of the foundation of [[statistical inference]]. A statistical model is usually specified as a mathematical relationship between one or more [[random variables]] and other non-random variables.  As such, a statistical model is "a formal representation of a theory" ([[Herman J. Adèr|Herman Adèr]] quoting [[Kenneth A. Bollen|Kenneth Bollen]]).<ref>{{Harvnb|Adèr|2008|p=[https://books.google.com/books?id=LCnOj4ZFyjkC&pg=PA280 280]}}</ref>

==Introduction==
Informally, a statistical model can be thought of as a [[statistical assumption]] (or set of statistical assumptions) with a certain property: that the assumption allows us to calculate the probability of any [[event (probability theory)|event]]. As an example, consider a pair of ordinary six-sided [[dice]]. We will study two different statistical assumptions about the dice.

The first statistical assumption is this: for each of the dice, the probability of each face (1, 2, 3, 4, 5, and 6) coming up is {{sfrac|1|6}}. From that assumption, we can calculate the probability of both dice coming up 5:&thinsp; {{nowrap|{{sfrac|1|6}} × {{sfrac|1|6}} {{=}}}} {{nowrap|{{sfrac|1|36}}.&thinsp;}} More generally, we can calculate the probability of any event: e.g. (1 and 2) or (3 and 3) or (5 and 6). The alternative statistical assumption is this: for each of the dice, the probability of the face 5 coming up is {{sfrac|1|8}} (because the dice are [[Dice#Loaded dice|weighted]]). From that assumption, we can calculate the probability of both dice coming up 5:&thinsp; {{nowrap|{{sfrac|1|8}} × {{sfrac|1|8}} {{=}}}} {{nowrap|{{sfrac|1|64}}.&thinsp;}} We cannot, however, calculate the probability of any other nontrivial event, as the probabilities of the other faces are unknown.

The first statistical assumption constitutes a statistical model: because with the assumption alone, we can calculate the probability of any event. The alternative statistical assumption does ''not'' constitute a statistical model: because with the assumption alone, we cannot calculate the probability of every event. In the example above, with the first assumption, calculating the probability of an event is easy. With some other examples, though, the calculation can be difficult, or even impractical (e.g. it might require millions of years of computation). For an assumption to constitute a statistical model, such difficulty is acceptable: doing the calculation does not need to be practicable, just theoretically possible.

==Formal definition==
In mathematical terms, a statistical model is a pair (<math>S, \mathcal{P}</math>), where <math>S</math> is the set of possible observations, i.e. the [[sample space]], and <math>\mathcal{P}</math> is a set of [[probability distributions]] on <math>S</math>.<ref name="McCullagh">{{Harvnb | McCullagh | 2002}}</ref> The set <math>\mathcal{P}</math> represents all of the models that are considered possible. This set is typically parameterized: <math>\mathcal{P}=\{F_{\theta} : \theta \in \Theta\}</math>. The set <math>\Theta</math> defines the [[Statistical parameter|parameters]] of the model. If a parameterization is such that distinct parameter values give rise to distinct distributions, i.e. <math>F_{\theta_1} = F_{\theta_2} \Rightarrow \theta_1 = \theta_2</math> (in other words, the mapping is [[Injective function|injective]]), it is said to be ''[[Identifiability|identifiable]]''.<ref name="McCullagh" />

In some cases, the model can be more complex.

* In [[Bayesian statistics]], the model is extended by adding a probability distribution over the parameter space <math>\Theta</math>.
* A statistical model can sometimes distinguish two sets of probability distributions. The first set <math>\mathcal{Q}=\{F_{\theta} : \theta \in \Theta\}</math> is the set of models considered for inference. The second set <math>\mathcal{P}=\{F_{\lambda} : \lambda \in \Lambda\}</math> is the set of models that could have generated the data which is much larger than <math>\mathcal{Q}</math>. Such statistical models are key in checking that a given procedure is [[Robust statistics|robust]], i.e. that it does not produce catastrophic errors when its assumptions about the data are incorrect.

==An example==
Suppose that we have a population of children, with the ages of the children distributed [[Discrete uniform distribution|uniformly]], in the population. The height of a child will be [[stochastic]]ally related to the age: e.g. when we know that a child is of age 7, this influences the chance of the child being 1.5 meters tall. We could formalize that relationship in a [[linear regression]] model, like this: 
height<sub>''i''</sub>&nbsp;= ''b''<sub>0</sub>&nbsp;+ ''b''<sub>1</sub>age<sub>''i''</sub>&nbsp;+ ε<sub>''i''</sub>, where ''b''<sub>0</sub> is the intercept, ''b''<sub>1</sub> is a parameter that age is multiplied by to obtain a prediction of height, ε<sub>''i''</sub> is the error term, and ''i'' identifies the child. This implies that height is predicted by age, with some error.

An admissible model must be consistent with all the data points.  Thus, a straight line (height<sub>''i''</sub>&nbsp;= ''b''<sub>0</sub>&nbsp;+ ''b''<sub>1</sub>age<sub>''i''</sub>) cannot be admissible for a model of the data—unless it exactly fits all the data points, i.e. all the data points lie perfectly on the line.  The error term, ε<sub>''i''</sub>, must be included in the equation, so that the model is consistent with all the data points. To do [[statistical inference]], we would first need to assume some probability distributions for the ε<sub>''i''</sub>.  For instance, we might assume that the ε<sub>''i''</sub> distributions are [[i.i.d.]] Gaussian, with zero mean.  In this instance, the model would have 3 parameters: ''b''<sub>0</sub>, ''b''<sub>1</sub>, and the variance of the Gaussian distribution. We can formally specify the model in the form (<math>S, \mathcal{P}</math>) as follows.  The sample space, <math>S</math>, of our model comprises the set of all possible pairs (age, height).  Each possible value of <math>\theta</math>&nbsp;= (''b''<sub>0</sub>, ''b''<sub>1</sub>, ''&sigma;''<sup>2</sup>) determines a distribution on <math>S</math>; denote that distribution by <math>F_{\theta}</math>.  If <math>\Theta</math> is the set of all possible values of <math>\theta</math>, then <math>\mathcal{P}=\{F_{\theta} : \theta \in \Theta\}</math>. (The parameterization is identifiable, and this is easy to check.)

In this example, the model is determined by (1) specifying <math>S</math> and (2) making some assumptions relevant to <math>\mathcal{P}</math>. There are two assumptions: that height can be approximated by a linear function of age; that errors in the approximation are distributed as i.i.d. Gaussian.  The assumptions are sufficient to specify <math>\mathcal{P}</math>&mdash;as they are required to do.

==General remarks==
A statistical model is a special class of [[mathematical model]]. What distinguishes a statistical model from other mathematical models is that a statistical model is non-[[Deterministic system|deterministic]]. Thus, in a statistical model specified via mathematical equations, some of the variables do not have specific values, but instead have probability distributions; i.e. some of the variables are [[Random variable|stochastic]]. In the above example with children's heights, ε is a stochastic variable; without that stochastic variable, the model would be deterministic. Statistical models are often used even when the data-generating process being modeled is deterministic. For instance, [[coin tossing]] is, in principle, a deterministic process; yet it is commonly modeled as stochastic (via a [[Bernoulli process]]). Choosing an appropriate statistical model to represent a given data-generating process is sometimes extremely difficult, and may require knowledge of both the process and relevant statistical analyses. Relatedly, the statistician [[David Cox (statistician)|Sir David Cox]] has said, "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis".<ref>{{Harvnb | Cox | 2006 | page=197}}</ref>

There are three purposes for a statistical model, according to Konishi&nbsp;& Kitagawa:<ref>{{Harvnb | Konishi | Kitagawa | 2008 | loc= §1.1}}</ref>
#Predictions
#Extraction of information
#Description of stochastic structures
Those three purposes are essentially the same as the three purposes indicated by Friendly&nbsp;& Meyer: prediction, estimation, description.<ref>{{Harvnb| Friendly| Meyer | 2016| loc= §11.6}}</ref>

==Dimension of a model==
Suppose that we have a statistical model (<math>S, \mathcal{P}</math>) with <math>\mathcal{P}=\{F_{\theta} : \theta \in \Theta\}</math>.  In notation, we write that <math>\Theta \subseteq \mathbb{R}^k</math> where {{mvar|k}} is a positive integer (<math>\mathbb{R}</math> denotes the [[real numbers]]; other sets can be used, in principle).  Here, {{mvar|k}} is called the '''dimension''' of the model. The model is said to be ''[[Parametric model|parametric]]'' if <math>\Theta</math> has finite dimension.{{citation needed|date=June 2023}} As an example, if we assume that data arise from a univariate [[Gaussian distribution]], then we are assuming that
:<math>\mathcal{P}=\left\{F_{\mu,\sigma }(x) \equiv \frac{1}{\sqrt{2 \pi} \sigma} \exp\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) : \mu \in \mathbb{R}, \sigma > 0 \right\}</math>. 
In this example, the dimension, {{mvar|k}}, equals 2. As another example, suppose that the data consists of points ({{mvar|x}}, {{mvar|y}}) that we assume are distributed according to a straight line with i.i.d. Gaussian residuals (with zero mean): this leads to the same statistical model as was used in the example with children's heights.  The dimension of the statistical model is 3: the intercept of the line, the slope of the line, and the variance of the distribution of the residuals. (Note the set of all possible lines has dimension 2, even though geometrically, a line has dimension 1.)

Although formally <math>\theta \in \Theta</math> is a single parameter that has dimension {{mvar|k}}, it is sometimes regarded as comprising {{mvar|k}} separate parameters. For example, with the univariate Gaussian distribution, <math>\theta</math> is formally a single parameter with dimension 2, but it is often regarded as comprising 2 separate parameters—the mean and the standard deviation. A statistical model is [[Nonparametric statistics#Non-parametric models|''nonparametric'']] if the parameter set <math>\Theta</math> is infinite dimensional.  A statistical model is [[Semiparametric model|''semiparametric'']] if it has both finite-dimensional and infinite-dimensional parameters.  Formally, if {{mvar|k}} is the dimension of <math>\Theta</math> and {{mvar|n}} is the number of samples, both semiparametric and nonparametric models have <math>k \rightarrow \infty</math> as <math>n \rightarrow \infty</math>.  If <math>k/n \rightarrow 0</math> as <math>n \rightarrow \infty</math>, then the model is semiparametric; otherwise, the model is nonparametric.

Parametric models are by far the most commonly used statistical models. Regarding semiparametric and nonparametric models, [[David Cox (statistician)|Sir David Cox]] has said, "These typically involve fewer assumptions of structure and distributional form but usually contain strong assumptions about independencies".<ref>{{Harvnb | Cox | 2006 | page=2}}</ref>

==Nested models==
{{More citations needed|section|date=November 2023}}
{{distinguish|Multilevel models|}}
Two statistical models are '''nested''' if the first model can be transformed into the second model by imposing constraints on the parameters of the first model. As an example, the set of all Gaussian distributions has, nested within it, the set of zero-mean Gaussian distributions: we constrain the mean in the set of all Gaussian distributions to get the zero-mean distributions. As a second example, the quadratic model 
:{{math|''y''&nbsp;{{=}} ''b''<sub>0</sub>&nbsp;+ ''b''<sub>1</sub>''x''&nbsp;+ ''b''<sub>2</sub>''x''<sup>2</sup>&nbsp;+ ε,&nbsp; &nbsp; ε&nbsp;~ 𝒩(0, ''σ''<sup>2</sup>)}} 
has, nested within it, the linear model 
:{{math|''y''&nbsp;{{=}} ''b''<sub>0</sub>&nbsp;+ ''b''<sub>1</sub>''x''&nbsp;+ ε,&nbsp; &nbsp; ε&nbsp;~ 𝒩(0, ''σ''<sup>2</sup>)}}
—we constrain the parameter {{math|''b''<sub>2</sub>}} to equal 0.

In both those examples, the first model has a higher dimension than the second model (for the first example, the zero-mean model has dimension&nbsp;1).  Such is often, but not always, the case.  As an example where they have the same dimension, the set of positive-mean Gaussian distributions is nested within the set of all Gaussian distributions; they both have dimension 2.

==Comparing models==
{{See also | Statistical model selection}}
Comparing statistical models is fundamental for much of [[statistical inference]]. {{Harvtxt|Konishi | Kitagawa | 2008 | page= 75}} state: "The majority of the problems in statistical inference can be considered to be problems related to statistical modeling. They are typically formulated as comparisons of several statistical models." Common criteria for comparing models include the following: [[Coefficient of determination|''R''<sup>2</sup>]], [[Bayes factor]], [[Akaike information criterion]], and the [[likelihood-ratio test]] together with its generalization, the [[relative likelihood]].

Another way of comparing two statistical models is through the notion of [[Deficiency (statistics)|deficiency]] introduced by [[Lucien Le Cam]].<ref>{{cite journal |first=Lucien |last=Le Cam |title=Sufficiency and Approximate Sufficiency |volume=35 |journal=[[Annals of Mathematical Statistics]] |number=4 |publisher=[[Institute of Mathematical Statistics]] |pages=1429 |date=1964 |doi=10.1214/aoms/1177700372 |doi-access=free }}</ref>

==See also==
{{portal|Mathematics}}
{{Div col|colwidth=20em}}
*[[All models are wrong]]
*[[Blockmodel]]
*[[Conceptual model]]
*[[Design of experiments]]
*[[Deterministic model]]
*[[Effective theory]]
*[[Predictive model]]
*[[Response modeling methodology]]
*[[SackSEER]]
*[[Scientific model]]
*[[Statistical inference]] 
*[[Statistical model specification]]
*[[Statistical model validation]]
*[[Statistical theory]]
*[[Stochastic process]]
{{div col end}}

==Notes==
{{Reflist|20em}}
{{More footnotes needed|date=September 2010}}

==References==
{{refbegin}}
* {{Citation | author-link=Herman J. Adèr | last = Adèr | first = H. J. | year= 2008 |  contribution= Modelling | editor1-first= H. J. | editor1-last= Adèr | editor2-first= G. J. | editor2-last= Mellenbergh |editor2-link= Gideon J. Mellenbergh | title= Advising on Research Methods: A consultant's companion | pages= 271–304 |location= Huizen, The Netherlands |publisher= Johannes van Kessel Publishing }}.
*{{Citation |last1=Burnham |first1=K. P. |last2=Anderson |first2=D. R. |year=2002 |title=Model Selection and Multimodel Inference |edition=2nd |publisher=Springer-Verlag }}.
*{{Citation | author-link= David Cox (statistician) | last= Cox | first= D. R. | year= 2006 | title= Principles of Statistical Inference | publisher= Cambridge University Press}}.
*{{Citation | first1= M. | last1= Friendly | author1-link= Michael Friendly | first2= D. | last2= Meyer | title= Discrete Data Analysis with R | publisher= [[Chapman & Hall]] | year= 2016}}.
*{{Citation | last1=Konishi | first1=S. | last2= Kitagawa | first2=G. | year=2008 | title=Information Criteria and Statistical Modeling | publisher=Springer}}.
*{{Citation | author-link= Peter McCullagh | last= McCullagh | first= P. | year= 2002 | title= What is a statistical model? | journal= [[Annals of Statistics]] | volume= 30 | issue= 5 | pages= 1225–1310 | doi=10.1214/aos/1035844977 | url = http://www.stat.uchicago.edu/~pmcc/pubs/AOS023.pdf | doi-access= free }}.
{{refend}}

== Further reading ==
* Davison, A. C. (2008), ''Statistical Models'', [[Cambridge University Press]]
*{{citation | author1-first= M. | author1-last= Drton | author2-first= S. |author2-last= Sullivant | title= Algebraic statistical models | journal= Statistica Sinica | year= 2007 | volume= 17 | pages= 1273–1297 | url= http://www3.stat.sinica.edu.tw/statistica/oldpdf/A17n41.pdf }}
* Freedman, D. A. (2009), ''Statistical Models'', [[Cambridge University Press]]
* Helland, I. S. (2010), ''Steps Towards a Unified Basis for Scientific Models and Methods'', [[World Scientific]]
* [[Dirk Kroese|Kroese, D. P.]]; Chan, J. C. C. (2014), ''Statistical Modeling and Computation'', [[Springer Science+Business Media|Springer]]
* {{citation | author-last= Shmueli | author-first= G. | year= 2010 | title = To explain or to predict? | journal=[[Statistical Science]] | volume= 25 | issue= 3 | pages = 289–310 |doi= 10.1214/10-STS330 |arxiv= 1101.0891 | s2cid= 15900983 }}

{{Clear}}
{{Statistics|inference}}
{{Authority control}}

[[Category:Mathematical modeling]]
[[Category:Statistical models| ]]
[[Category:Statistical theory]]