Editing Generalized linear model (section)

== Examples ==

=== General linear models ===
{{Further|General linear model}}

A possible point of confusion has to do with the distinction between generalized linear models and [[general linear model]]s, two broad statistical models. Co-originator [[John Nelder]] has expressed regret over this terminology.<ref>{{cite journal |last= Senn|first=Stephen |year=2003 |title=A conversation with John Nelder |journal=Statistical Science |volume=18 |issue=1 |pages=118–131 |doi=10.1214/ss/1056397489|quote=I suspect we should have found some more fancy name for it that would have stuck and not been confused with the general linear model, although general and generalized are not quite the same. I can see why it might have been better to have thought of something else.|doi-access=free }}</ref>

The general linear model may be viewed as a special case of the generalized linear model with identity link and responses normally distributed. As most exact results of interest are obtained only for the general linear model, the general linear model has undergone a somewhat longer historical development. Results for the generalized linear model with non-identity link are [[asymptotic]] (tending to work well with large samples).

=== Linear regression ===

A simple, very important example of a generalized linear model (also an example of a general linear model) is [[linear regression]]. In linear regression, the use of the [[least-squares]] estimator is justified by the [[Gauss–Markov theorem]], which does not assume that the distribution is normal.

From the perspective of generalized linear models, however, it is useful to suppose that the distribution function is the normal distribution with constant variance and the link function is the identity, which is the canonical link if the variance is known.  Under these assumptions, the least-squares estimator is obtained as the maximum-likelihood parameter estimate.

For the normal distribution, the generalized linear model has a [[Closed-form expression|closed form]] expression for the maximum-likelihood estimates, which is convenient. Most other GLMs lack [[Closed-form expression|closed form]] estimates.

=== Binary data ===
{{See also|Binary regression}}
When the response data, ''Y'', are binary (taking on only values 0 and 1), the distribution function is generally chosen to be the [[Bernoulli distribution]] and the interpretation of ''μ''<sub>i</sub> is then the probability, ''p'', of ''Y''<sub>i</sub> taking on the value one.

There are several popular link functions for binomial functions.

==== Logit link function ====
The most typical link function is the canonical [[logit]] link:

:<math>g(p) = \operatorname{logit} p = \ln \left( { p \over 1-p } \right).</math>

GLMs with this setup are [[logistic regression]] models (or ''logit models'').

==== Probit link function as popular choice of inverse cumulative distribution function ====
Alternatively, the inverse of any continuous [[cumulative distribution function]] (CDF) can be used for the link since the CDF's range is <math>[0,1]</math>, the range of the binomial mean. The [[Normal distribution#Cumulative distribution function|normal CDF]] <math>\Phi</math> is a popular choice and yields the [[probit model]]. Its link is

:<math>g(p) = \Phi^{-1}(p).\,\!</math>

The reason for the use of the probit model is that a constant scaling of the input variable to a normal CDF (which can be absorbed through equivalent scaling of all of the parameters) yields a function that is practically identical to the logit function, but probit models are more tractable in some situations than logit models. (In a Bayesian setting in which normally distributed [[prior distribution]]s are placed on the parameters, the relationship between the normal priors and the normal CDF link function means that a [[probit model]] can be computed using [[Gibbs sampling]], while a logit model generally cannot.)

==== Complementary log-log (cloglog) ====
The complementary log-log function may also be used:
:<math>g(p) = \log(-\log(1-p)).</math>
This link function is asymmetric and will often produce different results from the logit and probit link functions.<ref>{{Cite web|url=http://www.stat.ualberta.ca/~kcarrier/STAT562/comp_log_log.pdf|title=Complementary Log-log Model}}</ref> The cloglog model corresponds to applications where we observe either zero events (e.g., defects) or one or more, where the number of events is assumed to follow the [[Poisson distribution]].<ref>{{Cite web|url=https://bayesium.com/which-link-function-logit-probit-or-cloglog/|title=Which Link Function — Logit, Probit, or Cloglog?|date=2015-08-14|website=Bayesium Analytics|language=en-US|access-date=2019-03-17}}</ref> The Poisson assumption means that

:<math>\Pr(0) = \exp(-\mu),</math>

where ''&mu;'' is a positive number denoting the expected number of events.  If ''p'' represents the proportion of observations with at least one event, its complement

:<math> 1-p = \Pr(0) = \exp(-\mu),</math>

and then

:<math> -\log(1-p) = \mu.</math>

A linear model requires the response variable to take values over the entire real line. Since ''&mu;'' must be positive, we can enforce that by taking the logarithm, and letting log(''&mu;'') be a linear model.  This produces the "cloglog" transformation

:<math>\log(-\log(1-p)) = \log(\mu).</math>

==== Identity link ====
The identity link ''g(p) = p'' is also sometimes used for binomial data to yield a [[linear probability model]].  However, the identity link can predict nonsense "probabilities" less than zero or greater than one.  This can be avoided by using a transformation like cloglog, probit or logit (or any inverse cumulative distribution function). A primary merit of the identity link is that it can be estimated using linear math—and other standard link functions are approximately linear matching the identity link near ''p'' = 0.5.

==== Variance function ====
The [[variance function]] for "{{visible anchor|quasibinomial}}" data is:

:<math>\operatorname{Var}(Y_i)= \tau\mu_i (1-\mu_i)\,\!</math>

where the dispersion parameter ''&tau;'' is exactly 1 for the binomial distribution. Indeed, the standard binomial likelihood omits ''&tau;''.  When it is present, the model is called "quasibinomial", and the modified likelihood is called a [[quasi-likelihood]], since it is not generally the likelihood corresponding to any real family of probability distributions.  If ''&tau;'' exceeds 1, the model is said to exhibit [[overdispersion]].

===Multinomial regression===
The binomial case may be easily extended to allow for a [[multinomial distribution]] as the response (also, a Generalized Linear Model for counts, with a constrained total). There are two ways in which this is usually done:

====Ordered response====
If the response variable is [[ordinal data|ordinal]], then one may fit a model function of the form:

:<math> g(\mu_m) = \eta_m = \beta_0 + X_1 \beta_1 + \cdots + X_p \beta_p + \gamma_2 + \cdots + \gamma_m = \eta_1 + \gamma_2 + \cdots + \gamma_m \text{ where } \mu_m = \operatorname{P}(Y \leq m). \,</math>

for ''m'' > 2. Different links ''g'' lead to [[ordinal regression]] models like [[Ordered logit|proportional odds model]]s or [[ordered probit]] models.

====Unordered response====
If the response variable is a [[Level of measurement#Nominal level|nominal measurement]], or the data do not satisfy the assumptions of an ordered model, one may fit a model of the following form:

:<math> g(\mu_m) = \eta_m = \beta_{m,0} + X_1 \beta_{m,1} + \cdots + X_p \beta_{m,p} \text{ where } \mu_m = \mathrm{P}(Y = m \mid Y \in \{1,m\} ). \,</math>

for ''m'' > 2. Different links ''g'' lead to [[multinomial logit]] or [[multinomial probit]] models.  These are more general than the ordered response models, and more parameters are estimated.

===Count data===
Another example of generalized linear models includes [[Poisson regression]] which models [[count data]] using the [[Poisson distribution]].  The link is typically the logarithm, the canonical link.

The variance function is proportional to the mean

:<math>\operatorname{var}(Y_i) = \tau\mu_i,\, </math>

where the dispersion parameter ''&tau;'' is typically fixed at exactly one. When it is not, the resulting [[quasi-likelihood]] model is often described as Poisson with [[overdispersion]] or ''quasi-Poisson''.