Expand/Contract All

MLE Generalized

Page Contents

Now that we've seen the maximum likelihood method applied to both discrete and continuous distributions, we present a summary of the results on this page.

Maximum Likelihood Method

In an experiment, we observe an instance, ~x, of a random variable ~X which we assume has a known distribution (e.g. normal, binomial, etc.), but the distribution is dependent on an unknown parameter &theta. , _ &theta. &in. &Theta. . _ The #~{likelihood function} is:

L( &theta. ) _ #:= _ L( &theta. | ~x ) _ #:= _ _ array{ p ( ~x | &theta. ) , _ _ discrete distribution/ _ / _ f ( ~x | &theta. ) , _ _ continuous distribution}

The #~{maximum likelihood estimator} of &theta. , _ est{&theta.} , _ is such that

L( est{&theta.} ) = sup_{&theta. &in. &Theta.} \{ L( &theta. ) \}

Now suppose we want to test a hypothesis _ H_0: &theta. &in. &Theta._0 &subset. &Theta. . _ Let &theta._0 maximize L( &theta.) over &Theta._0, _ i.e. _ L( &theta._0 ) = sup_{&theta. &in. &Theta._0} \{ L( &theta. ) \}

Define the #~{likelihood ratio} as

LR( ~x ) _ = _ fract{L( &theta._0 | ~x ),L( est{&theta.} | ~x )} _ = _ fract{sup_{&theta. &in. &Theta._0} \{ L( &theta. ) \}, sup_{&theta. &in. &Theta.} \{ L( &theta. ) \}}

I.e. LR is the ratio of the maximum likelihood for the hypothesis (&theta. &in. &Theta._0) to the maximum likelihood for the alternative hypothesis (&theta. &in. &Theta.).

Note that LR(~x) will be between 0 and 1, and the greater its value, the more acceptable the hypothesis is.

The #~{log likelihood function} is defined as being:

LL ( ~x ) _ = _ -2 log_~e LR ( ~x )

and we have

LR ( ~y ) < LR ( ~x ) _ <=> _ LL ( ~y ) > LL ( ~x )

LR ( ~y ) = LR ( ~x ) _ <=> _ LL ( ~y ) = LL ( ~x )

The #~{significance probability} (SP) of ~x as:

SP ( ~x ) _ = _ P\{ ~y | LR(~y) &leq. LR(~x) \} _ = _ F_{LR} ( LR(~x) )

where F_{LR} is the distribution function of the random variable LR(~X). _ Alternatively

SP ( ~x ) _ = _ P\{ ~y | LL(~y) >= LL(~x) \} _ = _ 1 - F_{LL} ( LL(~x) ) _ *

where F_{LL} is the distribution function of the random variable LL(~X).

* Note that for discrete distributions 1 - F_{LL} ( LL(~x) ) = P\{ ~y | LL(~y) > LL(~x) \}, so SP(~x) = 1 - F_{LL} ( LL(~x) ) + P(~x)

The ~k% #~{confidence interval} as the range of values of &theta._0 for which SP > (100 - ~k)% - i.e. the range of values of &theta._0 for which we would accept the hypothesis H: &theta. = &theta._0, at the (100 - ~k)% level.

[ E.g. the usual value to use for the level of the test, &alpha., is 0.05 or 5%. The corresponding confidence interval is then (100 - 5)% or 95% ]

See examples of MLE statistics calculated for some common distributions.