Linear Models

Page Contents

Random Vectors

Random vector _ #~y _ = _ ( ~y_1, ... ~y_~n ) _ with components the random variables ~y_~i .

Expectation: _ E#~y _ = _ ( E~y_1, ... E~y_~n ). _ Expectation is a linear operator: _ E( #A#~x + #B#~y ) _ = _ #AE#~x + #BE#~y .

Covariance Matrix: _ cov( #~x , #~y ) _ = _ [ cov( ~x_~i , ~y_~j ) ]_{~i = 1 ... ~m , ~j = 1 ... ~n}

Dispersion Matrix, or Variance-Covariance Matrix: _ D( #~y ) _ = _ var( #~y ) _ #:= _ cov( #~y , #~y )

If #~a , #~b are constant vectors then _ cov( #~a^T#~x , #~b^T#~y ) _ = _ #~a^T cov( #~x , #~y ) #~b

If #A , #B are constant matrices then _ cov( #A#~x , #B#~y ) _ = _ #A cov( #~x , #~y ) #B^T

In particular _ D( #~a^T#~x ) _ = _ #~a^T D( #~x ) #~a

Linear Models

In a #~{linear model}, the distribution of #~y is linearly dependent on a number of unkown parameters _ &beta._1 , ... , &beta._~p , _ in the sense that
E~y_~i _ = _ ~x_{~i 1} &beta._1 + ... + ~x_{~i ~p} &beta._~p , _ ~i = 1 ... ~n .
Or in matrix terms: _ _ E#~y _ = _ #X#~{&beta.} , _ where #X is the ~n # ~p matrix of the (non-random) ~x_{~i ~j} , _ and _ #~{&beta.} = ( &beta._1 , ... , &beta._~p )^T

Also _ ~y_1, ... ~y_~n _ are uncorrelated, each with variance _ &sigma.^2 . _ I.e. _ D( #~y ) _ = _ &sigma.^2 #I .

An equivalent way to express the model is to say _ #~y _ = _ #X#~{&beta.} + #~{&epsilon.} , _ where _ E( #~{&epsilon.} ) _ = _ 0 ; _ and _ D( #~{&epsilon.} ) _ = _ &sigma.^2 #I .

#~y is known as the #~{response} _ ( ~y_1, ... ~y_~n _ are the response variables ) , _ and the matrix _ #X _ is called the #~{design matrix} of the model.

The aim is to provide estimates for _ &beta._1 , ... , &beta._~p , _ and also for _ &sigma.^2 .

Estimability

Model: _ _ E( #~y ) _ = _ #X#~{&beta.} , _ D( #~y ) _ = _ &sigma.^2 #I . _ No other assumptions are made about the distribution of #~y at this stage. (E.g. we do not assume normality).

A linear function of the parameters _ #~l^T#~{&beta.} _ = _ ~l_{1} &beta._1 + ... + ~l_{~p} &beta._~p , _ is said to be #~{estimable} _ if
&exist. _ #~c _ = _ (~c_1 ... ~c_n)^T _ such that _ E( #~c^T#~y ) _ = _ #~l^T#~{&beta.} , _ for any value of #~{&beta.} .

#~l^T#~{&beta.} _ = _ E( #~c^T#~y ) _ = _ #~c^TE( #~y ) _ = _ #~c^T#X#~{&beta.}, _ for any value of #~{&beta.} _ => _ #~l^T _ = _ #~c^T#X , _ some #~c &in. &reals.^~n , _ so _ #~l &in. RSp( #X ).

Invariance of g-inverse

For any ~p # ~n matrix #C , _ if we have matrices _ #A (~m # ~n) , _ and _ #B (~p # ~q) , _ such that _ RSp( #A ) &subseteq. RSp( #C ) , _ and _ CSp( #B ) &subseteq. CSp( #C ) , _ then _ #A #C^{-} #B _ is invariant under the choice of g-inverse , _ #C^{-} .

For any matrix #X , _ the following hold:
  1. RSp( #X ) _ = _ RSp( #X^T#X ) , _ so _ #X _ = _ #M#X^T#X _ for some matrix _ #M .
  2. #X (#X^T#X)^{-} #X^T _ is invariant under the choice of g-inverse , _ (#X^T#X)^{-} .
  3. #X (#X^T#X)^{-} #X^T#X _ = _ #X . _ _ I.e. _ (#X^T#X)^{-} #X^T _ is a g-inverse of _ #X .

Best Linear Unbiased Estimate

If _ #~l^T#~{&beta.} _ is an estimable function, and the linear function of the response , _ #~{&alpha.}^T#~y _ is an unbiased estimate of _ #~l^T#~{&beta.} , _ it is said to be the _ #~{best linear unbiased estimate} _ or _ #~{B.L.U.E.} _ if it is the estimate to have minimum variance among the class of estimates of #~l^T#~{&beta.} which are a linear function of the response.

Consider the model: _ _ E( #~y ) _ = _ #X#~{&beta.} , _ D( #~y ) _ = _ &sigma.^2 #I . _ If _ #~l^T#~{&beta.} _ is an estimable function, _ and _ #G _ is a least-squares g-inverse of _ #X , _ ( #X#G symmetric ) , _ then _ #~l^T#G#~y _ is the B.L.U.E. of _ #~l^T#~{&beta.} .
The variance of _ #~l^T#G#~y _ is _ &sigma.^2#~l^T ( #X^T#X )^{-} #~l