Multiple Regression

Page Contents

Several Explanatory Variables

For each observation there are ~r + 1 values: ~r explanatory variables, ~X_1, ... ~X_{~r} and the response variable ~Y. _ We have ~n observations, with where the response variables in each case are ~Y_{1}, ... ~Y_{~n} , _ ~Y_{~i} ~ N ( &mu._~i , &sigma.^2 ) _ where

&mu._{~i} _ = _ &alpha. + sum{&beta._~j ~x_{~i , ~j}, _ ~j = 1,~r}

~x_{~i , ~j} _ being the value of the ~{j^{th}} explanatory variable of the ~{i^{th}} observation .

Let _ #{&mu.} = ( &mu._1 , ... &mu._{~r} ) , _ and this has maximum likelihood estimator _ est{#{&mu.}} = p ( #~y ), _ where _ #~y = ( ~y_1, ... ~y_{~n} ) , _ and _ p ( ) _ is the orthogonal projection onto the linear space L_1.

L_1 is spanned by _ #~e , #~x_1, ... #~x_~r , _ i.e.

#{&mu.} _ = _ &alpha. #~e + sum{&beta._~j #~x_~j, _ ~j = 1,~r}

where #~e _ = _ ( 1, ... , 1 ), _ _ #{~x_~j} _ = _ ( ~x_{1 , ~j} , ... , ~x_{~n , ~j} )

Put

est{#{&mu.}} _ = _ p ( #~y ) _ = _ est{&alpha.} #~e + sum{est{&beta._~j} #{~x_~j}, _ ~j = 1,~r}

Using _ ( #~y - p ( #~y ) ) &dot. #{~v} = 0 , _ &forall. #{~v} &in. L_1 , _ we have:

_ #~y &dot. #~e _ = _ est{&alpha.} #~e &dot. #~e + sum{est{&beta._~j} #{~x_~j} &dot. #~e,~j = 1,~r} _ _ _ _ _ _ _ _ ... #{(1)}

_ #~y &dot. #{~x_{~k}} _ = _ est{&alpha.} #~e &dot. #{~x_{~k}} + sum{est{&beta._~j} #{~x_~j} &dot. #{~x_{~k}},~j = 1,~r} , _ ~k = 1, ... ~r _ _ ... #{(2)}

#{(1)} can be rewritten

~n $~y _ = _ ~n est{&alpha.} + sum{est{&beta._~j} ~n $~x_~j,~j = 1,~r}

est{&alpha.} _ = _ $~y &minus. sum{est{&beta._~j} $~x_~j,~j = 1,~r} _ _ _ _ ... #{(3)}

where _ $~y _ = _ ( 1/n ) &sum._{~i} ~y_{~i} _ , _ and _ $~x_~j _ = _ ( 1/n ) &sum._{~i} ~x_{~i , ~j}

Substitiuting for est{&alpha.} from #{(3)} into #{(2)}

#~y &dot. #{~x_{~k}} _ = _ ($~y &minus. sum{est{&beta._~j} $~x_~j,~j = 1,~r} ) #~e &dot. #{~x_{~k}} + sum{est{&beta._~j} #{~x_~j} &dot. #{~x_{~k}},~j = 1,~r}

#~y &dot. #{~x_{~k}} &minus. $~y #~e &dot. #{~x_{~k}} _ = _ sum{est{&beta._~j} (#{~x_~j} &dot. #{~x_{~k}} &minus. ~{$x_~j} #~e &dot. #{~x_{~k}}),~j = 1,~r} _ _ ... #{(4)}

Now for any vector #{~q} = (~q_1, ... ~q_{~n}),

&sum._{~i} ~q_{~i}(~x_{~i,~j} &minus. ~{$x_~j}) _ = _ &sum._{~i} ~q_{~i}~x_{~i,~j} &minus. ~{$x_~j} &sum._{~i} ~q_{~i}

_ _ _ _ _ _ _ _ _ = _ #{~q} &dot. #{~x_~j} &minus. ~{$x_~j} #{~q} &dot. #~e

similarly

&sum._{~i} ~q_{~i}(~y_{~i} &minus. ~{$y}) _ = _ #{~q} &dot. #~y &minus. ~{$y} #{~q} &dot. #~e

in particular

#~y &dot. #{~x_{~k}} &minus. $~y #~e &dot. #{~x_{~k}} _ = _ &sum._{~i} ~x_{~i,~k}(~y_{~i} &minus. ~{$y}) _ = _ S_{~k, ~y}

#{~x_~j} &dot. #{~x_{~k}} &minus. ~{$x_~j} #~e &dot. #{~x_{~k}} _ = _ &sum._{~i} ~x_{~i,~k}(~x_{~i,~j} &minus. ~{$x_~j}) _ = _ S_{~k, ~j}

So #{(4)} becomes:

S_{~k, ~y} _ = _ &sum._~j est{&beta._~j}S_{~k, ~j} _ _ _ _ ~k = 1, ... ~r

or in matrix notation: _ _ P _ = _ est{#{&beta.}} S , _ where _ P = [ S_{~k, ~y} ] _{~k = 1 ... ~r} , _ est{#{&beta.}} = [ est{&beta._~j} ] _{~k = 1 ... ~r} , _ S = [ S_{~k, ~j} ] _{~k , ~j = 1 ... ~r}

So if S has an inverse, the solution for est{#{&beta.}} is:

est{#{&beta.}} _ = _ S^{-1} P
est{&alpha.} _ = _ $~y - sum{est{&beta._~j} $~x_~j, _ ~j = 1,~r}