Question

Given the general MULTIPLE linear regression model with normal error, derive the LSE and MLE for the regression coefficients and show they are equivalent.

I need help on this question. it needs to be for a multiple linear regression model. thank you in advance.

Answer 1

Given the general MULTIPLE linear regression model with normal error

We first derive MLE for the regression coefficients let it denote by = B for convenience

Consider Model

y = XB + e

where B are regression coefficients

MLE is needed when one introduces the following assumptions

Error e follow i.i.d normal distribution with mean 0 and variance

i.e   e N ( 0 , )

and y N ( XB , )

only focus on the use of MLE in cases where y and e are normally distributed

The pdf of y is given by

f ( y | X ,B , ) =

and the log likelihood function is given by

log L = Log( f ( y | X ,B , ) ) = N/2 ln ( 1 / 2 ) + N ln ( 1 / ) +

         

Derivate log L w.r.t B and equate it to 0

here (y-XB) ' (y-XB) = y'y - y' XB -(XB)'y + (XB)'(XB)

                                  = y'y - 2 ( XB ) 'y - (XB)'y + (XB)'(XB)

                                  = y'y - 2 B'X'y - (XB)'y + (XB)'(XB)

Derivate it w.r.t B

                                  = 0 - 2 X'y + 2(X'X) B

                                  = - 2 X'y + 2(X'X) B                        .... ( eq * )

Thus equating Derivative of log L w.r.t B to 0

we get

=       = 0

0 + 0 + = 0

(- 2 X'y + 2(X'X)B ) / 2 = 0                     ( from eq * )

2 (- X'y + (X'X) B) = 0

   (X'X) B = X'y

(X'X) ^-1(X'X) B = (X'X) ^-1 X'y                     ( multiply both side by (X'X) ^-1 )

   B = (X'X) ^-1 X'y                                           ( MLE for B )

which is MLE for the regression coefficients .

Now we will find least square estimate LSE for regression coefficients

MULTIPLE linear regression model is give by

y = XB + e

we need to minimize e'e

Now     XB + e = y

            e = y - XB

therefore , e'e = ( y - XB) ' ( y - XB )

minimize e'e = y'y - y' XB -(XB)'y + (XB)'(XB)

                e'e = y'y - 2 B'X'y - (XB)'y + (XB)'(XB)

e'e = y′y−2B′X′y + B'X' XB

Derivative of e'e w.r.t B and equate to 0

= −2X′y+ 2X′XB = 0                  

       

   −X′y + X′XB = 0     

   X′y   = X′XB    

   X′XB     = X′y

B = ( X′X )^-1X′y

This is LSE for regression coefficients

So from both equations we can see that LSE and MLE for the regression coefficients are equivalent