Review of the last session

The Gauss-Markov Assumptions

1. Linearity assumption: \(y=\beta_0+\beta_1 x+\epsilon\)

2. \(X\) is a full rank matrix.

3. \(E(\epsilon|X)=0\)

4. \(E(\epsilon \epsilon'|X)=\sigma^2 I\)

5. \(X\) and \(\epsilon\) are orthogonal \(X \perp \epsilon\)

6. \(\epsilon | X \sim N(0,\sigma^2 I)\)

Th assumption 6 is not actually required for the Gauss-Markov Theorem. However, we often assume it to make hypothesis testing easier.

Gauss-Markov Theorem

The Gauss-Markov Theorem states that, conditional on assumptions 1-5, OLS estimator is the Best Linear, Unbiased, and Efficient estimator (BLUE):

1. \(\hat{\beta}\) is an unbiased estimator of \(\beta\).
2. \(\hat{\beta}\) is a linear estimator of \(\beta\).
3. \(\hat{\beta}\) has minimal variance among all linear and unbiased estimators.

Biased vs. unbiased estimation

• \(E(\hat{\beta})<\beta\)

Estimations of \(\beta: E(\hat{\beta})<\beta\)

Biased vs. unbiased estimation

• \(E(\hat{\beta})>\beta\)

Estimations of \(\beta: E(\hat{\beta})>\beta\)

Biased vs. unbiased estimation

• \(E(\hat{\beta})=\beta\)

Estimations of \(\beta: E(\hat{\beta})=\beta\)

OLS regression: an example

A regression model of \(y=\beta_0+\beta_1x+\epsilon\) estimated using OLS method has some properties that can help us to analyze the estimated model and whether some of the Gauss-Markov Theorem assumptions are satisfied.

# try jtools library for cleaner regression report and of course some other interesting tools
library(jtools)
# OLS model
lin.mod <- lm(logpgp95~avexpr, data=rawData)

summ(lin.mod)

Plotting the regression results

## You need "ggstance" package installed for plot_summs
plot_summs(lin.mod, scale = TRUE,
           coefs = c("Prot. from exprop."="avexpr" ),
           inner_ci_level = .9)

Properties of OLS: 1

The observed values of \(X\) are uncorrelated with the residuals(regression errors).

#Predictions
rawData$logpgp95_hat<- predict(lin.mod)
#Errors (residuals)
rawData$e_hat<- rawData$logpgp95-rawData$logpgp95_hat

Properties of OLS:2

The sum of the residuals is zero.

If there is a constant, then the first column in \(X\) will be a column of ones. This means that for the first element in the \(X'e\) vector (i.e. \(X_{11}e_1+x_{21}e_2+\dots+x_{N1}e_N\)) to be zero, it must be the case that \(\sum_1^N e_i= 0\).

sum(rawData$e_hat) 

## [1] 0.00000000000001865175

round(sum(rawData$e_hat), 3)

## [1] 0

Properties of OLS: 3

The sample mean of the residuals is zero. This follows straightforwardly from the previous property, \(\bar{e}=\frac{\sum_1^N e_i}{N}= 0\).

mean_ehat=mean(rawData$e_hat)
round(mean_ehat,2)

## [1] 0

library(ggplot2)

ggplot(rawData, aes(x = e_hat)) +
  geom_density(col='maroon', lwd=1)

Properties of OLS: 4

The regression hyperplane, line in a bivariate model, passes through the means of the observed values (\(\bar{x}\) and \(\bar{y}\)).

plot(rawData$avexpr, rawData$logpgp95 , col="navy", pch=19, cex=.5,
     xlab="X", ylab="Y")
lines(logpgp95_hat~avexpr,data=rawData , col="maroon", lwd=1)
points(mean(rawData$avexpr), mean(rawData$logpgp95), pch = 18, col = "green", cex = .9)

Properties of OLS: 5

The predicted values of \(y\) are uncorrelated with the residuals (errors).

How good is an estimated regression?

• We learned about the concept of (un)biased estimation. An unbiased estimation shows that how well we quantified the association between variable \(x\), or variable \(x\)s in multivariate regression, and variable \(y\).

• Another measure of a good estimation is how much of variations in the outcome variable \(y\) is explained using the independent variable(s).

• There is a subtle difference between these two measures. The first part is the focus of causal inference models while the second one is the focus of prediction/forecasting models.

• This part of the course focuses on the goodness of fit/prediction power/forecasting power/explanatory power of a regression model.

Root of Mean Square Error

• The sum of squared errors\(SSE_{OLS}\):

\[\begin{equation} SSE_{OLS}=\sum_{i=1}^N e^2=\sum_{i=1}^N (y_1-\hat{y}_1)^2+(y_2-\hat{y}_2)^2+\dots+(y_N-\hat{y}_N)^2 \end{equation}\]

\[\begin{equation} RMSE=\sqrt{\frac{\sum_{i=1}^N e^2}{N}} \end{equation}\]

R-squared (coefficient of determination)

• What if we want to compare RMSE with a benchmark? We can use \(SSE_{\bar{y}}\):

\[\begin{equation} SSE_{\bar{y}}=\sum_{i=1}^N(y_1-\bar{y})^2+(y_2-\bar{y})^2+\dots+(y_N-\bar{y})^2 \end{equation}\]

R-squared

• How much of the variation in y, measured by \(SSE_{\bar{y}}\), is not explained by the OLS model, measured \(SSE_{OLS}\):

\[\begin{equation} \frac{SSE_{OLS}}{SSE_{\bar{y}}} \end{equation}\]

• Therefore, the amount variation in outcome \(y\) that is explained by OLS is:

\[\begin{equation} R^2=1-\frac{SSE_{OLS}}{SSE_{\bar{y}}} \end{equation}\]

\(R^2\): An example

\(R^2\)-adjusted

How can we penalize unnecessary complexity of a regression model?

\[\begin{equation} R^2_{adjusted}=1-(1-R^2)[\frac{n-1}{n-k-1}] \end{equation}\]

where \(k\) is the number of regressors.