Linear Regression | Sahithyan's Notes

The simplest method in statistics used to model and predict the relationship between continuous variables.

Simple Linear Regression

Linear regression involving 2 variables, dependent (response) and independent (explanatory).

Indications of a Linear Relationship

Scatter Diagram
Plots observed pairs $(x_i, y_i)$ to visualize the relationship.
Correlation Coefficient
Measures the strength and direction of the linear relationship.

Model

General population model:

Y = \alpha + \beta X

For an individual observation:

y_i = \alpha + \beta x_i + \varepsilon_i

where

$\alpha$ = intercept (value of $Y$ when $X = 0$ )
$\beta$ = slope (rate of change of $Y$ with respect to $X$ )
$\varepsilon_i$ = random error, assumed $\varepsilon_i \sim N(0, \sigma^2)$

Coefficient of Determination

Shows how much of the variance in ( $Y$ ) is explained by ( $X$ ). Denoted by $R^2$ .

Error Sum of Squares

Denoted by $\text{ESS}$ .

ESS = \sum (y_i - \alpha - \beta x_i)^2

Estimation of Parameters

Suppose the fitted regression line is $\hat{y} = \hat{\alpha} + \hat{\beta}x$ . Finding $\hat{\alpha}, \hat{\beta}$ that minimize $\text{ESS}$ is the goal. Least Squares Method is used here.

By setting partial derivatives to zero gives the normal equations:

\sum y_i = n\alpha + \beta \sum x_i

\sum x_i y_i = \alpha \sum x_i + \beta \sum x_i^2

Solving gives:

\hat{\beta} = \frac{n\sum x_i y_i - (\sum x_i)(\sum y_i)}{n\sum x_i^2 - (\sum x_i)^2}

\hat{\alpha} = \bar{y} - \hat{\beta}\bar{x}

Alternate form using deviations:

\hat{\beta} = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}

Sampling Distribution of Beta

Under the normal error assumption $\varepsilon_i \sim N(0, \sigma^2)$ :

\hat{\beta} \sim N\left(\beta, \frac{\sigma^2}{S_{xx}}\right)

where $S\_{xx} = \sum (x_i - \bar{x})^2 = n\sigma$ .

When $\sigma^2$ is unknown, estimate it using:

s^2 = \frac{ESS}{n - 2}

A confidence interval for the true slope $\beta$ is:

\hat{\beta} \pm t_{\alpha/2, n-2} \times \frac{s}{\sqrt{S_{xx}}}

Hypothesis Testing on the Regression Coefficient

To test whether $X$ significantly predicts $Y$ :

H_0: \beta = 0 \quad \text{and} \quad H_1: \beta \neq 0

Test statistic:

t = \frac{\hat{\beta} - 0}{s / \sqrt{S_{xx}}} \sim t_{n-2}

If $|t| > \text{critical value}$ , reject $H_0$ , which means the relationship between X and Y is statistically significant.

Analysis of Variance (ANOVA) for Regression

Tests whether the regression line fits the data well.

Source of Variation	Sum of Squares (SS)	df	Mean Square (MS)
Regression (RSS)	$\sum_{i=1}^n {(\hat{y_i}-\bar{y})^2}$	$1$	$\text{RSS} / 1$
Error (ESS)	$\sum_{i=1}^n {(y_i-\hat{y_i})^2}$	$n–2$	$\text{ESS} / (n–2)$
Total (TSS)	$\sum_{i=1}^n {(y_i-\bar{y})^2}$	$n–1$

Here:

$y_i$ : Actual observed value of the dependent variable for observation $i$
$\hat{y_i}$ : Predicted value of $y_i$ from the regression line
$\bar{y}$ : Mean of all observed $y_i$ values (overall average)

If RSS is large relative to ESS, the model fits well. Computed by F-ratio.

F-Ratio

F_\text{calc} = \frac{\text{RSS}/1}{\text{ESS}/(n-2)} \sim F_{1, n-2}

Decision Rule

$H_0: \text{regression line does not fit the data}$

$H_1: \text{regression line fits the data}$

Reject $H_0$ if $F_{\text{calc}} > F_{1, n-2, \alpha}$ .