What Is a Linear Regression Model?

A linear regression model describes the relationship between a dependent variable, y, and one or more independent variables, X. The dependent variable is also called the response variable. Independent variables are also called explanatory or predictor variables. Continuous predictor variables are also called covariates, and categorical predictor variables are also called factors. The matrix X of observations on predictor variables is usually called the design matrix.

A multiple linear regression model is

$$y_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+\cdots +{\beta }_p{X}_{ip}+{\epsilon }_i,i=1,\cdots ,n,$$

where

• n is the number of observations.

• y_i is the ith response.

• β_k is the kth coefficient, where β₀ is the constant term in the model. Sometimes, design matrices might include information about the constant term. However, fitlm or stepwiselm by default includes a constant term in the model, so you must not enter a column of 1s into your design matrix X.

• X_ij is the ith observation on the jth predictor variable, j = 1, ..., p.

• ε_i is the ith noise term, that is, random error.

If a model includes only one predictor variable (p = 1), then the model is called a simple linear regression model.

In general, a linear regression model can be a model of the form

$$y_i={\beta }_0+{\displaystyle \sum _{k=1}^K{\beta }_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)}+{\epsilon }_i,i=1,\cdots ,n,$$

where f (.) is a scalar-valued function of the independent variables, X_ijs. The functions, f (X), might be in any form including nonlinear functions or polynomials. The linearity, in the linear regression models, refers to the linearity of the coefficients β_k. That is, the response variable, y, is a linear function of the coefficients, β_k.

Some examples of linear models are:

$$\begin{array}{l}{y}_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+{\beta }_3{X}_{i3}+{\epsilon }_i\\ y_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+{\beta }_3{X}_{i1}^3+{\beta }_4{X}_{i2}^2+{\epsilon }_i\\ y_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+{\beta }_3{X}_{i1}{X}_{i2}+{\beta }_4\log X_{i3}+{\epsilon }_i\end{array}$$

The following, however, are not linear models since they are not linear in the unknown coefficients, β_k.

$$\begin{array}{l}\log y_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+{\epsilon }_i\\ y_i={\beta }_0+{\beta }_1{X}_{i1}+\frac{1}{{\beta }_2{X}_{i2}}+e^{{\beta }_3{X}_{i1}{X}_{i2}}+{\epsilon }_i\end{array}$$

The usual assumptions for linear regression models are:

• The noise terms, ε_i, are uncorrelated.

• The noise terms, ε_i, have independent and identical normal distributions with mean zero and constant variance, σ². Thus,

  $$\begin{array}{l}E\left(y_i\right)=E\left({\displaystyle \sum _{k=0}^K{\beta }_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)}+{\epsilon }_i\right)\\ ={\displaystyle \sum _{k=0}^K{\beta }_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)}+E\left({\epsilon }_i\right)\\ ={\displaystyle \sum _{k=0}^K{\beta }_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)}\end{array}$$

  and

  $$V\left(y_i\right)=V\left({\displaystyle \sum _{k=0}^K{\beta }_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)}+{\epsilon }_i\right)=V\left({\epsilon }_i\right)={\sigma }^2$$

  So the variance of y_i is the same for all levels of X_ij.

• The responses y_i are uncorrelated.

The fitted linear function is

$${\widehat{y}}_i={\displaystyle \sum _{k=0}^K{b}_k{f}_k\left(X_{i1},X_{i2},\cdots ,X_{ip}\right)},i=1,\cdots ,n,$$

where $${\widehat{y}}_i$$ is the estimated response and b_ks are the fitted coefficients. The coefficients are estimated so as to minimize the mean squared difference between the prediction vector $$\widehat{y}$$ and the true response vector $$y$$, that is $$\widehat{y}-y$$. This method is called the method of least squares. Under the assumptions on the noise terms, these coefficients also maximize the likelihood of the prediction vector.

In a linear regression model of the form y = β₁X₁ + β₂X₂ + ... + β_pX_p, the coefficient β_k expresses the impact of a one-unit change in predictor variable, X_j, on the mean of the response E(y), provided that all other variables are held constant. The sign of the coefficient gives the direction of the effect. For example, if the linear model is E(y) = 1.8 – 2.35X₁ + X₂, then –2.35 indicates a 2.35 unit decrease in the mean response with a one-unit increase in X₁, given X₂ is held constant. If the model is E(y) = 1.1 + 1.5X₁² + X₂, the coefficient of X₁² indicates a 1.5 unit increase in the mean of Y with a one-unit increase in X₁² given all else held constant. However, in the case of E(y) = 1.1 + 2.1X₁ + 1.5X₁², it is difficult to interpret the coefficients similarly, since it is not possible to hold X₁ constant when X₁² changes or vice versa.

References

[1] Neter, J., M. H. Kutner, C. J. Nachtsheim, and W. Wasserman. Applied Linear Statistical Models. IRWIN, The McGraw-Hill Companies, Inc., 1996.

[2] Seber, G. A. F. Linear Regression Analysis. Wiley Series in Probability and Mathematical Statistics. John Wiley and Sons, Inc., 1977.