Linear regression is a method we can use to quantify the relationship between one or more predictor variables and a response variable.
Typically we use linear regression with quantitative variables. Sometimes referred to as “numeric” variables, these are variables that represent a measurable quantity. Examples include:
- Number of square feet in a house
- Population size of a city
- Age of an individual
However, sometimes we wish to use categorical variables as predictor variables. These are variables that take on names or labels and can fit into categories. Examples include:
- Eye color (e.g. “blue”, “green”, “brown”)
- Gender (e.g. “male”, “female”)
- Marital status (e.g. “married”, “single”, “divorced”)
When using categorical variables, it doesn’t make sense to just assign values like 1, 2, 3, to values like “blue”, “green”, and “brown” because it doesn’t make sense to say that green is twice as colorful as blue or that brown is three times as colorful as blue.
Instead, the solution is to use dummy variables. These are variables that we create specifically for regression analysis that take on one of two values: zero or one.
The number of dummy variables we must create is equal to k-1 where k is the number of different values that the categorical variable can take on.
For example, suppose we have the following dataset and we would like to use marital status and age to predict income:
To use marital status as a predictor variable in a regression model, we must convert it into a dummy variable.
Since it is currently a categorical variable that can take on three different values (“Single”, “Married”, or “Divorced”), we need to create k-1 = 3-1 = 2 dummy variables.
To create this dummy variable, we can let “Single” be our baseline value since it occurs most often. Thus, here’s how we would convert marital status into dummy variables:
We could then use Age, Married, and Divorced as predictor variables in a regression model.
When creating dummy variables, a problem that can arise is known as the dummy variable trap. This occurs when we create k dummy variables instead of k-1 dummy variables.
When this happens, at least two of the dummy variables will suffer from perfect multicollinearity. That is, they’ll be perfectly correlated. This causes incorrect calculations of regression coefficients and their corresponding p-values.
Dummy Variable Trap: When the number of dummy variables created is equal to the number of values the categorical value can take on. This leads to multicollinearity, which causes incorrect calculations of regression coefficients and p-values.
For example, suppose we converted marital status into the following dummy variables:
In this case, Single and Married are perfectly correlated and have a correlation coefficient of -1.
Thus, when we go to perform multiple linear regression the calculations for the regression coefficients will be incorrect.
How to Avoid the Dummy Variable Trap
You only need to remember one rule to avoid the dummy variable trap:
If a categorical variable can take on k different values, then you should only create k-1 dummy variables to use in the regression model.
For example, suppose you’d like to convert a categorical variable “school year” into dummy variables. Suppose this variable takes on the following values:
- Freshman
- Sophomore
- Junior
- Senior
Since this variable can take on 4 different values, we will only create 3 dummy variables. For example, our dummy variables might be:
- X1 = 1 if Sophomore; 0 otherwise
- X2 = 1 if Junior; 0 otherwise
- X3 = 1 if Senior; 0 otherwise
Since the number of dummy variables is one less than the number of values that “school year” can take on, we can avoid the dummy variable trap and the problem of multicollinearity.
Additional Resources
How to Use Dummy Variables in Regression Analysis
Introduction to Multiple Linear Regression
A Guide to Multicollinearity in Regression