What is One-Hot Encoding? When Should You Use One-Hot Encoding Instead Of Tag Encoding?
These are typical data science interview questions that every aspiring data scientist should know the answer to.. After all, You will often have to choose between the two in a data science project!!
Machines understand numbers, not text. We need to convert each category of text into numbers for the machine to process them using mathematical equations. Have you ever wondered how we can do that? What are the different ways?
This is where Tag Encoding and One-Hot Encoding come into the picture.. We will discuss both in this article and understand the difference between them..
Note: Starting Your Machine Learning Journey? I recommend taking our complete and popular Applied Machine Learning Course!
Table of Contents
- What is categorical coding?
- Different approaches to categorical coding
- Label encoding
- One-Hot Coding
- When to use tag encoding vs hot encoding?
What is categorical coding?
As usual, any structured data set includes multiple columns, a combination of numerical and categorical variables. A machine can only understand numbers. Can't understand the text. That is also essentially the case with machine learning algorithms..
That is mainly the reason why we need to convert categorical columns to numeric columns for a machine learning algorithm to understand them.. This process is called categorical coding.
Categorical coding is a process of converting categories into numbers.
In the next section, I will touch on different ways of handling categorical variables.
Different approaches to categorical coding
Then, How should we handle categorical variables? It turns out that there are multiple ways to handle categorical variables. In this article, I will discuss the two most used techniques:
- Label encoding
- One-Hot Coding
Now, let's see them in detail.
Label encoding
Label encoding is a popular coding technique for handling categorical variables. In this technique, each tag is assigned a unique integer in alphabetical order.
Let's see how to implement tag encoding in Python using scikit-learn library and also understand the challenges with tag encoding.
Let's first import the necessary libraries and dataset:
Production:
Understand the data types of functions:
Production:
As you can see here, the first column, Country, is the categorical characteristic since it is represented by the object data type and the rest are numerical characteristics, since they are represented by int64.
Now, let's implement tag encoding in python:
Production:
As you can see here, tag encoding uses alphabetical order. Therefore, India has been coded with 0, EE. UU. With 2 and Japan with 1.
Challenges with tag coding
In the above scenario, the names of the countries do not have an order or rank. But, when tag encoding is done, the names of the countries are classified according to the alphabets. Because of this, there is a very high probability that the model captures the relationship between countries like India <Japan <EE. UU.
This is something we don't want!! Then, How can we overcome this obstacle? Here comes the concept of One-Hot Coding.
One-Hot Coding
One-Hot Encoding is another popular technique for treating categorical variables. Just create additional characteristics based on the number of unique values in the categorical characteristic. Each unique value in the category will be added as a characteristic.
One-Hot Encoding is the process of creating dummy variables.
In this coding technique, each category is represented as a one-hot vector. Let's see how to implement one-hot encoding in Python:
Production:
As you can see here, Add to 3 new functions as the country contains 3 unique values: India, Japan and USA. UU. In this technique, we solved the classification problem since each category is represented by a binary vector.
Can you see any drawbacks with this approach? Think about it before reading on.
Challenges of One-Hot Coding: trap variableIn statistics and mathematics, a "variable" is a symbol that represents a value that can change or vary. There are different types of variables, and qualitative, that describe non-numerical characteristics, and quantitative, representing numerical quantities. Variables are fundamental in experiments and studies, since they allow the analysis of relationships and patterns between different elements, facilitating the understanding of complex phenomena.... ficticia
One-Hot Encoding results in a dummy variable trap, since the outcome of one variable can be easily predicted with the help of the remaining variables.
The dummy variable trap is a scenario in which the variables are highly correlated with each other.
The dummy variable trap leads to the problem known as multicollinearity. Multicollinearity occurs when there is a dependency between independent characteristics. Multicollinearity is a serious problem in machine learning models like linear regression and logistic regression.
Therefore, to overcome the problem of multicollinearity, one of the dummy variables must be discarded. Here I will practically demonstrate how multicollinearity problem is introduced after doing one-hot encoding.
One of the common ways to check for multicollinearity is the variance inflation factor (LIVELY):
- VIF = 1, very less multicollinearity
- LIVELY <5, moderate multicollinearity
- LIVELY> 5, extreme multicollinearity (this is what we have to avoid)
Calculate VIF Scores:
Production:
At the exit, we can see that dummy variables that are created using one-hot encoding have VIF above 5. We have a multicollinearity problem.
Now, let's drop one of the dummy variables to solve the multicollinearity problem:
Production:
Wow! VIF has decreased. We solved the problem of multicollinearity. Now, the dataset is ready to build the model.
I would recommend that you read Going Deeper Into Regression Analysis With Assumptions, graphs and solutions to understand the assumptions of linear regression.
We have seen two different techniques: Label and One-Hot Encoding to handle categorical variables. In the next section, I'll talk about when to prefer Tag Encoding vs. One-Hot Encoding.
When to use tag encoding versus hot encoding
This question usually depends on your dataset and the model you want to apply. But still, a few points to consider before choosing the right coding technique for your model:
We apply One-Hot encoding when:
- The categorical characteristic is not ordinal (like the countries above)
- The number of categorical characteristics is less, so one-hot encoding can be applied effectively
We apply tag encoding when:
- The categorical characteristic is ordinal (like Jr. kg, Sr. kg, primary school, high school)
- The number of categories is quite large as one-hot encoding can lead to high memory consumption.
Final notes
As cited by Jeff Hawkins:
"The key to artificial intelligence has always been representation".
Representation has been the key for developers and from time to time new techniques are emerging to better represent the data and improve the accuracy and learning of our model..
I encourage you to follow the following course to become an expert in machine learning:
