What is tokenization? Tokenization in NLP

Reflexes

• Tokenization is a key aspect (and mandatory) of working with text data
• We will discuss the various nuances of tokenization, including how to handle words outside the vocabulary (OOV)

Introduction

Language is a beautiful thing. But mastering a new language from scratch is a pretty daunting prospect.. If you ever learned a language that was not your mother tongue, Will identify with this! There are so many layers to remove and syntax to consider, it's quite a challenge.

And so it is exactly with our machines. So that our computer understands any text, we need to break down that word in a way that our machine can understand. That's where the concept of tokenization comes in in natural language processing. (NLP).

In a nutshell, we can't work with text data if we don't do tokenization. Yes, it's really that important!

And here's the intriguing thing about tokenization: It is not solo about breaking down the text. Tokenization plays an important role in handling text data. Then, in this article, We will explore the depths of tokenization in natural language processing and how you can implement it in Python.

I recommend taking some time to review the following resource if you are new to NLP.:

Table of Contents

1. A quick overview of tokenization
2. The real reasons behind tokenization
3. What tokenization (word, character or subword) we should use?
4. Tokenization implementation: byte pair encoding in Python

A quick overview of tokenization

Tokenization is a common task in natural language processing (NLP). It is a fundamental step in traditional NLP methods such as Count Vectorizer and Advanced Deep Learning-based architectures such as Transformers.

Tokens are the building blocks of natural language.

Tokenization is a way of separating a piece of text into smaller units called tokens. Here, tokens can be words, characters or subwords. Therefore, Tokenization can be broadly classified into 3 types: word tokenization, character and subword (n-gram characters).

For instance, consider the sentence: “Never give up”.

The most common way of forming tokens is space-based. Assuming space is a delimiter, sentence tokenization results in 3 tokens: Never give up. As each token is a word, becomes an example of Word tokenization.

Similarly, tokens can be characters or subwords. For instance, let's consider “smartest”:

1. Character Tokens: smartest
2. Subword tabs: smartest

But then is this necessary? Do we really need tokenization to do all of this?

Note: If you are new to NLP, check our Online NLP course

The real reasons behind tokenization

Since tokens are the building blocks of Natural Language, the most common way to process plain text occurs at the token level.

For instance, Transformer-based models, The architectures of deep learningDeep learning, A subdiscipline of artificial intelligence, relies on artificial neural networks to analyze and process large volumes of data. This technique allows machines to learn patterns and perform complex tasks, such as speech recognition and computer vision. Its ability to continuously improve as more data is provided to it makes it a key tool in various industries, from health... Cutting-edge (UNDER) and NLP, process the raw text at the token level. Similarly, the most popular deep learning architectures for NLP like RNN, GRU and LSTM also process raw text at the token level.

As shown here, RNN receives and processes each token in a particular time step.

Therefore, Tokenization is the most important step when modeling text data. Tokenization is done in the corpus to obtain tokens. The following cards are then used to prepare a vocabulary. The vocabulary refers to the set of unique tokens in the corpus. Remember that the vocabulary can be built considering each unique token in the corpus or considering the K most frequent words.

Building vocabulary is the ultimate goal of tokenization.

One of the simplest tricks to improve the performance of the NLP model is to create a vocabulary from the K most frequent words..

Now, understand the use of vocabulary in traditional and advanced NLP methods based on deep learning.

• Traditional NLP approaches, como Count Vectorizer y TF-IDF, use vocabulary as characteristics. Each vocabulary word is treated as a unique characteristic:

• In NLP architectures based on advanced deep learning, the vocabulary is used to create the tokenized input sentences. Finally, the tokens of these sentences are passed as inputs to the model.

What tokenization should I use?

As mentioned earlier, tokenization can be done at the word level, character or subword. It's a common question: What tokenization should we use when solving a NLP task? Let's tackle this question here.

Word tokenization

Word tokenization is the most widely used tokenization algorithm. Divide a piece of text into individual words based on a specified delimiter. Depending on the delimiters, different word level tokens are formed. Pre-trained word embeds like Word2Vec and GloVe are included in word tokenization.

But this has some drawbacks.

Drawbacks of word tokenization

One of the main problems with word tokens is dealing with Words without vocabulary (OOV). The OOV words refer to the new words found in the tests. These new words do not exist in the vocabulary. Therefore, these methods fail to handle OOV words.

But wait, Don't jump to conclusions just yet!

• A little trick can rescue word tokenizers from OOV words. The trick is to build the vocabulary with the most frequent K words and replace the rare words in the data of trainingTraining is a systematic process designed to improve skills, physical knowledge or abilities. It is applied in various areas, like sport, Education and professional development. An effective training program includes goal planning, regular practice and evaluation of progress. Adaptation to individual needs and motivation are key factors in achieving successful and sustainable results in any discipline.... with unknown tokens (UNK). This helps the model to learn the representation of OOV words in terms of UNK tokens
• Therefore, during the test time, any word that is not present in the vocabulary will be mapped to a UNK token. This is how we can tackle the OOV problem in word tokenizers.
• The problem with this approach is that all the word information is lost when we are mapping OOV to UNK tokens. The structure of the word can be useful to represent it accurately. And another problem is that every OOV word has the same representation

Another problem with word tokens is related to the size of the vocabulary. Generally, pre-trained models are trained on a large volume of the text corpus. Then, imagine building vocabulary with all the unique words in such a large corpus. This explodes the vocabulary!

This opens the door to character tokenization.

Character Tokenization

Character tokenization splits each text into a character set. Overcomes the drawbacks we saw earlier about word tokenization.

• Character tokenizers handle OOV words consistently by preserving word information. Divide the word OOV into characters and represent the word in terms of these characters.
• It also limits the size of the vocabulary. Do you want to guess the size of the vocabulary? 26 since the vocabulary contains a unique set of characters

Drawbacks of character tokenization

Character tokens solve the OOV problem, but the length of the input and output sentences increases rapidly as we render a sentence as a sequence of characters. As a result, it becomes a challenge to learn the relationship between the characters to form meaningful words.

This brings us to another tokenization known as subword tokenization, which is between a tokenization of words and characters.

Subword Tokenization

Subword tokenization splits the text snippet into subwords (or n-gram characters). For instance, words like lowest can be segmented as lowest, smarter as smarter, etc.

Transform-based models, the SOTA in NLP, rely on subword tokenization algorithms to prepare vocabulary. Now, I will discuss one of the most popular subword tokenization algorithms known as Byte Pair Encoding (BPE).

Welcome to Byte Pair Encoding (BPE)

Byte pair encoding (BPE) is a widely used tokenization method among transformer-based models. BPE addresses the problems of character and word tokenizers:

• BPE addresses OOV effectively. Segment OOV as subwords and represent the word in terms of these subwords.
• The length of input and output sentences after BPE is shorter compared to character tokenization

BPE is a segmentationSegmentation is a key marketing technique that involves dividing a broad market into smaller, more homogeneous groups. This practice allows companies to adapt their strategies and messages to the specific characteristics of each segment, thus improving the effectiveness of your campaigns. Targeting can be based on demographic criteria, psychographic, geographic or behavioral, facilitating more relevant and personalized communication with the target audience.... of words that merges the most frequently occurring characters or sequences of characters iteratively. Here is a step by step guide to learn BPE.

Steps to learn BPE

1. Divide the words in the corpus into characters after adding
2. Initialize the vocabulary with unique characters in the corpus.
3. Calculate the frequency of a pair of characters or sequences of characters in the corpus
4. Merge the most frequent pair in corpus
5. Save the best pair in vocabulary
6. Repeat steps 3 a 5 for a certain number of iterations

We will understand the steps with an example.

Consider a corpus:

1a) Add the symbol at the end of the word (say it ) to each word of the corpus:

1b) Tokenize the words of a corpus in characters:

2. Initialize the vocabulary:

Iteration 1:

3. Calculate frequency:

4. Merge the most frequent pair:

5. Save the best pair:

Repeat steps 3-5 for each iteration from now on. Let me illustrate one more iteration.

Iteration 2:

3. Calculate frequency:

4. Merge the most frequent pair:

5. Save the best pair:

After 10 iterations, BPE merge operations look like this:

Pretty straightforward, truth?

Apply BPE to OOV palabras

But, How can we represent the word OOV at test time using learned operations BPE? Any ideas? Let's answer this question now.

At the time of testing, the word OOV is divided into character sequences. Later, learned operations are applied to merge characters into larger known symbols.

– Neural machine translation of rare words with subword units, 2016

Then, a step-by-step procedure for rendering OOV words is shown:

1. Divide the word OOV into characters after adding
2. Calculate a couple of characters or sequences of characters in a word
3. Select the pairs present in the learned operations
4. Merge the most frequent pair
5. Repeat steps 2 Y 3 until it is possible to merge

Let's see all of this in action below!!

Tokenization implementation: byte pair encoding in Python

Now we are aware of how BPE works: learn and apply OOV words. Then, it's time to implement our knowledge in Python.

Python code for BPE is now available in the original document (Neural machine translation of rare words with subword units, 2016)

Reading corpus

We will consider a simple corpus to illustrate the idea of ​​BPE. But nevertheless, the same idea also applies to another corpus:

Preparation of texts

Tokenize the words in characters in the corpus and add at the end of each word:

Learning BPE

Calculate the frequency of each word in the corpus:

Production:

Let's define a function to calculate the frequency of a pair of characters or sequences of characters. Accepts the corpus and returns the pair with its frequency:

Now, the next task is to merge the most frequent pair of the corpus. We will define a function that accepts the corpus, best pair and returns the modified corpus:

Then, it's time to learn BPE operations. How BPE is an iterative procedure, we will perform and understand the steps for an iteration. Let's calculate the frequency of the bigrams:

Production:

Find the most frequent pair:

Production: (‘e’, ‘s’)

Finally, match the best pair and save it to vocabulary:

Production:

We will follow similar steps for certain iterations:

Production:

The most interesting part is yet to come!! That's applying BPE to OOV words.

Apply BPE to the OOV word

Now, we will see how to segment the word OOV into subwords using learned operations. Consider the word OOV as “more low”:

Applying BPE to an OOV word is also an iterative process. We will implement the steps discussed earlier in the article:

Production:

As you can see here, the unknown word “more low” is segmented as lowest.

Final notes

Tokenization is a powerful way of dealing with text data. We saw a glimpse of that in this article and also implemented tokenization using Python.

Go ahead and test this on any text-based dataset you have. The more i practice, the better your understanding of how tokenization works (and why is it such a critical NLP concept). Feel free to reach out to me in the comments below if you have any questions or ideas about this article..

Related