Introduction
How can they scan everything over the internet and return relevant results in just Approximately 5,43,00,000 results (0,004 seconds). Good, work on the concept of Crawling Y Indexing.
- Crawling: Automated bots search for new or updated pages. And store the key information like: URL, qualification, keywords, etc. of the pages to be used later.
- Indexing: The data captured from the trace is analyzed as: what is the page about. The key content, the images and video files on the page are used in the procedure. This information is indexed and stored to be returned later for a search query.
Therefore, every time we ask them to find something for us, they are not scanning the length and breadth of the internet, but just scanning those indexed urls in step 2.
Good, today we would work on how to develop a small prototype, very equivalent to the indexing functionality of any search engine. We would be using a dataset of tweets in #COVID and we would try to index them based on our search term.
A. Package import
import pandas as pd from rank_bm25 import *
What is BM25?
BM25 is a simple Python package and can be used to index the data, tweets in our case, based on the search query. Works on TF concept / IDF, In other words.
- TF or termination frequency: briefly, indicates the number of occurrences of the search term in our tweet
- Reverse document frequency or IDF: measures the relevance of your search term. Since TF considers that all terms are equally important, therefore, we cannot use only term frequencies to calculate the weight of a definition in its text. We would have to weigh the frequent terms while expanding on the rare terms that show their relevance to the tweet.
Once you run the query, BM25 will show the relevance of your search term with each of the tweets. You can sort them to index the most relevant ones.
B. Preparing your tweets
Since this is not a discussion on the Twitter API, start using an Excel-based feed. You can clean up your text data in these key steps to make your search more robust.
1. Tokenización:
Divide the sentence into words. So that each word can be considered uniquely.
from nltk.tokenize import word_tokenize sentence = “Jack is a sharp minded fellow” words = word_tokenize(sentence) print(words)
Output: ['Jack', 'sharp', 'minded', 'fellow']
2. Delete special characters:
Delete special characters from your tweets
def spl_chars_removal(lst):
lst1=list()
for element in lst:
str=””
str = re.sub(“[⁰-9a-zA-Z]”,” “,element)
lst1.append(str)
return lst1
3. Delete empty words:
Empty words are common words (it is, to, the, etc.) in tweets. These words do not mean any relevance since they do not help to distinguish two tweets. used to Gensim pack to erase my irrelevant words, you can also test it using nltk, but I found Gensim much faster than others.
In addition, new words can easily be added to the stopword list., in case your data is specifically surrounded by those words and they occur many times.
#adding words to stopwords from nltk.tokenize import word_tokenize from gensim.parsing.preprocessing import STOPWORDS
#adding custom words to the pre-defined stop words list all_stopwords_gensim = STOPWORDS.union(set([‘disease’]))
def stopwprds_removal_gensim_custom(lst):
lst1=list()
for str in lst:
text_tokens = word_tokenize(str)
tokens_without_sw = [word for word in text_tokens if not word in all_stopwords_gensim]
str_t = “ “.join(tokens_without_sw)
lst1.append(str_t)
return lst1
4. NormalizationStandardization is a fundamental process in various disciplines, which seeks to establish uniform standards and criteria to improve quality and efficiency. In contexts such as engineering, Education and administration, Standardization makes comparison easier, interoperability and mutual understanding. When implementing standards, cohesion is promoted and resources are optimised, which contributes to sustainable development and the continuous improvement of processes....:
Text normalization is the procedure of transforming a text into a canonical form (standard). As an example, the word “good” Y “God” can be transformed into “good”, its canonical form. Another example is the mapping of nearly identical words like “empty words”, “empty words” Y “empty words” to only “empty words”.
This technique is essential for noisy texts such as social media comments., text messages and comments on blog posts where abbreviations prevail, spelling mistakes and the use of words that are not in the vocabulary (oov). People tend to write comments in shorthand and, therefore, this pre-processing becomes very important.
| Crude | Standardized |
| Yes, yday | yesterday |
| tomo, 2moro, 2mrw, tmrw | morning |
| brb | I'll be right back |
5. Derivative:
Procedure for transforming words to their root form. It is the procedure of reducing the inflection of words (as an example, problems, problems) to its root form (as an example, problems). The “root” in this circumstance it may not be a real root word, but simply a canonical form of the original word.
Stemming uses a heuristic procedure that cuts off the ends of words in the hope of correctly transforming the words into their root form. Must be reviewed, since in the following example you can see that “Machine” It transforms in “Cars”, “e” gets cut in bypass procedure..
import nltk
from nltk.stem
import PorterStemmer
ps = PorterStemmer() sentence = “Machine Learning is cool”
for word in sentence.split():
print(ps.stem(word))
Production: [‘Car’, ‘Learning’, ‘cool’]
C. Tokenization of tweets and execution of BM25
This is the centerpiece where we run the search query. We search for tweets based on the word “vaccine” user-based. Additionally you can enter a phrase and it will do so fluently as we convert our search term to the second line below.
tokenized_corpus = [doc.split(” “) for doc in lst1]
bm25 = BM25Okapi (tokenized_body)
query = "vaccine" ## Enter your search query
tokenized_query = query.split (“”)
You can check the association of each tweet with your search term using .get_scores function.
doc_scores = bm25.get_scores(tokenized_query) print(doc_scores)
To enter n = 5 in .get_top_n we would get five more associated tweets as our result. You can put the value of n according to your needs.
docs = bm25.get_top_n (tokenized_query, lst1, n = 5)
df_search = df[df[‘Text’].ray (docs)]
df_search.head ()
D. Top Five Associated Tweets
| Top 5 tweets | Tweeted by |
| @ MikeCarlton01 Regarding #ABC funding, searched the budget documents. After previous massive cuts, got $ 4.7 additional million in funds (.00044% much less than inflation).
#Morrison wastes $ More in expensive & amp; ineffective services, as an example, the useless #Covid app; #agedcare delivery vaccine; vaccine implementation consultancies. |
MORRIGAN |
| @TonyHWindsor @barriecassidy @ 4corners @ abc730 For their invaluable work, #ABC got $ 4.7Extra M in funds (.00044% much less than inflation).
While #Morrison Govt spends like a drunken sailor buying at expensive prices & amp; ineffective peer services (p. not., useless #Covid app; #agedcare vaccine delivery; vaccine launch) #auspol |
MORRIGAN |
| It's going to be a month after my #Covid recovery. Now I will go to vaccinate 😎😎😎😎 | Simi Elizabeth😃 |
| RT @pradeepkishan: What a despicable politician is #ArvindKejariwal! The moment the accumulation of oxygen came to light, their propaganda turned to vaccine shortages. It's more dangerous than #COVID itself! @BJP4India @TajinderBagga | p.hariharan |
| RT @AlexBerenson: TL: DR – At @pfizer's Teen #Covid Vaccine Trial, 4 O 5 (the exact figure is hidden) of 1,100 children who got the vaccine had serious side effects, in comparison with 1 who received placebo.
@US_FDA did not reveal specific details, so we have no idea what they were or if they follow any pattern. https://t.co/n5igf2xXFN |
Wisdom |
E. Additional BM25 Use Cases
There can be many use cases where a search function is required. One of the most relevant is the analysis of the PDF and the development of a search function on the content of the PDF..
This is one of the most used cases for BM25. As the world slowly shifts towards better data strategy and efficient storage techniques, old PDF documents can be efficiently recovered using algorithms like BM25.
Hope you enjoyed reading this and found it helpful. Thanks friends!
Reference Links
The media shown in this post is not the property of DataPeaker and is used at the author's discretion.
