This article was published as part of the Data Science Blogathon.
Introduction
“It's part of the content editor internship”
“Every time I go to the movies, it's magic, no matter what it is”. – Steven Spielberg
Everybody loves movies, regardless of your age, sex, race, color or geographic location. All, somehow, we are connected to each other through this incredible medium. But nevertheless, the most interesting thing is the fact that unique our choices and combinations are in terms of movie preferences. Some people like genre-specific movies, either suspense, romance or science fiction, while others focus on the main actors and directors. When we take all that into account, it's amazingly difficult to generalize a movie and say that everyone would like it. But with all that said, Similar movies are still seen to be liked by a specific part of society.
So this is where we, as data scientists, we come into play and extract the juice from all behavior patterns not only from the audience but also from the movies themselves. Then, without any more preambles, let's get straight to the basics of a recommendation system.
What is a recommendation system?
Just put a Recommender system is a filtering program whose main objective is to predict the “qualification” o la “preference” from a user to a specific element or element of the domain. In our case, this domain specific item is a movie, Thus, the main focus of our recommendation system is to filter and predict only those movies that a user would prefer given some data about the user himself.
-
content-based filtering
This filtering strategy is based on the data provided about the articles. The algorithm recommends products that are similar who liked a user in the last. This similarity (generally cosine similarity) is calculated from the data we have about the elements, as well as past user preferences.
For instance, if a user likes movies like 'The Prestige’ then we can recommend you the ‘Christian Bale movies’ or films of the genre ‘Thriller’ or maybe even movies directed by ‘Christopher Nolan’. The recommendation system checks the user's past preferences and finds the movie “The prestige”, then try to find movies similar to the one using the available information in the databaseA database is an organized set of information that allows you to store, Manage and retrieve data efficiently. Used in various applications, from enterprise systems to online platforms, Databases can be relational or non-relational. Proper design is critical to optimizing performance and ensuring information integrity, thus facilitating informed decision-making in different contexts...., as the main actors, director, the genre of the film, the production house, etc y, based on this information, Look for movies like “The Prestige”.
Disadvantages
- Different products don't get much exposition to user.
- Businesses cannot be expanded because the user does not try different types of products.
-
Collaborative filtering
This filtration strategy is based on combining user behavior and comparing and contrasting it with Other users behavior in the database. The story of all users plays an important role in this algorithm. The main difference between content-based filtering and collaborative filtering is that in the latter, the interaction of all users with articles influences the recommendation algorithm, while for content-based filtering only data of the interested user is taken into account.
There are several ways to implement collaborative filtering, but the main concept to understand is that in collaborative filtering multiple User data influences recommendation outcome. and it does not depend on only one user data to model.
There is 2 types of collaborative filtering algorithms:
-
Collaborative user-based filtering
The basic idea here is to find users who have similar previous preference patterns How has the user ‘A’ and then recommend items that are liked to those users similar to 'A’ still haven't found. This is accomplished by making a array of items that each user has rated, seen, like or clicked depending on the task at hand, and then calculate the similarity score between users and finally recommend items that the user in question does not know, but that to users similar to him / they do like her.
For instance, yes to user 'A’ he likes 'Batman Begins', ‘Justice League’ y ‘The Avengers’ while user 'B’ he likes 'Batman Begins', ‘Justice League’ and ‘Thor’, so they have similar interests because we know that these movies belong to the superhero genre. Therefore, there is a high probability that the user ‘A’ like 'Thor’ and to user 'B’ you like The Avengers'.
Disadvantages
- People are voluble namely, your taste changes from time to time and as this algorithm is based on the similarity of the user, can detect initial similarity patterns between 2 users who after a while may have completely different preferences.
- There are many more users than elements Thus, it is very difficult to maintain such large matrices and, Thus, they need to be recalculated very regularly.
- This algorithm is very susceptible to shilling attacks where fake user profiles consisting of biased preference patterns are used to manipulate key decisions.
-
Collaborative element-based filtering
The concept in this case is search for similar movies instead of similar users and then recommend movies similar to the ones that ‘A’ has had in your past preferences. This is done by finding every pair of items that were rated / visas / they like me / clicked by the same user, then measuring the similarity of those rated / visas / liked / clicked on all users who rated / they saw / I liked them / they clicked on both, and finally recommending them based on the similarity scores.
Here, for instance, we take 2 movies' A’ and 'B’ and we check your ratings from all users who have rated both movies and based on the similarity of these ratings, and based on this similarity of rating by users who have rated both, we find similar movies. Then, if the most common users have rated 'A’ and 'B’ similarly and it is very likely that 'A’ and 'B’ are similar, Thus, if someone has seen and liked 'A', he should be recommended 'B’ and vice versa.
Advantages over collaborative user-based filtering
- Unlike the taste of the people, the movies don't change.
- There are usually many fewer articles than people, Thus, it is easier to maintain and calculate the matrices.
- Shilling attacks are much more difficult because items cannot be counterfeited.
-
Let's start coding our own movie recommendation system.
In this implementation, when the user searches for a movie, we will recommend the 10 best similar movies using our movie recommendation system. We will use collaborative element-based filtering algorithm for our purpose. The data set used in this demonstration is the movielens-small data set.
Put the data to work
First, we need to import libraries that we will use in our movie recommendation system. What's more, we will import the dataset by adding the path of the CSV records.
import pandas as pd import numpy as np from scipy.sparse import csr_matrix from sklearn.neighbors import NearestNeighbors import matplotlib.pyplot as plt import seaborn as sns movies = pd.read_csv("../input/movie-lens-small-latest-dataset/movies.csv") ratings = pd.read_csv("../input/movie-lens-small-latest-dataset/ratings.csv")
Now that we have added the data, Let's take a look at the files using the dataframe.head () command to print the first 5 dataset rows.
Let's take a look at the movie dataset:
movies.head()
The movie data set has
- movieId: once the recommendation is made, we get a list of all similar movieIds and get the title of each movie from this dataset.
- genders – What is it not required for this filtering approach.
ratings.head()
The grade data set has
- userId: unique for each user.
- movieId: with this function, we take the movie title from the movie dataset.
- rating – Ratings given by each user to all movies using this, we are going to predict the 10 best similar movies.
Here, we can see that userId 1 has Seen movieId 1 Y 3 and both scored with 4.0, but it has Not Rated movieId 2 absolutely. This interpretation is more difficult to extract from this data frame. Therefore, to make things easier to understand and work with, we are going to create a new data frame where each column would represent each unique user ID and each row would represent each unique movie ID.
final_dataset = ratings.pivot(index='movieId',columns="userId",values="rating") final_dataset.head()
Now, it is much easier to interpret than userId 1 rated movieId 1 & 3 4.0 but it has not rated movieId 3,4,5 absolutely (Thus, are represented as NaN) Y, Thus, your rating data is missing.
Let's fix this and impute NaN con 0 to make things understandable to the algorithm and also make the data more reassuring to the eye.
final_dataset.fillna(0,inplace=True) final_dataset.head()
Remove noise from data
In the real world, the grades are very scarce and the data points are collected primarily from very popular movies and highly engaged users. We do not want movies that have been rated by a small number of users because it is not credible enough. In the same way, users who have rated only a handful of movies it should not be taken into account either.
Then, with all that taken into account and some trial and error experiments, we will reduce the noise by adding some filters for the final dataset.
- To rate a movie, a minimum of 10 users should have voted a movie.
- To rate a user, a minimum of 50 the movies should have voted for the user.
Let's visualize what these filters look like
Adding the number of users who voted and the number of movies that were voted.
no_user_voted = ratings.groupby('movieId')['rating'].agg('count')
no_movies_voted = ratings.groupby('userId')['rating'].agg('count')
Let's visualize the number of users who voted with our threshold of 10.
f,ax = plt.subplots(1,1,figsize=(16,4))
# ratings['rating'].plot(kind='hist')
plt.scatter(no_user_voted.index,no_user_voted,color="mediumseagreen")
plt.axhline(y = 10, color ="r")
plt.xlabel('MovieId')
plt.ylabel('No. of users voted')
plt.show()
Make the necessary modifications according to the established threshold.
final_dataset = final_dataset.loc[no_user_voted[no_user_voted > 10].index,:]
Let's visualize the number of votes of each user with our threshold of 50.
f,ax = plt.subplots(1,1,figsize=(16,4))
plt.scatter(no_movies_voted.index,no_movies_voted,color="mediumseagreen")
plt.axhline(y = 50, color ="r")
plt.xlabel('UserId')
plt.ylabel('No. of votes by user')
plt.show()
Carrying out the necessary modifications according to the established threshold.
final_dataset=final_dataset.loc[:,no_movies_voted[no_movies_voted > 50].index] final_dataset
Eliminate the shortage
Our final_dataset has dimensions of 2121 * 378 where most values are scarce. We are using only a small data set, but for the original large film lens data set which has more than 100000 features, our system may run out of computational resources when fed to the model. To reduce the dispersion we use the csr_matrix function from the scipy library.
I will give an example of how it works:
sample = np.array([[0,0,3,0,0],[4,0,0,0,2],[0,0,0,0,1]]) sparsity = 1.0 - ( np.count_nonzero(sample) / float(sample.size) ) print(sparsity)
csr_sample = csr_matrix(sample) print(csr_sample)
As you can see, there is no sparse value in csr_sample and the values are assigned as indexThe "Index" It is a fundamental tool in books and documents, which allows you to quickly locate the desired information. Generally, it is presented at the beginning of a work and organizes the contents in a hierarchical manner, including chapters and sections. Its correct preparation facilitates navigation and improves the understanding of the material, making it an essential resource for both students and professionals in various areas.... of rows and columns. for the row 0 and the second column, the value is 3.
Applying the csr_matrix method to the dataset:
csr_data = csr_matrix(final_dataset.values) final_dataset.reset_index(inplace=True)
Model the movie recommendation system
We will use the KNN algorithm to calculate the similarity with cosine distance metric which is very fast and more preferable than pearson's coefficient.
knn = NearestNeighbors(metric="cosine", algorithm='brute', n_neighbors=20, n_jobs=-1) knn.fit(csr_data)
Doing the recommendation function
The principle of operation is very simple. First we check if the movie name entry is in the database and if it is, we use our recommendation system to find similar movies and sort them based on their similarity distance and generate only the top 10 movies with their distances from the input movie.
def get_movie_recommendation(movie_name):
n_movies_to_reccomend = 10
movie_list = movies[movies['title'].str.contains(movie_name)]
if len(movie_list):
movie_idx= movie_list.iloc[0]['movieId']
movie_idx = final_dataset[final_dataset['movieId'] == movie_idx].index[0]
distances , indices = knn.kneighbors(csr_data[movie_idx],n_neighbors=n_movies_to_reccomend+1)
rec_movie_indices = sorted(list(zip(indices.squeeze().tolist(),distances.squeeze().tolist())),key=lambda x: x[1])[:0:-1]
recommend_frame = []
for val in rec_movie_indices:
movie_idx = final_dataset.iloc[val[0]]['movieId']
idx = movies[movies['movieId'] == movie_idx].index
recommend_frame.append({'Title':movies.iloc[idx]['title'].values[0],'Distance':val[1]})
df = pd.DataFrame(recommend_frame,index=range(1,n_movies_to_reccomend+1))
return df
else:
return "No movies found. Please check your input"
Finally, we will recommend some movies!
get_movie_recommendation('Iron Man')
Personally, I think the results are pretty good. All the movies at the top are superhero or animation films that are ideal for children such as the entry film “Iron Man”.
Let's try another:
get_movie_recommendation('Memento')
All top movies 10 son serious and conscientious movies like "Memento" itself, so I think the result, in this case, It's also good.
Our model works quite well: a movie recommendation system based on user behavior. Therefore, we conclude our collaborative filtering here. You can get the full deployment notebook here.
