This article was published as part of the Data Science Blogathon
Data preprocessing
It is also an important step in data mining, since we cannot work with raw data. Data quality should be verified before applying machine learning or data mining algorithms.
Why is data preprocessing important?
Data pre-processing is primarily to check the quality of the data. The quality can be checked by the following
- Precision: To check if the data entered is correct or not.
- I complete it: To check whether the data is available or not recorded.
- Consistency: To check whether the same data is saved in all matching places or not.
- Chance: The data must be updated correctly.
- Credibility: Data must be reliable.
- Interpretability: The understandability of the data.
- Data cleansing
- Data integration
- Data reduction
- Data transformation
Source: medium.com
Data cleansing:
Data cleansing is the process of removing bad data, incomplete data and inaccurate data from data sets, and also replace missing values. There are some data cleaning techniques
Handling missing values:
- You can use standard values like “Not available” O “NA” to replace missing values.
- Missing values can also be filled in manually, but not recommended when the data set is large.
- The middle value of the attribute can be used to replace the missing value when the data is normally distributed.
in which, in the case of a non-normal distribution, you can use the median value of the attribute. - When using regression or decision tree algorithms, the missing value can be replaced by the most probable value.
value.
Ruidoso:
Noisy generally means random error or containing unnecessary data points. Here are some of the methods for handling noisy data.
- Binning: This method is for smoothing or handling noisy data. First, the data is sorted and then the ordered values are separated and stored in the form of containers. There are three methods to smooth the container data. Smoothing by bin mean method: In this method, the container values are replaced by the container mean value; Smoothed by medianThe median is a statistical measure that represents the central value of a set of ordered data. To calculate it, the data is organized from lowest to highest and the number in the middle is identified. If there are an even number of observations, the two core values are averaged. This indicator is especially useful in asymmetric distributions, since it is not affected by extreme values.... From bin: In this method, the container values are replaced by the median value; Container Boundary Smoothing: In this method, the minimum and maximum usage values are taken from the location values and the values are replaced by the closest limit value.
- Regression: It is used to smooth the data and will help to handle the data when there is unnecessary data. For analysis, Purpose regression helps to decide the variableIn statistics and mathematics, a "variable" is a symbol that represents a value that can change or vary. There are different types of variables, and qualitative, that describe non-numerical characteristics, and quantitative, representing numerical quantities. Variables are fundamental in experiments and studies, since they allow the analysis of relationships and patterns between different elements, facilitating the understanding of complex phenomena.... that is suitable for our analysis.
- Grouping: Used to find outliers and also to group data. Clustering is generally used in the Unsupervised learningUnsupervised learning is a machine learning technique that allows models to identify patterns and structures in data without predefined labels. Through algorithms such as k-means and principal component analysis, This approach is used in a variety of applications, such as customer segmentation, anomaly detection and data compression. Its ability to reveal hidden information makes it a valuable tool in the....
Data integration:
The process of combining multiple sources into a single dataset. The data integration process is one of the main components in data management. There are a few issues to be aware of during data integration.
- Schema integration: Integrate metadata (a data set that describes other data) from different sources.
- Entity identification problem: Identification of entities from multiple databases. For instance, The system or usage must know the student _id of a databaseA database is an organized set of information that allows you to store, Manage and retrieve data efficiently. Used in various applications, from enterprise systems to online platforms, Databases can be relational or non-relational. Proper design is critical to optimizing performance and ensuring information integrity, thus facilitating informed decision-making in different contexts.... and the student name from another database belongs to the same entity.
- Detect and solve data value concepts: Data taken from different databases during the merge may differ. How attribute values in one database may differ from another database. For instance, the date format may differ as “MM / DD / YYYY” O “DD / MM / YYYY”.
Data reduction:
This process helps reduce the volume of data, which facilitates analysis and produces the same or almost the same result. This reduction also helps reduce storage space.. Some of the techniques in data reduction are Dimensionality reduction, Numerality reduction, Data compression.
- Dimensionality reduction: This process is necessary for real world applications, since the data size is large. In this process, the reduction of attributes or random variables is done so that the dimensionality of the data set can be reduced. Combine and merge the attributes of the data without losing its original characteristics. This also helps reduce storage space and computing time.. When data is very dimensional, the problem called “The curse of dimensionality”.
- Reduction in Numerousity: In this method, data representation becomes smaller as volume is reduced. There will be no data loss in this reduction.
- Data compression: The compressed form of the data is called data compression. This compression can be lossless or lossy. When there is no loss of information during compression, is called lossless compression. While lossy compression reduces information, but it only removes the unnecessary information.
Data transformation:
The change made to the format or structure of the data is called a data transformation. This step can be simple or complex depending on the requirements. There are some methods in data transformation.
- Smoothing: With the help of algorithms, we can remove the noise from the dataset and help to know the important characteristics of the dataset. By smoothing we can find even a simple change that helps in the prediction.
- Aggregation: In this method, data is stored and presented in summary form. The data set that comes from multiple sources is integrated with the description of the data analysis. This is an important step as the accuracy of the data depends on the quantity and quality of the data.. When the quality and quantity of data are good, the results are more relevant.
- Discretization: Continuous data here is divided into intervals. Discretization reduces data size. For instance, instead of specifying the class time, we can set an interval like (3 pm-5 pm, 6 pm-8 pm).
- NormalizationStandardization is a fundamental process in various disciplines, which seeks to establish uniform standards and criteria to improve quality and efficiency. In contexts such as engineering, Education and administration, Standardization makes comparison easier, interoperability and mutual understanding. When implementing standards, cohesion is promoted and resources are optimised, which contributes to sustainable development and the continuous improvement of processes....: It is the method of scaling the data so that it can be represented in a smaller range. Example that goes from -1.0 a 1.0.
Data preprocessing steps in machine learning
Import libraries and the dataset
import pandas as pd import numpy as np dataset = pd.read_csv('Datasets.csv') print (data_set)
Extracting independent variable:
Extracting dependent variable:
Fill the dataset with the mean value of the attribute
from sklearn.preprocessing import Imputer
impute = impute(missing_values="NaN", strategy='mean', axis = 0)
imputerimputer = imputer.fit(x[:, 1:3])
x[:, 1:3]= imputer.transform(x[:, 1:3])
x
Coding of the country variable
Machine learning models use mathematical equations. Then, categorical data is not accepted, so we convert them into numerical form.
from sklearn.preprocessing import LabelEncoder label_encoder_x= LabelEncoder() x[:, 0]= label_encoder_x.fit_transform(x[:, 0])
Dummy coding
These dummy variables replace categorical data as 0 Y 1 in the absence or presence of specific categorical data.
Coding of the purchased variable
labelencoder_y= LabelEncoder() y= labelencoder_y.fit_transform(Y)
Split the dataset into a set of trainingTraining is a systematic process designed to improve skills, physical knowledge or abilities. It is applied in various areas, like sport, Education and professional development. An effective training program includes goal planning, regular practice and evaluation of progress. Adaptation to individual needs and motivation are key factors in achieving successful and sustainable results in any discipline.... and test:
from sklearn.model_selection import train_test_split x_train, x_test, y_train, y_test = train_test_split(x, Y, test_size= 0.2, random_state=0)
Feature scale
from sklearn.preprocessing import StandardScaler
st_x= StandardScaler() x_train= st_x.fit_transform(x_train)
x_test= st_x.transform(x_test)
Conclution:
In this article, I have explained about the most crucial step in machine learning is data preprocessing. Hope this article helps you better understand the concept.
Reference:
https://www.javatpoint.com/data-preprocessing-machine-learning
The media shown in this article is not the property of DataPeaker and is used at the author's discretion.
