This article was published as part of the Data Science Blogathon.
"Don't get into modeling. First, Understand and explore your data! “
Upon
This is common advice for many data scientists. If your dataset is messy, model building won't help you solve your problem. What will happen is “trash inside, garbage outside”. To build a powerful machine learning algorithm. We need to explore and understand our dataset before defining a predictive task and solving it.
Introduction
before continuing, data scientists spend most of their time exploring, cleaning and preparing your data for modeling. This helps them build accurate models and check the assumptions needed to fit models..
Create meaningful data visualizations, predict future trends from data.
If you are good at understanding data preparation, almost completed 80% from work.
Table of Contents
- Do you ask the right questions?
- Analyze different subsets of data
- Explore the trends
- FFind your blind spots
- Investigate the whys
Ask the right questions
Whether it's survey results, sales data or an email campaign, you have collected data for a specific purpose. By extension, apply this purpose to the questions you ask about the data itself. Starting with a few specific questions can keep your investigation focused and allow you to see the forest through the trees.. A question like “How is my income during the last 3 years?” It is vague and allows exploration, but also the confusion.
However, Something like “which channel generates more income during the last 3 years” has a clearer answer. Subsequent questions can be: “Which department generates the most revenue per year” O “climbing equipment sales are up or down this year?” It is important to keep a specific question in mind when starting data analysis to provide some structure and avoid stumbling upon false positives.
Analyze different subsets of data:
It is easier to detect relationships if you analyze data from different subsets. For instance, segment your revenue data by channel like the chart above, or by department. Experiment with the subsets and variables that make the most sense for the questions you developed in the previous step..
This design focuses on allowing you to stay within your train of thought and smoothly transition from one question to another., without stumbling over the format or equations. It may also be useful to use what would be called a dynamic tablePivotTable is a powerful tool in spreadsheet programs, such as Microsoft Excel and Google Sheets. Allows you to summarize, Analyze and visualize large volumes of data efficiently. Through its intuitive interface, users can rearrange information, apply filters and create custom reports, facilitating informed decision-making in various contexts, from the business field to academic research.... in excel. In our outdoor gear retailer example, you can switch from a quarterly view to revenue for a quarter of the year simply by selecting from a drop down menu. The graph below is an aggregate of revenue for each quarter between 2010 Y 2013.
Explore the trends
Experiment with your time variables. Look at the quarter, the month or the week, whatever makes sense based on what you're looking for. Sometimes, what is missing is also as important as what is there. If there are holes in your data analysis, take note. It can be helpful to take notes throughout your analysis., reminders of what you would like to research or discuss with colleagues later.
Take a look at this quarterly revenue analysis for the department. It is not very useful because it is difficult to detect trends.
This yearly line graph makes it much easier to see that Climbing is the fastest growing department and that Running sales have declined over the last three years..
Find your blind spots
- Has really been collected
for the task you are asked to do. And you are asked to do the
the data validates a result that has already been decided. - Most organizations don't think scientifically. They don't create a hypothesis and then decide what data they need to collect to validate it. They choose a result and then fit the data.
- Often, the data comes from something completely different, often as a by-product of a commercial process. So someone has the brilliant idea “We could use this to work”
-
By analyzing the following graph, graph illustrates information about blind spots in a data set. Hidden data will be one of the drawbacks to obtain a solution. In general, finding outliers will be a solution.
- Outlier correction based on R parameter. The graph on the left shows the original data with detected outliers. The middle graph uses a noise value of zero to place or correct the location of outliers in the linear model. The graph on the right places the outlier near the linear model at a distance based on a positive value for R (R = 0.5).
-
Investigate the whys:
Data analysis is an ongoing process and the best way to approach it is to try to make less and less mistakes. You will probably never have all the data you want or need to answer all the questions about your business., but at least you can move towards more answers and better decisions. This feedback loop continues (to ask, analyze, investigate, repeat) It can improve, but it will never be perfect.
Final notes
Understanding and interpreting data is a very important step in machine learning. In this blog post, we try to provide an overview of techniques that can help you better understand your data
Depending on the size, dimension"Dimension" It is a term that is used in various disciplines, such as physics, Mathematics and philosophy. It refers to the extent to which an object or phenomenon can be analyzed or described. In physics, for instance, there is talk of spatial and temporal dimensions, while in mathematics it can refer to the number of coordinates necessary to represent a space. Understanding it is fundamental to the study and... and type of their data, you can choose the algorithm. For instance, when you have big raw data, you can use representative examples instead of random samples. If you have a large data set, you can also find the important dimensions to understand representative samples.
Different techniques can give you different insights about your data. It is your job to use the tools to solve the mystery as a detective..
