This article was published as part of the Data Science Blogathon.
Introduction
The first step towards resolutionThe "resolution" refers to the ability to make firm decisions and meet set goals. In personal and professional contexts, It involves defining clear goals and developing an action plan to achieve them. Resolution is critical to personal growth and success in various areas of life, as it allows you to overcome obstacles and keep your focus on what really matters.... Problem Solving in Data Science Projects It's not about building machine learning models. Yes, you read it right!
That distinction belongs to the generation of hypotheses, the step where our problem solving skills are combined with our business intuition. It is a truly crucial step in ensuring a successful data science project.
Let's be honest: we all think of a hypothesis almost every day. Consider the example of a famous sport in India: cricket. It's that time of year when IPL fever is high and we're all engrossed in predicting the winner..
If you've been guessing which team would win based on various factors, such as the size of the stadium and the batters present in the team with six batting abilities or batters with high averages of T20, Congratulations to all. You have all been making an educated guess and generating hypotheses based on your domain knowledge of the sport..
In the same way, the first step in solving any business problem using machine learning is hypothesis generation. Understanding the problem statement with a good knowledge of the domain is important and formulating a hypothesis will further expose you to new problem-solving ideas..
Then, in this article, Let's dive into what hypothesis generation is and find out why it matters to all data scientists.
Table of Contents
- What is hypothesis generation?
- Hypothesis generation vs hypothesis testing
- How does hypothesis generation help?
- When should hypothesis generation?
- Case study: Generating hypotheses with predicting the length of the New York City taxi ride
What is hypothesis generation?
Hypothesis The generation is one “assumption” informed of various factors that are affecting the business problem that needs to be solved by machine learning. When formulating a hypothesis, the data scientist should not know the result of the hypothesis that has been generated based on any evidence.
"A hypothesis can be defined simply as an assumption. A scientific hypothesis is an intelligent assumption “. – Isaac asimov
Hypothesis generation is a crucial step in any data science project. If you skip this or review it, the probability that the project will fail increases exponentially.
Hypothesis generation vs. Hypothesis testing
This is a very common mistake that data science beginners make.
Hypothesis generation is a process that begins with an educated guess, whereas hypothesis testing is a process of concluding that the grounded assumption is true / false or that the relationship between the variables is statistically significant or not.
This last part could be used for future research using statistical tests. A hypothesis is accepted or rejected based on the level of significance and the score of the test used to test the hypothesis.
To understand more about hypothesis testing in detail, you can read it here or you can also learn it through this course.
How does hypothesis generation help?
Here there is 5 key reasons hypothesis generation is so important in data science:
- La generación de hipótesis ayuda a comprender el problema empresarial a medida que profundizamos en la inferencia de los diversos factores que afectan nuestra variableIn statistics and mathematics, a "variable" is a symbol that represents a value that can change or vary. There are different types of variables, and qualitative, that describe non-numerical characteristics, and quantitative, representing numerical quantities. Variables are fundamental in experiments and studies, since they allow the analysis of relationships and patterns between different elements, facilitating the understanding of complex phenomena.... objective.
- You will get a much better idea of what are the main factors that are responsible for solving the problem.
- Data to be collected from various sources that are key to turning your business problem into a data science based problem
- Improve your domain awareness if you are new to the domain as you spend time understanding the issue
- Helps to approach the problem in a structured way
When should hypothesis generation?
The million dollar question: At what time in the world should you perform hypothesis generation?
- Hypothesis generation should be done before looking at the dataset or data collection.
- You will notice that if you have generated your hypothesis properly, would have included all the variables present in the data set in the generation of its hypothesis.
- It may also have included variables that are not present in the dataset
Case study: generation of hypotheses about “predicting the length of the taxi ride in New York City”
Let's see now the “NEW YORK CITY TAXI TRIP DURATION FORECAST“ statement of the problem and generate some hypotheses that would affect the duration of our taxi trip to understand the generation of hypotheses.
Here is the problem statement:
Predict the duration of a trip so that the company can assign the taxis that are left free for the next trip. This will help reduce customer waiting time and also help gain customer trust..
Let's start!
Hypothesis generation based on various factors
1. Distance-based functions / speed
Let's try to come up with a formula that is related to the duration of the trip and helps us generate several hypotheses for the problem:
TIME = DISTANCE / SPEED
Distance and speed play an important role in predicting trip duration.
We can notice that the duration of the trip is directly proportional to the distance traveled and inversely proportional to the speed of the taxi. With this we can arrive at a hypothesis based on distance and speed.
- Distance: The greater the distance traveled by the taxi, the longer the duration of the trip.
- Inner drop point: Disembarkation points in congested or inland lanes could result in an increase in the duration of the trip.
- Speed: At higher speed, shorter trip duration
2. Car-based functions
Cars are of various types, sizes and brands, and these characteristics of the car could be vital for travel not only for the safety of the passengers but also for the duration of the trip. Let's now generate some hypotheses based on the characteristics of the car.
- Condition of the acr: Well-conditioned cars are unlikely to have breakdown problems and could have a shorter trip duration
- Car size: Small cars (Hatchback) may have a shorter trip duration and larger size cars (XUV) may have a longer travel time depending on the size of the car and congestion in the city.
3. Trip type
Travel types may be different depending on the travel providers: it may be a trip in a remote station, individual or group trips. Now let's define a hypothesis based on the type of trip used.
- Pool car: Bundled trips can increase the duration of the trip, since the car reaches several places before reaching the assigned destination.
4. Functions based on controller details
A driver is an important person when it comes to travel time. Various factors about the driver can help to understand the reason behind the duration of the trip and here are some hypotheses about it.
- Driver's age: Older drivers could be more careful and contribute to a longer journey time.
- Gender: Female drivers are likely to drive slowly and this could contribute to a longer journey time.
- Driver experience: Drivers with less driving experience can lead to a longer trip duration.
- Medical condition: Drivers with a medical condition can contribute to a longer trip duration.
5. Passenger data
Passengers can influence the duration of the trip knowingly or unknowingly. As usual, we come across passengers asking drivers to speed up as they are getting late and there could be other hypothesis factors that we can consider.
- Age of passengers: Older people as passengers can contribute to a longer trip duration, as drivers tend to slow down on trips involving older people.
- Medical conditions or pregnancy: Passengers with medical conditions contribute to a longer trip duration.
- Emergency: Passengers with an emergency could help reduce travel time.
- Passenger counting: Higher passenger numbers lead to shorter trips due to seat congestion
6. Date and time functions
The day and time of the week are important, as New York is a busy city and could be very congested during office hours or on weekdays. Let's now generate some hypotheses about the characteristics based on date and time.
Pick up day:
- Weekends could contribute to more trips in remote stations and could have a longer trip duration.
- Weekdays tend to have a longer travel time due to high traffic.
- If the pick-up day is a holiday, the duration of the trip may be shorter.
- If the pick-up day falls on a holiday week, travel time could be shorter due to less traffic.
Weather:
- Early morning trips have a shorter trip duration due to less traffic
- Night trips have a longer trip duration due to rush hours
7. Road-based features
The roads are of different types and the condition of the road or the obstructions in the road are factors that cannot be ignored. Let's form some hypotheses based on these factors.
- State of the road: The duration of the trip is longer if the condition of the road is bad
- Road type: Trips on concrete roads tend to have a shorter trip duration.
- Strike on the road: Strikes carried out on the roads in the direction of travel increase the duration of the journey
8. Weather-based features
The weather can change at any time and could possibly affect the trip if the weather worsens. Therefore, this is an important feature to consider in our hypothesis.
- Weather at the start of the trip: The rainy weather condition contributes to a longer trip duration
Final notes
- After writing our hypothesis and looking at the data set You will notice that you would have covered hypothesis writing about most of the features present in the dataset. There could also be the possibility that you have to work with fewer functions and the functions that you have hypothesized about are not being captured. / currently stored by the company and not available.
- Always go ahead and capture data from external sources if you think the data is relevant to your prediction. Not .: Get weather information
- It is also important to note that, since hypothesis generation is an estimated assumption, the hypothesis generated could turn out to be true or false once exploratory data analyzes and hypothesis tests are performed on the data.
I hope you have been able to get something of value out of this post. If there is something I missed or something inaccurate or if you have any comments, let me know in the comments. I would be greatly appreciated.
