Introduction
Data scientists are a lazy breed of animals!! We detest the practice of doing any repeatable work manually. We cower in fear at the mere thought of doing tedious manual tasks and when we come across a, we try to automate it so that the world becomes a better place.
We have been hosting some meetings in India for the last few months and we wanted to see what some of the best meetings around the world were doing. For a normal human being, this would mean browsing the meeting pages and finding this information manually.
Not for a data scientist!
What are meetings?
Meetup can be better understood as a self-organized gathering of people to achieve a predefined goal. Meetup.com is the world's largest local group network. Meetup's mission is “revitalize the local community and help people around the world organize themselves”.
The meeting search process can take a long time (I prefer to say it). There are multiple attached limitations (which I have explained in the next section). But, How would a data scientist perform this task to save time? Of course, would endeavor to automate this process!
In this article, I will introduce you to a data scientist's approach to locating meeting groups using Python. Taking this as a reference, you can find groups located in any corner of the earth. You can also add your own analysis layer to discover some cool ideas.
Sign up for Data Hackathon 3.X: win an Amazon coupon worth 10.000 rupees (~ 200 Dollars)
The challenge with manual focus
Let's say you want to find out and join some of the best meetings in your area.. Obviously, you can do this task manually, but there are some challenges you face:
- There could be several groups with similar names and purposes. It becomes difficult to find the correct ones just by reading the names.
- Let's say you're looking for meetings in data science, you will need to manually navigate through each of the groups, to see several parametersThe "parameters" are variables or criteria that are used to define, measure or evaluate a phenomenon or system. In various fields such as statistics, Computer Science and Scientific Research, Parameters are critical to establishing norms and standards that guide data analysis and interpretation. Their proper selection and handling are crucial to obtain accurate and relevant results in any study or project.... to judge their quality (for instance, meeting frequency, membership, average review, etc.) and then make the decision to join the group or not, It seems like a lot of work to me!
- What's more, if you have any specific requirements, as if you want to see groups present in several cities, you will end up browsing the groups of each city manually; I already shrink at the thought.
Suppose you are in a locality with more than 200 groups in your area of interest. How would you find the best?
The data scientist's solution
In this article, I have identified several Python Meetups of cities in India, EE. UU., United Kingdom, HK, TW y Australia. The following are the steps that I will perform:
- Get information from meetup.com using the API they have provided.
- Move the data to a DataFrame and
- Analyze it and join the right groups
These steps are quite easy to perform. Then, I list the steps to perform them. As mentioned earlier, this is just the beginning of the possibilities that open up. You can use this information to gain a wealth of knowledge about various communities around the world..
Paso 0: import libraries
Below is the list of libraries I have used to code this project.
import urllib import json import pandas as pd import matplotlib.pyplot as plt from geopy.geocoders import Nominatim
Here is a quick overview of these libraries:
- urllib: This module provides a high-level interface to obtain data from the World Wide Web.
- json (Java script object notation): the library jsonJSON, o JavaScript Object Notation, It is a lightweight data exchange format that is easy for humans to read and write, and easy for machines to analyze and generate. It is commonly used in web applications to send and receive information between a server and a client. Its structure is based on key-value pairs, making it versatile and widely adopted in software development.. can analyze JSON from strings or files. The library parses JSON into a Python dictionary or list.
- pandas: Used for structured data manipulations and operations. Used a lot for data preparation and processing.
- matplotlib: Used to plot a wide variety of graphics, from histogramasHistograms are graphical representations that show the distribution of a dataset. They are constructed by dividing the range of values into intervals, O "Bins", and counting how much data falls in each interval. This visualization allows you to identify patterns, trends and variability of data effectively, facilitating statistical analysis and informed decision-making in various disciplines.... up to line charts and heatmaps.
- geocoders: Simple and consistent geocoding library written in Python.
Paso 1: use API to read data in JSON format
You can get data from any website in various ways:
- Track web pages using a combination of libraries like BeautifulSoup and Scrapy. Find underlying trends in html using regular expressions to extract the required data.
- If the website provides an API (application programming interface), use it to get the data. You can understand this as an intermediary between a programmer and an application. This broker accepts requests and, if that request is allowed, returns the data.
- Tools like import.io can also help you do this.
For websites that provide an API, is usually the best way to get the information. The first method mentioned above is susceptible to layout changes on a page and, sometimes, it can be very complicated. Fortunately, Meetup.com offers several API to access the required data. Using this API, we can access information about various groups.
To access the automated API-based solution, we would need courage to sig_id Y sig (different for different users). Follow the steps below to access these.
Paso 2: Generate a list of signed urls for all given cities
Now, we should request a signed url for each search (in our case, town + theme) and the output of these signed urls will provide the detailed information about the matching groups:
- Create a list of all cities
- Create an object to access the longitude and latitude of the city.
- Access the city from the given list and generate the latitude and longitude using “geolocator “ object
- Generate URL string with required attributes as data format (json), radio (number of miles from the city center, 50), theme (Python), latitude and longitude
- Repeat this step for each city and add all the URLs in a list
places = [ "san francisco", "california", "boston ", "new york" , "pennsylvania", "colorado", "seattle", "washington","the Angels", "San Diego", "houston", "austin", "kansas", "delhi", "chennai", "bangalore", "mumbai" , "Sydney","Melbourne", "Perth", "Adelaide", "Brisbane", "Launceston", "Newcastle" , "beijing", "shanghai", "Suzhou", "Shenzhen","Guangzhou","Dongguan", "Taipei", "Chengdu", "Hong Kong"] urls = [] #url lists radius = 50.0 #add the radius in miles data_format = "json" topic = "Python" #add your choice of topic here sig_id = "########" # initialize with your sign id, check sample signed key sig = "##############" # initialize with your sign, check sample signed key
for place in places:
location = geolocator.geocode(place)
urls.append("https://api.meetup.com/2/groups?offset=0&format=" + data_format + "&lon=" + str(location.longitude) + "&topic=" + topic + "&photo-host=public&page=500&radius=" + str(radius)+"&fields=&years =" + str(location.latitude) + "&order=id&desc = false&sig_id=" +sig_id + "&sig =" + sig)
Paso 3: read data from url and access relevant functions in a DataFrame
Now, we have a list of urls for all cities. Then, we will use the urllib library to read data in JSON format. Later, we will read the data in a list before converting it to a DataFrame.
city,country,rating,name,members = [],[],[],[],[] for url in urls: response = urllib.urlopen(url) data = json.loads(response.read()) data = data["results"] #accessed data of results key only for i in data : city.append(i['city']) country.append(i['country']) rating.append(i['rating']) name.append(i['name']) members.append(i['members']) df = pd.DataFrame([city,country,rating,name,members]).T df.columns=['city','country','rating','name','members']
Paso 4: compare Meetup groups in various cities
It's time to analyze the data now and find the right groups based on various metrics, as the number of members, The qualifications, the city and others. Here are some basic findings, that I have generated for groups of pythons in different cities of India, EE. UU., United Kingdom, HK, TW y Australia.
To know more about these Python codes, You can read articles on exploring and visualizing data using Python
Number of Python groups in six countries
freq = df.groupby('country').city.count()
fig = plt.figure(figsize=(8,4))
ax1 = fig.add_subplot(121)
ax1.set_xlabel('Country')
ax1.set_ylabel('Count of Groups')
ax1.set_title("Number of Python Meetup Groups")
freq.plot(kind='bar') 
Above you can notice that US is the leader in python meetup groups. This stats can also help us to estimate the penetration of python in US data science industry compare to others.
Average size of groups in all countries
freq = df.groupby('country').members.sum()/df.groupby('country').members.count()
fig = plt.figure(figsize=(8,4))
ax1 = fig.add_subplot(121)
ax1.set_xlabel('Country')
ax1.set_ylabel('Average Members in each group')
ax1.set_title("Python Meetup Groups")
freq.plot(kind='bar')
One more time, EE. UU. Emerges as the leader in average number of members in each group, while CN has the lowest average.
Average rating of groups in all countries
freq = df.groupby('country').rating.sum()/df.groupby('country').rating.count()
fig = plt.figure(figsize=(8,4))
ax1 = fig.add_subplot(121)
ax1.set_xlabel('Country')
ax1.set_ylabel('Average rating')
ax1.set_title("Python Meetup Groups")
freq.plot(kind='bar')
AU and EE. UU. They have a similar average rating (~ 4) in all groups.
The 2 best groups from each country
df=df.sort(['country','members'], ascending=[False,False])
df.groupby('country').head(2)
It is time to identify the two main groups in each country based on the number of members. You can also identify groups based on rating. Here I have done a basic analysis to illustrate this approach. You can access other APIs as well to find information like upcoming events, number of events, duration of events and others and then merge all relevant information based on group_id (or key value).
Final code
Below is the final code for this exercise, you can play around with it by putting your sig_id and sig key and search multiple results from different topics in different cities. I have also uploaded it in GitHub.
import urllib import json import pandas as pd import matplotlib.pyplot as plt from geopy.geocoders import Nominatim
geolocator = By name() #create object
places = [ "san francisco", "california", "boston ", "new york" , "pennsylvania", "colorado", "seattle", "washington","the Angels", "San Diego", "houston", "austin", "kansas", "delhi", "chennai", "bangalore", "mumbai" , "Sydney","Melbourne", "Perth", "Adelaide", "Brisbane", "Launceston", "Newcastle" , "beijing", "shanghai", "Suzhou", "Shenzhen","Guangzhou","Dongguan", "Taipei", "Chengdu", "Hong Kong"]
# login on meetup.com. if you dont have an account, then please signup # Go to https://secure.meetup.com/meetup_api/console/?path=/2/groups # In the topics like "Python", enter topic of your choice. and click on show response # copy the signed key. in the singed key, copy the sig_id and sig and initialise variables sig_id and sig # sample signed key : "https://api.meetup.com/2/groups?offset=0&format=json&topic=python&photo-host=public&page=20&radius=25.0&fields=&order=id&desc = false&sig_id=******&sig = *****************"
urls = [] #url lists radius = 50.0 #add the radius in miles data_format = "json" #you can add another format like XML topic = "Python" #add your choice of topic here
sig_id = "186640998" # initialise with your sign id, check sample signed key sig = "6dba1b76011927d40a45fcbd5147b3363ff2af92" # initialise with your sign, check sample signed key
for place in places:
location = geolocator.geocode(place)
urls.append("https://api.meetup.com/2/groups?offset=0&format=" + data_format + "&lon=" + str(location.longitude) + "&topic=" + topic + "&photo-host=public&page=500&radius=" + str(radius)+"&fields=&years =" + str(location.latitude) + "&order=id&desc = false&sig_id=" +sig_id + "&sig =" + sig)
city,country,rating,name,members = [],[],[],[],[] for url in urls: response = urllib.urlopen(url) data = json.loads(response.read()) data = data["results"] for i in data : city.append(i['city']) country.append(i['country']) rating.append(i['rating']) name.append(i['name']) members.append(i['members']) df = pd.DataFrame([city,country,rating,name,members]).T df.columns=['city','country','rating','name','members'] df.sort(['members','rating'], ascending=[False, False])
freq = df.groupby('country').city.count()
fig = plt.figure(figsize=(8,4))
ax1 = fig.add_subplot(121)
ax1.set_xlabel('Country')
ax1.set_ylabel('Count of Groups')
ax1.set_title("Number of Python Meetup Groups")
freq.plot(kind='bar')
freq = df.groupby('country').members.sum()/df.groupby('country').members.count()
ax1.set_xlabel('Country')
ax1.set_ylabel('Average Members in each group')
ax1.set_title("Python Meetup Groups")
freq.plot(kind='bar')
freq = df.groupby('country').rating.sum()/df.groupby('country').rating.count()
ax1.set_xlabel('Country')
ax1.set_ylabel('Average rating')
ax1.set_title("Python Meetup Groups")
freq.plot(kind='bar')
df=df.sort(['country','members'], ascending=[False,False])
df.groupby('country').head(2)
Final notes
In this article, We analyze the Python application to automate a manual process and the level of precision to find the right Meetup groups. We use API to access information from the web and transfer it to a DataFrame. Subsequently, we analyze this information to generate practical insights.
We can make this app smarter by adding additional information like upcoming events, number of events, RSVP and various other metrics. You can also use this data to get interesting information about the community and people. For instance, Does the RSVP to the rate of attendance to review the rate funnel differ from country to country? Which countries plan their meetings in advance?
Give it a try at the end and share your knowledge in the comment section below.
