Interstate 94 (or I-94) is a serious interstate freeway within the US that connects Minneapolis and St Paul, Minnesota. As a consequence of its strategic significance, the I-94 experiences important site visitors volumes, usually influenced by elements comparable to time of day, climate situations, holidays, and so forth. Based on a report cited by WILX 10 (a tv station primarily based in Michigan, USA), it is usually the deadliest freeway in Michigan when it comes to folks dying from automobile crashes. These elements alone spotlight the essential necessity for each policymakers and commuters alike to know the elements that drive site visitors quantity alongside this freeway, in addition to what might be executed to beat them.
On this article, we might be utilizing the Jupyter surroundings to discover the Interstate Site visitors Dataset (US) from Kaggle, which comprises hourly knowledge on the site visitors quantity for westbound I-94. The information was collected by the Minnesota Division of Transportation (MnDOT) from 2012 to 2018 at a station roughly halfway between the 2 cities. This text is break up into two sections: Exploration, the place we might be doing exploratory knowledge evaluation and answering key questions concerning the dataset, and Prediction, the place might be making use of Machine Studying (ML) fashions for regression evaluation (prediction) on site visitors quantity primarily based on the attributes within the dataset.
Knowledge Preprocessing
Earlier than we proceed with our exploratory knowledge evaluation (EDA), we should do a bit of information cleansing. Duplicate rows should be recognized and eliminated, the date-time attribute should be appropriately formatted, and lacking values should be dealt with appropriately.
df = pd.read_csv("Metro_Interstate_Traffic_Volume.csv")# Formatting date_time
df['date_time'] = pd.to_datetime(df['date_time'], format='%d-%m-%Y %H:%M')
df.sort_values('date_time', inplace=True)
# Take away duplicate rows
df = df.drop_duplicates()
# Deal with lacking values
df['holiday'] = df['holiday'].fillna('No Vacation')
df['temp'] = df['temp'].fillna(df['temp'].imply())
df['rain_1h'] = df['rain_1h'].fillna(0)
df['snow_1h'] = df['snow_1h'].fillna(0)
df['clouds_all'] = df['clouds_all'].fillna(df['clouds_all'].median())
df['weather_main'] = df['weather_main'].fillna(df['weather_main'].mode()[0])
df['weather_description'] = df['weather_description'].fillna(df['weather_description'].mode()[0])
Subsequent, we’ll extract ‘yr’, ‘month’, ‘day’, and ‘hour’ from the date_time attribute to be parsed as separate columns. This can give us extra granularity when analyzing variables in relation to time.
Query 1: What does the distribution of rain and snow appear to be?
The very first thing we wish to examine is the distribution of rain and snow all through the 7-year interval. In different phrases, how a lot rain and snow (in mm/hr) has been falling alongside the I-94 freeway from 2012 to 2018? We will reply this by plotting a line chart of the entire rain over time by month, and observing the form of the graph for potential outliers.
# Substitute 'rain' with 'snow' to plot an analogous chart for snow# Create a brand new column for the year-month
df['year_month'] = df['year'].astype(str) + '-' + df['month'].astype(str).str.zfill(2)
# Mixture rain by month
monthly_rain = df.groupby('year_month')['rain_1h'].sum().reset_index()
plt.determine(figsize=(12, 6))
plt.plot(monthly_rain['year_month'], monthly_rain['rain_1h'], marker='o', linestyle='-')
plt.xlabel('Yr-Month')
plt.ylabel('Whole Rain (mm)')
plt.title('Whole Rain Over Time (Month-to-month)')
# solely label months the place complete rain exceeds 8000 mm
xticks_labels = monthly_rain['year_month'].the place(monthly_rain['rain_1h'] > 8000, '')
plt.xticks(ticks=vary(len(monthly_rain)), labels=xticks_labels, rotation=0)
plt.tight_layout()
plt.present()
Primarily based on the rain plot above, complete rain per thirty days hardly ever exceeded 1000mm by way of the entire 7-year interval, save for one peculiar July in 2016 the place the entire rain was near 10000mm, which could point out lengthy intervals of thunderstorms and heavy rain.
Much like the rain distribution, the distribution of snow remained comparatively stagnant all through the entire interval (near 0). The one exceptions are throughout December 2015 and January 2016, the place the quantity of snow was near 8mm within the first month and 3mm within the second month, which can point out lengthy intervals of heavy snow.
Query 2 : What are the completely different classes of climate?
Subsequent, we’ll discover the various kinds of climate situations out there within the dataset. The weather_main attribute comprises the various kinds of climate situations (Clouds, Rain, Snow and many others.) of every occasion (recorded each hour), whereas the weather_description attribute describes every climate situation in additional element (overcast clouds, mild rain, mild snow and many others.) The sunburst plot under presents an in depth breakdown of every climate situation :
import pandas as pd
import plotly.graph_objs as go
import plotly.offline as pyo# Group by 'weather_main' and distinctive 'weather_description' for every class
weather_categories = df.groupby('weather_main')['weather_description'].distinctive()
labels = []
mother and father = []
for class, subcategories in weather_categories.gadgets():
labels.append(class)
mother and father.append("")
for subcat in subcategories:
labels.append(subcat)
mother and father.append(class)
hint = go.Sunburst(
labels=labels,
mother and father=mother and father,
branchvalues='complete'
)
structure = go.Structure(
margin=dict(t=0, l=0, r=0, b=0)
)
fig = go.Determine(hint, structure)
pyo.iplot(fig)
It’s clear from the chart that Thunderstorm has probably the most variety of subcategories (9), adopted by Rain and Snow (7 and 9 respectively), together with the opposite climate situations. Squall, Smoke, Mist, Haze, and Fog have just one subcategory every.
Query 3 : What number of completely different holidays are there?
Subsequent, we will discover all of the completely different holidays. Since days that are holidays are far much less frequent than common days, together with the “None” class within the plot (i.e common day) will considerably skew the info, so our breakdown will embody all the vacations excluding the common days :
# Take away 'None' values
df = df[df['holiday'] != 'None']counts = df['holiday'].value_counts().reset_index()
counts.columns = ['holiday', 'count']
# Generate an inventory of colours (one coloration for every vacation)
colours = plt.cm.get_cmap('tab20', len(counts))
plt.determine(figsize=(12, 8))
plt.barh(counts['holiday'], counts['count'], coloration=colours(vary(len(counts))))
plt.xlabel('Rely')
plt.ylabel('Vacation')
plt.title('Breakdown of Holidays (Excluding "None")')
plt.gca().invert_yaxis() # Invert y-axis to have the very best depend on the prime
for bar, coloration in zip(bars, colours(vary(len(counts)))):
bar.set_color(coloration)
plt.present()
A complete of 11 distinctive holidays had been current within the dataset. Probably the most frequent vacation by depend is Labor Day (7 cases), adopted by 6 and 5 counts from the opposite attributes within the bar chart above.
Query 4: What’s the distribution of site visitors quantity by hour?
Subsequent, we wish to know at which hours of the day (over a 24 hour interval) is site visitors quantity the very best and lowest. This may be executed by plotting the distributions of site visitors quantity per thirty days utilizing boxplots.
plt.determine(figsize=(12, 6))
bp = plt.boxplot([df[df['hour'] == h]['traffic_volume'] for h in vary(24)], showfliers=False, patch_artist=True)for field in bp['boxes']:
field.set_facecolor('orange')
for median in bp['medians']:
median.set_color('black')
plt.xlabel('Hour of the Day')
plt.ylabel('Site visitors Quantity')
plt.title('Site visitors Quantity Distribution by Hour of the Day')
plt.xticks(vary(1, 25), labels=vary(0, 24))
plt.grid(True)
plt.present()
As anticipated, we are able to observe that site visitors quantity peaks throughout the early morning hours when folks drive to work (0600 to 0700hrs), and throughout the late afternoon when folks return from work (1600 to 1700hrs). The least site visitors quantity is throughout the late hours from 0200 to 0300hrs.
Query 5: What’s the common site visitors quantity per thirty days?
Subsequent, we wish to discover out the typical site visitors quantity of every month from 2012 to 2018. We will do that by plotting a line chart of Common Site visitors Quantity towards Yr-Month to assist us visualize which months and years include the very best and lowest common site visitors volumes.
# Mixture site visitors quantity by month
df['year_month'] = df['date_time'].dt.to_period('M') # Create a brand new column for year-month
monthly_traffic = df.groupby('year_month')['traffic_volume'].imply().reset_index()plt.determine(figsize=(12, 6))
plt.plot(monthly_traffic['year_month'].astype(str), monthly_traffic['traffic_volume'], marker='o', linestyle='-')
plt.xlabel('Yr-Month')
plt.ylabel('Common Site visitors Quantity')
plt.title('Common Site visitors Quantity Over Time (Month-to-month)')
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.present()
A number of peaks and troughs might be noticed within the line graph above. The very best recorded peak was throughout the month of Labor Day in Could 2017, adopted by New Yr month in January 2016 and New Yr month in January 2013. That is anticipated, since nationwide holidays like Labor Day and New Yr’s usually invite greater site visitors ranges than ordinary.
Query 6: Which elements contribute probably the most to site visitors quantity?
Lastly, a very powerful query is to find out which of the attributes we now have mentioned contribute probably the most to the spike in site visitors quantity. Because the time collection knowledge we now have explored is non-linear, a easy correlation evaluation can be inadequate. We should make use of extra complicated fashions comparable to Random Forest and Gini Index to find out which attributes are a very powerful in predicting the goal label, which on this case is site visitors quantity.
mannequin = RandomForestRegressor(random_state=42)
mannequin.match(x_train, y_train)feature_importances = mannequin.feature_importances_
feature_names = x_train.columns
importance_dict = dict(zip(feature_names, feature_importances))
sorted_importances = sorted(importance_dict.gadgets(), key=lambda x: x[1], reverse=True)
print("Function Importances:")
for function, significance in sorted_importances:
print(f"{function}: {significance:.4f}")
plt.determine(figsize=(10, 6))
plt.bar([x[0] for x in sorted_importances], [x[1] for x in sorted_importances])
plt.xlabel('Options')
plt.ylabel('Significance')
plt.title('Function Importances')
plt.xticks(rotation=45)
plt.tight_layout()
plt.grid
plt.present()
Primarily based on the Gini Significance Index obtained by the Random Forest Regressor, it may be noticed that the ‘hour’ attribute contributes probably the most in predicting site visitors quantity (Gini Index: 0.8255). That is anticipated since we had beforehand noticed a transparent enhance and reduce in site visitors quantity all through numerous hours of the day. Different attributes comparable to climate, rain, snow, and vacation contribute little or no to site visitors quantity, since these attributes stay comparatively stagnant all through a lot of the period, save for the occasional outliers.
Within the second a part of the article, we might be performing regression evaluation to foretell site visitors quantity primarily based on the opposite attributes within the dataset. The regression mannequin we might be utilizing is Random Forest. Recall that this what our cleaned dataset appears like :
Earlier than feeding the info into the Random Forest, we should first put together it accordingly. First, we encode the explicit attributes into numerical attributes utilizing the LabelEncoder operate:
catcol = ['holiday','weather_main','weather_description','day']
encoder = LabelEncoder()
for col in catcol:
df[col] = encoder.fit_transform(df[col])
Subsequent, we should normalize the traffic_volume and temp attributes into utilizing the MinMaxScaler methodology. That is to make sure the ranges of those attributes fall inside a predetermined vary (min and max).
st = MinMaxScaler()
df['traffic_volume'] = st.fit_transform(df[['traffic_volume']])
df['temp'] = st.fit_transform(df[['temp']])
Then we should break up the info into x and y labels, which can then be break up into coaching and take a look at units (15% take a look at set for our case).
x = df.drop('traffic_volume',axis = 1)
y = df['traffic_volume']
x_train,x_test,y_train,y_test = train_test_split(x,y,train_size = 0.85,shuffle = True,random_state=42)
Lastly, we are able to now prepare and take a look at the Random Forest mannequin and consider its efficiency primarily based on three key metrics : Imply Absolute Sq. Error, Imply Sq. Error, and R-squared.
mannequin = RandomForestRegressor(n_estimators=100, random_state=42)
mannequin.match(x_train, y_train)
y_pred = mannequin.predict(x_test)mse = mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test,y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Imply Squared Error: {mse:.8f}')
print(f'Imply Absolute Error: {mae:8f}')
print(f'R-squared: {r2:.8f}')
The outcomes obtained by the Random Forest mannequin are as follows :
Imply Squared Error: 0.00302850
Imply Absolute Error: 0.030635
R-squared: 0.95976209
The MSE rating achieved could be very low, indicating that the mannequin’s predictions are very near the precise values. Much like the MSE, the MAE rating can be fairly low, indicating that the mannequin predicts very carefully to the precise values. Lastly, the R-squared worth could be very excessive, indicating that the mannequin has sturdy explanatory energy and matches the info very properly.
