Waze User Churn Prediction: Random Forest vs. XGBoost | by Lekha Topil | May, 2024

Welcome to the last word a part of the Waze Particular person Churn Prediction Endeavor!

On this concluding a part of our enterprise, we’re shifting previous the logistic regression model to find further superior strategies using Random Forest and XGBoost. Our goal is to bolster the predictive effectivity and effectiveness of our churn prediction model. Beforehand, we completed key phases along with Exploratory Data Analysis (EDA), Hypothesis Testing, and the occasion of a Logistic Regression Model.

Waze, a free navigation app owned by Google, makes it easier for drivers all around the world to achieve their places. Waze’s neighborhood of map editors, beta testers, translators, companions, and clients helps make each drive larger and safer.

Develop a machine finding out model to predict client churn. An appropriate model will help cease churn, improve client retention, and contribute to the growth of Waze’s enterprise.

Sooner than persevering with, let’s cope with some obligatory questions:

1. What are you being requested to do?

• Predict whether or not or not a purchaser will churn or keep retained on the Waze app.

2. What are the ethical implications of the model? What are the implications of your model making errors?

1. What is the likely impression of the model when it predicts a false unfavorable (i.e. when the model says a Waze client acquired’t churn, nevertheless they actually will (Type II Error))?

• Waze may miss the prospect to take proactive steps to retain clients who’re vulnerable to stop using the app.
• By exactly determining at-risk clients, Waze can implement centered strategies resembling sending personalized emails showcasing app choices, providing concepts for seamless navigation, or conducting surveys to know client ache components and causes for potential churn.

2. What is the likely impression of the model when it predicts a false constructive (i.e. when the model says a Waze client will churn, nevertheless they actually acquired’t (Type I Error))?

• Waze may take pointless proactive steps to retain clients who had been not at-risk of churning, doubtlessly leading to actions which may annoy or irritate loyal clients.
• This could result in elevated client dissatisfaction ensuing from frequent and pointless emails or notifications, affecting client experience.

It’s important to strike a stability in model accuracy to scale back every false positives and false negatives. This ensures that proactive retention efforts are centered notably in direction of clients liable to churning, whereas avoiding any inconvenience or annoyance to totally different clients who often will not be vulnerable to churn.

3. Do some great benefits of such a model outweigh the potential points?

• If the model performs successfully, it could help Waze decide clients who’re liable to churning, enabling the company to implement proactive retention efforts and in the long run enhance retention prices.
• Nonetheless, there are ethical points related to false predictions (false positives and false negatives). False positives may set off pointless interventions with loyal clients, whereas false negatives might suggest missed alternate options to retain at-risk clients.

For full code implementation, you might go to my Kaggle Notebook

# Import regular operational packages
import pandas as pd
import numpy as np
# Packages for visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Import packages for info modeling
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import roc_auc_score, roc_curve, auc
from sklearn.metrics import accuracy_score, precision_score, recall_score, 
f1_score, confusion_matrix, ConfusionMatrixDisplay, RocCurveDisplay, PrecisionRecallDisplay
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
# function that helps plot operate significance
from xgboost import plot_importance

# Study Information
df = pd.read_csv('waze_dataset.csv')
# Present
df.head()

# Chceck info
df.info()

1. km_per_driving_day

• We’ll create a model new operate known as km_per_driving_day, which calculates the suggest distance pushed per driving day for each client. This can most likely be achieved by dividing the general kilometers pushed (driven_km_drives) by the number of driving days (driving_days).

There are some clients with zero driving_days, inflicting Pandas to assign infinity (inf) values to the corresponding rows inside the new column ensuing from undefined division by zero. Subsequently, it is important to transform these infinite values to zero.

# Create `km_per_driving_day` operate
df['km_per_driving_day'] = df['driven_km_drives'] / df['driving_days']
# Convert infinite values to zero
df.loc[df['km_per_driving_day'] == np.inf, 'km_per_driving_day'] = 0
# Descriptive Statistics
df['km_per_driving_day'].describe()

2. percent_sessions_in_last_month

• Subsequent, we’ll create the percent_sessions_in_last_month operate by dividing the number of month-to-month intervals (intervals) by the estimated full intervals (total_sessions) as a result of the patron onboarded.
• This new variable will help us understand the proportion of full intervals that occurred inside the ultimate month as a result of the patron was onboarded.

# Create `percent_sessions_in_last_month` operate
df['percent_sessions_in_last_month'] = df['sessions'] / df['total_sessions']
# Descriptive Statistics
df['percent_sessions_in_last_month'].describe()

Based mostly totally on our EDA, we discovered that 50% of shoppers have 40% or further of their intervals occurring inside the ultimate month. This discovering implies that the percent_sessions_in_last_month operate could be a useful predictor for our model.

3. professional_driver

The following operate we’ll create is professional_driver, which identifies clients based mostly totally on specific thresholds for the number of drives (drives) and driving days (driving_days) inside the ultimate month. To be labeled as a professional_driver, a client ought to meet every of the following circumstances:

1. The buyer must have completed 60 or further drives (drives) inside the ultimate month.
2. The buyer must have pushed for 15 or further days (driving_days) inside the ultimate month.

# Create `professional_driver` operate
df['professional_driver'] = np.the place((df['drives']>=60) & (df['driving_days'] >=15), 1, 0)
# Present
df[['drives', 'driving_days', 'professional_driver']].head()

In our earlier Logistic Regression Model, this operate ranked as a result of the third most important predictor based mostly totally on its coefficient inside the operate significance.

4. total_sessions_per_day

The following operate will most likely be total_sessions_per_day, representing the suggest number of intervals per day since onboarding.

• intervals: The number of prevalence of a client opening the app in the middle of the month.
• total_sessions: A model estimate of the general number of intervals since a client has onboarded.

# Create `total_sessions_per_day` operate
df['total_sessions_per_day'] = df['total_sessions'] / df['n_days_after_onboarding']
# Descriptive Statistic
df['total_sessions_per_day'].describe() 

5. km_per_hour

km_per_hour represents the suggest kilometers per hour pushed inside the ultimate month. It is calculated by dividing driven_km_drives by duration_minutes_drives and altering the tip end result into hours by extra dividing by 60.

• driven_km_drives: Complete kilometers pushed in the middle of the month.
• duration_minutes_drives: Complete size pushed in minutes in the middle of the month.

# Create `km_per_hour` operate
df['km_per_hour'] = df['driven_km_drives'] / df['duration_minutes_drives'] / 60
# Descriptive Statistics
df['km_per_hour'].describe() 

6. km_per_drive

km_per_drive represents the suggest number of kilometers pushed per drive made by each client inside the ultimate month.

• driven_km_drives: Complete kilometers pushed in the middle of the month.
• drives: Complete size pushed in minutes in the middle of the month.

# Create `km_per_drive` operate
df['km_per_drive'] = df['driven_km_drives'] / df['drives']
# Convert infinite values to zero
df.loc[df['km_per_drive']==np.inf, 'km_per_drive'] = 0
# Descriptive Statistic
df['km_per_drive'].describe()

7. percent_of_sessions_to_favorite

• Lastly, we’ll create percent_of_sessions_to_favorite, which represents the proportion of full intervals directed in route of a client’s favorite places.
• We lack info on the full drives since clients had been onboarded, we’ll use total_sessions as a cheap estimate or proxy for the general number of drives.

This methodology permits us to attain insights into client habits and preferences based mostly totally on the on the market info.

# Create `percent_of_sessions_to_favorite` operate
df['percent_of_sessions_to_favorite'] = (df['total_navigations_fav1'] +
df['total_navigations_fav2']) / df['total_sessions']
df['percent_of_sessions_to_favorite'].describe()

Prospects whose drives to non-favorite places characterize the following share of their full drives could also be less vulnerable to churn, as they’re exploring new locations. Conversely, clients who allocate the following share of their drives to favorite places could also be further vulnerable to churn, most likely ensuing from their familiarity with the routes.

All through our earlier Exploratory Data Analysis (EDA), we acknowledged 700 missing values inside the dataset. These missing values are believed to be missing at random (MAR), as there is no proof of a non-random set off. Supplied that the missing values characterize decrease than 5% of the general dataset, eradicating them is unlikely to have an enormous affect on the dataset’s integrity.

# Drop rows with missing values
df = df.dropna(subset=['label'])
# Dimension after removig missing info
print('Dimension after eradicating missing info:', df.kind)
# Output:
Dimension after eradicating missing info: (14299, 20)

Based mostly totally on our earlier Exploratory Data Analysis (EDA), we acknowledged outliers and extreme values in plenty of columns. Nonetheless, tree-based fashions are sturdy to outliers, eliminating the need for imputation or outlier eradicating.

Our Exploratory Data Analysis (EDA) revealed strong correlations amongst certain variables, indicating multicollinearity. Supplied that tree-based fashions can efficiently deal with collinearities between neutral variables, there is no should take away any variables in addition to the ID column, which holds no relevance to churn prediction.

# Drop `ID` column
df = df.drop(['ID'], axis=1)

Dummying Choices

We now have one categorical variable, gadget, amongst our neutral variables, which consists of two groups: iPhone and Android. Whereas there are a selection of methods on the market, resembling pd.get_dummies() or OneHotEncoder(), for encoding categorical variables, we’ll use a straightforward methodology with np.the place():

• Assign 1 for iPhone
• Assign 0 for Android

# Create new `device2` variable
df['device2'] = np.the place(df['device']=='Android', 0, 1)
# Check
df[['device', 'device2']].tail()

Lastly, we’ll encode the objective variable label into binary format:

• Assign 0 for all Retained
• Assign 1 for all Churned

# Create binary `label2` column
df['label2'] = np.the place(df['label']=='churned', 1, 0)
# Check
df[['label', 'label2']].tail()

# Get class stability of 'label' col
df['label'].value_counts(normalize=True) * 100

• Our dataset reveals class imbalance, with 17.73% of shoppers churned and 82.26% retained — typical for churn prediction datasets. Although the imbalance is noticeable, it isn’t extreme and could also be addressed with out rebalancing programs. Tree-based fashions like Random Forest and XGBoost excel in coping with such imbalances.
• For our imbalanced dataset, accuracy is an unsuitable evaluation metric. A false constructive, the place the model incorrectly predicts churn for a client who will actually preserve, might end in pointless retention measures, doubtlessly irritating clients. Nonetheless, false positives do not result in direct financial losses or totally different important penalties.
• Alternatively, false negatives, the place the model fails to predict churn for patrons who actually churn, are essential. The company can’t afford false negatives because of they cease proactive measures to retain at-risk clients, doubtlessly rising the churn payment. Subsequently, we’ll prioritize selecting the model based mostly totally on the recall score.

Modeling workflow and model alternative course of

The final word dataset used for modeling consists of 14,299 samples, which is on the lower end of what is often considered ample for conducting a sturdy model alternative course of. Nonetheless, it nonetheless provides an ample basis for our analysis.

Break up the data

• Break up the data into put together/validation/verify models (60/20/20)

We’ll observe these steps:

1. Define X
2. Define y
3. Break up the data into an interim teaching set and a verify set using an 80/20 ratio.
4. Further lower up the interim teaching set proper right into a teaching set and a validation set using a 75/25 ratio, resulting in a remaining ratio of 60/20/20 for teaching/validation/verify models.

# Isolate X variables
X = df.drop(columns=['label', 'label2', 'device'])
# Isolate y variable
y = df['label2']
# Break up into put together and verify models
X_tr, X_test, y_tr, y_test = train_test_split(X, y, stratify=y, test_size=0.2, random_state=42)
# Break up into put together and validate models
X_train, X_val, y_train, y_val = train_test_split(X_tr, y_tr, stratify=y_tr, test_size=0.25, random_state=42)

Now, let’s verify the number of samples inside the partitioned info.

# Check the scale of the data
data_len = [len(x) for x in (X_train, X_val, X_test)]
data_len
# Output:
[8579, 2860, 2860]

# 1. Instantiate the random forest classifier
rf = RandomForestClassifier(random_state=42)
# 2. Create a dictionary of hyperparameters to tune
cv_params  = {'max_depth': [None],
'max_features': [1.0],
'max_samples': [1.0],
'min_samples_leaf': [2],
'min_samples_split':[2],
'n_estimators': [300]}
# 3. Define a dictionary of scoring metrics to grab
scoring = {'accuracy', 'precision', 'recall', 'f1'}
# 4. Instantiate the GridSearchCV object
rf_cv = GridSearchCV(rf, cv_params, scoring=scoring, cv=4, refit='recall')

%%time
rf_cv.match(X_train, y_train)
# Output:
CPU events: client 2min 26s, sys: 115 ms, full: 2min 26s
Wall time: 2min 26s

# Have a look at best score
rf_cv.best_score_
# Ouput
0.12678201409034398

# Have a look at best hyperparameter combo
rf_cv.best_params_

def make_results(model_name: str, model_object, metric: str):
'''
Arguments:
model_name (string): what you want the model to be known as inside the output desk (determine of the model)
model_object: a match GridSearchCV object
metric (string): precision, recall, f1, and accuracy
Returns a pandas df with precision, recall, f1, and accuracy scores
for the model with the easiest suggest 'metric' score all through all validation folds.
'''
# Map the metric determine to the corresponding column in cv_results
metric_dict = {
'precision': 'mean_test_precision',
'recall': 'mean_test_recall',
'f1': 'mean_test_f1',
'accuracy': 'mean_test_accuracy'
}
# Extract cv_results from the GridSearchCV object
cv_results = pd.DataFrame(model_object.cv_results_)
# Decide the row index with the utmost price for the required metric
best_index = cv_results[metric_dict[metric]].idxmax()
# Retrieve the metrics for the best-performing model configuration
metrics = {
'model': model_name,
'precision': cv_results.loc[best_index, 'mean_test_precision'],
'recall': cv_results.loc[best_index, 'mean_test_recall'],
'F1': cv_results.loc[best_index, 'mean_test_f1'],
'accuracy': cv_results.loc[best_index, 'mean_test_accuracy']
}
# Create a DataFrame containing the extracted metrics
result_table = pd.DataFrame(metrics, index=[0])
return result_table

# Get the scores
outcomes = make_results('RF CV', rf_cv, 'recall')
outcomes

• The precision, recall, and F1 scores from the Random Forest model are underneath the required thresholds.
• The recall for the Random Forest model is 0.12, indicating that the model predicts further false negatives. A recall of 0.12 implies that solely 12% of the actual churned cases are appropriately acknowledged by the model. In numerous phrases, the model misses a superb portion of shoppers who actually churned, failing to grab them in its predictions.
• Compared with our earlier Logistic Regression model, the recall has improved from 0.09 to 0.12 inside the Random Forest model, representing a 33% enhance.
• Nonetheless, precision decreased from 0.52 (Logistic Regression model) to 0.458, and the F1-score confirmed a slight enchancment from 0.16 (Logistic Regression model) to 0.199. Lastly, accuracy stays almost the equivalent between the two fashions.

# 1. Instantiate the XGBoost classifier
xgb = XGBClassifier(objective='binary:logistic', random_state=42)
# 2. Create a dictionary of hyperparameters to tune
cv_params = {'max_depth': [6, 12],
'min_child_weight': [3, 5],
'learning_rate': [0.01, 0.1],
'n_estimators': [300]
}
# 3. Define a dictionary of scoring metrics to grab
scoring = {'accuracy', 'precision', 'recall', 'f1'}
# 4. Instantiate the GridSearchCV object
xgb_cv = GridSearchCV(xgb, cv_params, scoring=scoring, cv=4, refit='recall')

%%time
xgb_cv.match(X_train, y_train)
# Output
CPU events: client 2min 59s, sys: 1.3 s, full: 3min 1s
Wall time: 46.3 s

# Have a look at best score
xgb_cv.best_score_
# Ouput
0.1708264263019754

# Have a look at best parameters
xgb_cv.best_params_

# Title 'make_results()' on the GridSearch object
xgb_cv_results = make_results('XGB CV', xgb_cv, 'recall')
outcomes = pd.concat([results, xgb_cv_results], axis=0)
outcomes

• The XGBoost model performs larger as compared with the Random Forest model, notably with its recall score, which is sort of double that of the Logistic Regression model and about 35% elevated than the Random Forest model.
• Nonetheless, no matter these enhancements, the XGBoost model can solely seize spherical 17% of exact churned cases, indicating a notable presence of false negatives.
• Furthermore, whereas precision has barely decreased, the F1-score reveals enchancment over every Logistic Regression model and Random Forest fashions.

Now let’s make predictions on the validation set using every the Random Forest and XGBoost fashions. We’ll take into account their effectivity, and the model that performs larger on the validation set will most likely be chosen as a result of the champion model.

For this operate, we’ll define a function known as get_test_scores() to generate a desk of scores based mostly totally on the predictions made on the validation info.

Random Forest

# Use random forest model to predict on validation info
rf_val_preds = rf_cv.best_estimator_.predict(X_val)

def get_test_scores(model_name:str, preds, y_test_data):
'''
Generate a desk of verify scores.
In:
model_name (string): determine of the model of your different
preds: numpy array of verify predictions
y_test_data: numpy array of y_test info
Out:
desk: a pandas df of precision, recall, f1, and accuracy scores to your model
'''
# Calculate evaluation metrics
metrics = {
'precision': precision_score,
'recall': recall_score,
'F1': f1_score,
'accuracy': accuracy_score
}
scores = {metric: metrics[metric](y_test_data, preds) for metric in metrics}
# Create DataFrame
desk = pd.DataFrame({
'model': [model_name],
**scores
})
return desk

# Get validation scores for RF model
rf_val_scores = get_test_scores('RF Val', rf_val_preds, y_val)
# Append to the outcomes desk
outcomes = pd.concat([results, rf_val_scores], axis=0)
outcomes

The scores for the validation set (RF Val) are barely lower as compared with the teaching set (RF CV). The diploma of variation in scores between the teaching and validation models is inside an appropriate range, suggesting that the model should not be overfitting the teaching info.

XGBoost

# XGBoost model to predict on validation info
xgb_val_preds = xgb_cv.best_estimator_.predict(X_val)
# Get validation scores for XGBoost model
xgb_val_scores = get_test_scores('XGB val', xgb_val_preds, y_val)
# Append to the outcomes desk
outcomes = pd.concat([results, xgb_val_scores], axis=0)
outcomes

• The XGBoost model’s effectivity on the validation set (XGB Val) reveals a slight decrease in scores as compared with the teaching set (XGB CV).
• This minor decline falls inside an appropriate range, suggesting that the XGBoost model did not overfit the teaching info.
• Among the many many evaluated fashions (Random Forest and XGBoost), the XGBoost model emerges as a result of the clear champion based mostly totally on its effectivity on the validation set.

As a result of the XGBoost model carried out larger as compared with the random forest and logistic regression fashions, we’ll proceed by using the XGBoost model to make predictions on the verify dataset. This step will allow us to guage the model’s effectivity on new and unseen info, providing insights into its effectiveness for future predictions.

# XGBoost model to predict on verify info
xgb_test_preds = xgb_cv.best_estimator_.predict(X_test)
# Get verify scores for XGBoost model
xgb_test_scores = get_test_scores('XGB verify', xgb_test_preds, y_test)
# Append to the outcomes desk
outcomes = pd.concat([results, xgb_test_scores], axis=0)
outcomes

• The recall on the verify set (0.181) is elevated than that on the validation set (0.161), indicating a slight enchancment inside the model’s capability to find out true positives (churned clients) in new and unseen info.
• Furthermore, there was a slight enhance in precision, and the F1 score moreover improved.

Basic, the comparability between verify and validation scores for the XGBoost model implies that the model’s effectivity barely improved when evaluated on the verify set.

Now, we’ll create a confusion matrix to visualise the XGBoost model’s predictions on the verify info.

# Generate array of values for confusion matrix
cm = confusion_matrix(y_test, xgb_test_preds, labels=xgb_cv.classes_)
# Plot confusion matrix
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=['retained', 'churned'])
disp.plot();

True Unfavorable (TN): 2228

• These are the conditions appropriately predicted as not churned by the model.
• In our case, the model exactly acknowledged 2228 retained clients.

False Optimistic (FP): 125

• These are conditions the place the model incorrectly predicted as “churned” when the exact finish end result was “retained”.

False Unfavorable (FN): 415

• These conditions characterize cases the place the model predicted “retained”, nevertheless the exact finish end result was “churned”, indicating a wide selection of false negatives.
• The model predicted 3 instances further false negatives (415) than false positives (92).

True Optimistic (TP): 92

• These are conditions the place the model appropriately acknowledged 92 churned clients, which is significantly fewer as compared with the number of false negatives.

The ratio of false negatives to false positives is roughly 4.51 (415/92 ≈ 4.51), indicating that false negatives occur further ceaselessly than false positives inside the model’s predictions.

The plot_importance() function in XGBoost permits us to visualise the mandatory choices of our educated model. This visualization signifies which choices are most influential in making predictions, providing insights into the essential factor parts driving the model’s choices.

# Plot operate significance for XGBoost model
plot_importance(xgb_cv.best_estimator_);

• In step with the operate significance of the XGBoost model, km_per_hour significantly influences predictions. In distinction to logistic regression, which carefully weighted a single operate (activity_days), the XGBoost model makes use of many choices.
• In our logistic regression model, professional_driver was comparatively obligatory, score third, nevertheless inside the XGBoost model, it is the least obligatory operate.
• This highlights that obligatory choices can vary between fashions, emphasizing the need for a radical understanding of operate relationships with the dependent variable. Such discrepancies often come up ensuing from difficult operate interactions.

Among the many many prime three most important choices, two are engineered (km_per_hour and percent_sessions_in_last_month). Equally, among the many many prime 5, three are engineered. Engineered choices account for six out of the best 10 choices, underscoring the importance of operate engineering.

Our model’s effectivity was suboptimal, with a recall score of 0.18 indicating that it could solely decide 18% of exact churned clients. To boost our recall score, one approach is to lower the selection threshold.

By default, the sting is about to 0.5 for a lot of classification algorithms, along with scikit-learn. Which implies if our model predicts {{that a}} client has a 50% probability or elevated of churning, it assigns a predicted price of 1, indicating churn. Nonetheless, in cases of imbalanced datasets the place the minority class (churned clients) is of curiosity, this threshold won’t be ideally suited.

Now, let’s have a look on the precision-recall curve for the XGBoost model on the verify info.

# Plot precision-recall curve
present = PrecisionRecallDisplay.from_estimator(xgb_cv.best_estimator_,
X_test, y_test, determine='XGBoost')
plt.title('Precision-Recall Curve, XGBoost Model');

The precision-recall curve plot illustrates that as recall will enhance, the precision decreases, highlighting the trade-off between these two metrics for the XGBoost model. This trade-off is typical in classification duties, the place attaining elevated recall often comes on the value of lower precision, and vice versa.

In our Waze client churn prediction model, false negatives pose an enormous topic because of missing the chance to take proactive measures to cease churn is pricey, making recall important on this context. Alternatively, false positives means sending notifications to clients who acquired’t actually churn, which is way much less harmful.

Using the precision-recall curve as a info, we found that aiming for a recall of roughly 50% corresponds to a precision of spherical 30%. Subsequently, we decided to lower the sting from 0.5 to 0.18 to prioritize recall in our predictions. This adjustment is supposed to bolster our model’s recall effectivity, and we’ll assess whether or not or not this leads to an complete enchancment in model effectivity.

To try this, we’ll pay money for prospects using the predict_proba() function, which returns a 2-D array of prospects. Each row on this array represents a client, and the first column corresponds to the prospect of the unfavorable class (Retained), whereas the second column corresponds to the prospect of the constructive class (churned).

# Get predicted prospects on the verify info
predicted_probabilities = xgb_cv.best_estimator_.predict_proba(X_test)
predicted_probabilities

Any clients who’ve a probability price ≥ 0.18 inside the second column (churned class) of the predict_proba() output will most likely be assigned a predicted price of 1, indicating that they are predicted to churn.

# Create a list of merely the second column values (probability of objective)
probs = [x[1] for x in predicted_probabilities]
# Create an array of newest predictions that assigns a 1 to any price >= 0.4
new_preds = np.array([1 if x >= 0.18 else 0 for x in probs])
new_preds
# Ouput
array([0, 1, 0, ..., 1, 0, 1])

# Get evaluation metrics for when the sting is 0.18
get_test_scores('XGB, threshold 0.18', new_preds, y_test)

• With the lowered threshold of 0.18, we seen notable enhancements inside the recall score, rising from 0.18 to 0.53.
• Nonetheless, precision decreased from 0.42 to 0.29 with the lowered threshold, highlighting the trade-off the place elevated recall comes on the expense of precision.
• The F1 score improved from 0.23 to 0.37, indicating a higher stability between precision and recall as compared with the default threshold.
• Notably, the accuracy decreased from 0.81 to 0.69 when lowering the sting.

By lowering the selection threshold to 0.18, we aimed to realize a recall of spherical 0.50 (we achieved 0.53), which implies that the model now captures 53% of shoppers who actually churn. Nonetheless, this adjustment moreover decreased precision, indicating that when the model predicts a client will churn, it is applicable solely about 29% of the time. This trade-off highlights the importance of balancing recall and precision based mostly totally on specific enterprise goals and priorities in a classification course of.

On this enterprise, we carried out a Random forest and XGBoost model using a 60/20/20 lower up methodology, dividing the data into teaching, validation, and verify models. Through validation set predictions, we acknowledged XGBoost as a result of the champion model ensuing from its good effectivity as compared with random forest.

Subsequently, we assessed the model’s effectivity on the verify set, analyzed confusion metrics, and examined operate significance. To spice up recall, we carried out experiments by lowering the selection threshold, and evaluating the model’s responsiveness to this adjustment. Now, let’s proceed to answer some questions.

Would you recommend that Waze use this model? Why or why not?

• Every our fashions, Random Forest and XGBoost, did not receive satisfactory effectivity scores. Whereas XGBoost outperformed Random Forest, its effectivity was nonetheless underneath the deployment threshold. Our attempt to improve recall resulted in an increase, albeit on the expense of precision.
• The selection to utilize the model relies upon its meant operate. If the model is supposed to inform essential enterprise choices, then neither model meets the necessary necessities because of low recall and common precision seen. As a substitute, leveraging the model for added exploration and using the insights gained for refinement may current useful path.

What further choices would you favor to have to help improve the model?

• To boost our capability to predict client churn, we should always at all times accumulate info on utilization patterns, along with the events of day and days of the week when clients are most or least vigorous.
• We moreover need entry to location historic previous to find out frequent routes, and frequent places.
• Furthermore, accumulating strategies and evaluations will help us assess client satisfaction ranges, ache components, and understand the reasons behind client dissatisfaction.

Gathering and analyzing this information will enhance our understanding of client habits and improve our churn prediction capabilities.

What’s the advantage of using a logistic regression model over an ensemble of tree-based fashions (like random forest or XGBoost) for classification duties?

• Logistic regression fashions are easier to interpret. They assign coefficients to each predictor variable, which provides clear insights into the have an effect on of each operate on the model’s predictions.
• The magnitude and sign (constructive or unfavorable) of these coefficients instantly replicate the affect of the corresponding choices on the anticipated probability of the objective class.
• Optimistic coefficients level out a constructive relationship with the objective variable, whereas unfavorable coefficients level out a unfavorable relationship.

What’s the advantage of using an ensemble of tree-based fashions like random forest or XGBoost over a logistic regression model for classification duties?

• Ensemble tree-based fashions like Random Forest or XGBoost present a number of advantages over logistic regression for classification duties.
• These fashions are renowned for his or her predictive accuracy, making them good predictors in numerous eventualities. They’re sturdy to outliers and extreme values, requiring fewer assumptions regarding the underlying info distribution.
• In distinction to logistic regression, which assumes linearity, tree-based fashions can efficiently model difficult non-linear relationships inside the knowledge.
• Furthermore, tree-based fashions are well-suited for big datasets with fairly a number of choices, overcoming the challenges of the curse of dimensionality often encountered by logistic regression.