Waze User Churn Prediction: Random Forest vs. XGBoost | by Lekha Topil | May, 2024

Welcome to the ultimate part of the Waze Person Churn Prediction Undertaking!

On this concluding part of our venture, we’re shifting past the logistic regression mannequin to discover extra superior methods utilizing Random Forest and XGBoost. Our purpose is to reinforce the predictive efficiency and effectiveness of our churn prediction mannequin. Beforehand, we accomplished key phases together with Exploratory Data Analysis (EDA), Hypothesis Testing, and the event of a Logistic Regression Model.

Waze, a free navigation app owned by Google, makes it simpler for drivers all over the world to succeed in their locations. Waze’s neighborhood of map editors, beta testers, translators, companions, and customers helps make every drive higher and safer.

Develop a machine studying mannequin to foretell consumer churn. An correct mannequin will assist stop churn, enhance consumer retention, and contribute to the expansion of Waze’s enterprise.

Earlier than continuing, let’s deal with some necessary questions:

1. What are you being requested to do?

• Predict whether or not a buyer will churn or stay retained on the Waze app.

2. What are the moral implications of the mannequin? What are the implications of your mannequin making errors?

1. What’s the doubtless impact of the mannequin when it predicts a false unfavorable (i.e. when the mannequin says a Waze consumer received’t churn, however they really will (Kind II Error))?

• Waze could miss the chance to take proactive steps to retain customers who’re prone to cease utilizing the app.
• By precisely figuring out at-risk customers, Waze can implement focused methods resembling sending customized emails showcasing app options, offering ideas for seamless navigation, or conducting surveys to know consumer ache factors and causes for potential churn.

2. What’s the doubtless impact of the mannequin when it predicts a false constructive (i.e. when the mannequin says a Waze consumer will churn, however they really received’t (Kind I Error))?

• Waze could take pointless proactive steps to retain customers who had been not at-risk of churning, doubtlessly resulting in actions that might annoy or irritate loyal customers.
• This can lead to elevated consumer dissatisfaction resulting from frequent and pointless emails or notifications, affecting consumer expertise.

It’s essential to strike a stability in mannequin accuracy to reduce each false positives and false negatives. This ensures that proactive retention efforts are focused particularly towards customers liable to churning, whereas avoiding any inconvenience or annoyance to different customers who usually are not prone to churn.

3. Do the advantages of such a mannequin outweigh the potential issues?

• If the mannequin performs effectively, it may assist Waze determine customers who’re liable to churning, enabling the corporate to implement proactive retention efforts and in the end improve retention charges.
• Nevertheless, there are moral issues associated to false predictions (false positives and false negatives). False positives might trigger pointless interventions with loyal customers, whereas false negatives would possibly imply missed alternatives to retain at-risk customers.

For full code implementation, you may go to my Kaggle Notebook

# Import normal operational packages
import pandas as pd
import numpy as np
# Packages for visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Import packages for information modeling
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import roc_auc_score, roc_curve, auc
from sklearn.metrics import accuracy_score, precision_score, recall_score, 
f1_score, confusion_matrix, ConfusionMatrixDisplay, RocCurveDisplay, PrecisionRecallDisplay
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
# operate that helps plot function significance
from xgboost import plot_importance

# Learn Knowledge
df = pd.read_csv('waze_dataset.csv')
# Show
df.head()

# Chceck information
df.information()

1. km_per_driving_day

• We’ll create a brand new function referred to as km_per_driving_day, which calculates the imply distance pushed per driving day for every consumer. This will probably be achieved by dividing the overall kilometers pushed (driven_km_drives) by the variety of driving days (driving_days).

There are some customers with zero driving_days, inflicting Pandas to assign infinity (inf) values to the corresponding rows within the new column resulting from undefined division by zero. Subsequently, it’s essential to convert these infinite values to zero.

# Create `km_per_driving_day` function
df['km_per_driving_day'] = df['driven_km_drives'] / df['driving_days']
# Convert infinite values to zero
df.loc[df['km_per_driving_day'] == np.inf, 'km_per_driving_day'] = 0
# Descriptive Statistics
df['km_per_driving_day'].describe()

2. percent_sessions_in_last_month

• Subsequent, we’ll create the percent_sessions_in_last_month function by dividing the variety of month-to-month periods (periods) by the estimated complete periods (total_sessions) because the consumer onboarded.
• This new variable will assist us perceive the proportion of complete periods that occurred inside the final month because the consumer was onboarded.

# Create `percent_sessions_in_last_month` function
df['percent_sessions_in_last_month'] = df['sessions'] / df['total_sessions']
# Descriptive Statistics
df['percent_sessions_in_last_month'].describe()

Based mostly on our EDA, we found that 50% of customers have 40% or extra of their periods occurring inside the final month. This discovering means that the percent_sessions_in_last_month function might be a helpful predictor for our mannequin.

3. professional_driver

The subsequent function we’ll create is professional_driver, which identifies customers based mostly on particular thresholds for the variety of drives (drives) and driving days (driving_days) within the final month. To be labeled as a professional_driver, a consumer should meet each of the next circumstances:

1. The consumer ought to have accomplished 60 or extra drives (drives) within the final month.
2. The consumer ought to have pushed for 15 or extra days (driving_days) within the final month.

# Create `professional_driver` function
df['professional_driver'] = np.the place((df['drives']>=60) & (df['driving_days'] >=15), 1, 0)
# Show
df[['drives', 'driving_days', 'professional_driver']].head()

In our earlier Logistic Regression Model, this function ranked because the third most necessary predictor based mostly on its coefficient within the function significance.

4. total_sessions_per_day

The subsequent function will probably be total_sessions_per_day, representing the imply variety of periods per day since onboarding.

• periods: The variety of prevalence of a consumer opening the app in the course of the month.
• total_sessions: A mannequin estimate of the overall variety of periods since a consumer has onboarded.

# Create `total_sessions_per_day` function
df['total_sessions_per_day'] = df['total_sessions'] / df['n_days_after_onboarding']
# Descriptive Statistic
df['total_sessions_per_day'].describe() 

5. km_per_hour

km_per_hour represents the imply kilometers per hour pushed within the final month. It’s calculated by dividing driven_km_drives by duration_minutes_drives and changing the end result into hours by additional dividing by 60.

• driven_km_drives: Whole kilometers pushed in the course of the month.
• duration_minutes_drives: Whole length pushed in minutes in the course of the month.

# Create `km_per_hour` function
df['km_per_hour'] = df['driven_km_drives'] / df['duration_minutes_drives'] / 60
# Descriptive Statistics
df['km_per_hour'].describe() 

6. km_per_drive

km_per_drive represents the imply variety of kilometers pushed per drive made by every consumer within the final month.

• driven_km_drives: Whole kilometers pushed in the course of the month.
• drives: Whole length pushed in minutes in the course of the month.

# Create `km_per_drive` function
df['km_per_drive'] = df['driven_km_drives'] / df['drives']
# Convert infinite values to zero
df.loc[df['km_per_drive']==np.inf, 'km_per_drive'] = 0
# Descriptive Statistic
df['km_per_drive'].describe()

7. percent_of_sessions_to_favorite

• Lastly, we’ll create percent_of_sessions_to_favorite, which represents the proportion of complete periods directed in direction of a consumer’s favourite locations.
• We lack information on the complete drives since customers had been onboarded, we’ll use total_sessions as an inexpensive estimate or proxy for the overall variety of drives.

This method allows us to achieve insights into consumer habits and preferences based mostly on the out there information.

# Create `percent_of_sessions_to_favorite` function
df['percent_of_sessions_to_favorite'] = (df['total_navigations_fav1'] +
df['total_navigations_fav2']) / df['total_sessions']
df['percent_of_sessions_to_favorite'].describe()

Customers whose drives to non-favorite locations represent the next share of their complete drives may be less prone to churn, as they’re exploring new places. Conversely, customers who allocate the next share of their drives to favourite locations may be extra prone to churn, probably resulting from their familiarity with the routes.

Throughout our earlier Exploratory Data Analysis (EDA), we recognized 700 lacking values within the dataset. These lacking values are believed to be lacking at random (MAR), as there isn’t any proof of a non-random trigger. Provided that the lacking values characterize lower than 5% of the overall dataset, eradicating them is unlikely to have a big influence on the dataset’s integrity.

# Drop rows with lacking values
df = df.dropna(subset=['label'])
# Dimension after removig lacking information
print('Dimension after eradicating lacking information:', df.form)
# Output:
Dimension after eradicating lacking information: (14299, 20)

Based mostly on our earlier Exploratory Data Analysis (EDA), we recognized outliers and excessive values in a number of columns. Nevertheless, tree-based fashions are strong to outliers, eliminating the necessity for imputation or outlier removing.

Our Exploratory Data Analysis (EDA) revealed robust correlations amongst sure variables, indicating multicollinearity. Provided that tree-based fashions can successfully handle collinearities between impartial variables, there isn’t any must take away any variables besides the ID column, which holds no relevance to churn prediction.

# Drop `ID` column
df = df.drop(['ID'], axis=1)

Dummying Options

We now have one categorical variable, gadget, amongst our impartial variables, which consists of two teams: iPhone and Android. Whereas there are a number of strategies out there, resembling pd.get_dummies() or OneHotEncoder(), for encoding categorical variables, we’ll use a easy method with np.the place():

• Assign 1 for iPhone
• Assign 0 for Android

# Create new `device2` variable
df['device2'] = np.the place(df['device']=='Android', 0, 1)
# Test
df[['device', 'device2']].tail()

Lastly, we’ll encode the goal variable label into binary format:

• Assign 0 for all Retained
• Assign 1 for all Churned

# Create binary `label2` column
df['label2'] = np.the place(df['label']=='churned', 1, 0)
# Test
df[['label', 'label2']].tail()

# Get class stability of 'label' col
df['label'].value_counts(normalize=True) * 100

• Our dataset reveals class imbalance, with 17.73% of customers churned and 82.26% retained — typical for churn prediction datasets. Though the imbalance is noticeable, it’s not excessive and may be addressed with out rebalancing courses. Tree-based fashions like Random Forest and XGBoost excel in dealing with such imbalances.
• For our imbalanced dataset, accuracy is an unsuitable analysis metric. A false constructive, the place the mannequin incorrectly predicts churn for a consumer who will really keep, would possibly result in pointless retention measures, doubtlessly irritating customers. Nevertheless, false positives don’t lead to direct monetary losses or different vital penalties.
• Alternatively, false negatives, the place the mannequin fails to foretell churn for customers who really churn, are important. The corporate can not afford false negatives as a result of they stop proactive measures to retain at-risk customers, doubtlessly rising the churn fee. Subsequently, we’ll prioritize choosing the mannequin based mostly on the recall rating.

Modeling workflow and mannequin choice course of

The ultimate dataset used for modeling consists of 14,299 samples, which is on the decrease finish of what’s usually thought of ample for conducting a sturdy mannequin choice course of. Nevertheless, it nonetheless supplies an ample foundation for our evaluation.

Break up the info

• Break up the info into prepare/validation/check units (60/20/20)

We’ll observe these steps:

1. Outline X
2. Outline y
3. Break up the info into an interim coaching set and a check set utilizing an 80/20 ratio.
4. Additional cut up the interim coaching set right into a coaching set and a validation set utilizing a 75/25 ratio, leading to a remaining ratio of 60/20/20 for coaching/validation/check units.

# Isolate X variables
X = df.drop(columns=['label', 'label2', 'device'])
# Isolate y variable
y = df['label2']
# Break up into prepare and check units
X_tr, X_test, y_tr, y_test = train_test_split(X, y, stratify=y, test_size=0.2, random_state=42)
# Break up into prepare and validate units
X_train, X_val, y_train, y_val = train_test_split(X_tr, y_tr, stratify=y_tr, test_size=0.25, random_state=42)

Now, let’s confirm the variety of samples within the partitioned information.

# Test the size of the info
data_len = [len(x) for x in (X_train, X_val, X_test)]
data_len
# Output:
[8579, 2860, 2860]

# 1. Instantiate the random forest classifier
rf = RandomForestClassifier(random_state=42)
# 2. Create a dictionary of hyperparameters to tune
cv_params  = {'max_depth': [None],
'max_features': [1.0],
'max_samples': [1.0],
'min_samples_leaf': [2],
'min_samples_split':[2],
'n_estimators': [300]}
# 3. Outline a dictionary of scoring metrics to seize
scoring = {'accuracy', 'precision', 'recall', 'f1'}
# 4. Instantiate the GridSearchCV object
rf_cv = GridSearchCV(rf, cv_params, scoring=scoring, cv=4, refit='recall')

%%time
rf_cv.match(X_train, y_train)
# Output:
CPU occasions: consumer 2min 26s, sys: 115 ms, complete: 2min 26s
Wall time: 2min 26s

# Look at finest rating
rf_cv.best_score_
# Ouput
0.12678201409034398

# Look at finest hyperparameter combo
rf_cv.best_params_

def make_results(model_name: str, model_object, metric: str):
'''
Arguments:
model_name (string): what you need the mannequin to be referred to as within the output desk (identify of the mannequin)
model_object: a match GridSearchCV object
metric (string): precision, recall, f1, and accuracy
Returns a pandas df with precision, recall, f1, and accuracy scores
for the mannequin with the very best imply 'metric' rating throughout all validation folds.
'''
# Map the metric identify to the corresponding column in cv_results
metric_dict = {
'precision': 'mean_test_precision',
'recall': 'mean_test_recall',
'f1': 'mean_test_f1',
'accuracy': 'mean_test_accuracy'
}
# Extract cv_results from the GridSearchCV object
cv_results = pd.DataFrame(model_object.cv_results_)
# Determine the row index with the utmost worth for the required metric
best_index = cv_results[metric_dict[metric]].idxmax()
# Retrieve the metrics for the best-performing mannequin configuration
metrics = {
'mannequin': model_name,
'precision': cv_results.loc[best_index, 'mean_test_precision'],
'recall': cv_results.loc[best_index, 'mean_test_recall'],
'F1': cv_results.loc[best_index, 'mean_test_f1'],
'accuracy': cv_results.loc[best_index, 'mean_test_accuracy']
}
# Create a DataFrame containing the extracted metrics
result_table = pd.DataFrame(metrics, index=[0])
return result_table

# Get the scores
outcomes = make_results('RF CV', rf_cv, 'recall')
outcomes

• The precision, recall, and F1 scores from the Random Forest mannequin are under the specified thresholds.
• The recall for the Random Forest mannequin is 0.12, indicating that the mannequin predicts extra false negatives. A recall of 0.12 implies that solely 12% of the particular churned instances are appropriately recognized by the mannequin. In different phrases, the mannequin misses a good portion of customers who really churned, failing to seize them in its predictions.
• In comparison with our earlier Logistic Regression model, the recall has improved from 0.09 to 0.12 within the Random Forest mannequin, representing a 33% improve.
• Nevertheless, precision decreased from 0.52 (Logistic Regression model) to 0.458, and the F1-score confirmed a slight enchancment from 0.16 (Logistic Regression model) to 0.199. Lastly, accuracy stays nearly the identical between the 2 fashions.

# 1. Instantiate the XGBoost classifier
xgb = XGBClassifier(goal='binary:logistic', random_state=42)
# 2. Create a dictionary of hyperparameters to tune
cv_params = {'max_depth': [6, 12],
'min_child_weight': [3, 5],
'learning_rate': [0.01, 0.1],
'n_estimators': [300]
}
# 3. Outline a dictionary of scoring metrics to seize
scoring = {'accuracy', 'precision', 'recall', 'f1'}
# 4. Instantiate the GridSearchCV object
xgb_cv = GridSearchCV(xgb, cv_params, scoring=scoring, cv=4, refit='recall')

%%time
xgb_cv.match(X_train, y_train)
# Output
CPU occasions: consumer 2min 59s, sys: 1.3 s, complete: 3min 1s
Wall time: 46.3 s

# Look at finest rating
xgb_cv.best_score_
# Ouput
0.1708264263019754

# Look at finest parameters
xgb_cv.best_params_

# Name 'make_results()' on the GridSearch object
xgb_cv_results = make_results('XGB CV', xgb_cv, 'recall')
outcomes = pd.concat([results, xgb_cv_results], axis=0)
outcomes

• The XGBoost mannequin performs higher in comparison with the Random Forest mannequin, notably with its recall rating, which is almost double that of the Logistic Regression mannequin and about 35% increased than the Random Forest mannequin.
• Nevertheless, regardless of these enhancements, the XGBoost mannequin can solely seize round 17% of precise churned instances, indicating a notable presence of false negatives.
• Moreover, whereas precision has barely decreased, the F1-score reveals enchancment over each Logistic Regression model and Random Forest fashions.

Now let’s make predictions on the validation set utilizing each the Random Forest and XGBoost fashions. We’ll consider their efficiency, and the mannequin that performs higher on the validation set will probably be chosen because the champion mannequin.

For this function, we’ll outline a operate referred to as get_test_scores() to generate a desk of scores based mostly on the predictions made on the validation information.

Random Forest

# Use random forest mannequin to foretell on validation information
rf_val_preds = rf_cv.best_estimator_.predict(X_val)

def get_test_scores(model_name:str, preds, y_test_data):
'''
Generate a desk of check scores.
In:
model_name (string): identify of the mannequin of your alternative
preds: numpy array of check predictions
y_test_data: numpy array of y_test information
Out:
desk: a pandas df of precision, recall, f1, and accuracy scores to your mannequin
'''
# Calculate analysis metrics
metrics = {
'precision': precision_score,
'recall': recall_score,
'F1': f1_score,
'accuracy': accuracy_score
}
scores = {metric: metrics[metric](y_test_data, preds) for metric in metrics}
# Create DataFrame
desk = pd.DataFrame({
'mannequin': [model_name],
**scores
})
return desk

# Get validation scores for RF mannequin
rf_val_scores = get_test_scores('RF Val', rf_val_preds, y_val)
# Append to the outcomes desk
outcomes = pd.concat([results, rf_val_scores], axis=0)
outcomes

The scores for the validation set (RF Val) are barely decrease in comparison with the coaching set (RF CV). The diploma of variation in scores between the coaching and validation units is inside a suitable vary, suggesting that the mannequin shouldn’t be overfitting the coaching information.

XGBoost

# XGBoost mannequin to foretell on validation information
xgb_val_preds = xgb_cv.best_estimator_.predict(X_val)
# Get validation scores for XGBoost mannequin
xgb_val_scores = get_test_scores('XGB val', xgb_val_preds, y_val)
# Append to the outcomes desk
outcomes = pd.concat([results, xgb_val_scores], axis=0)
outcomes

• The XGBoost mannequin’s efficiency on the validation set (XGB Val) reveals a slight lower in scores in comparison with the coaching set (XGB CV).
• This minor decline falls inside a suitable vary, suggesting that the XGBoost mannequin didn’t overfit the coaching information.
• Among the many evaluated fashions (Random Forest and XGBoost), the XGBoost mannequin emerges because the clear champion based mostly on its efficiency on the validation set.

Because the XGBoost mannequin carried out higher in comparison with the random forest and logistic regression fashions, we’ll proceed by utilizing the XGBoost mannequin to make predictions on the check dataset. This step will enable us to evaluate the mannequin’s efficiency on new and unseen information, offering insights into its effectiveness for future predictions.

# XGBoost mannequin to foretell on check information
xgb_test_preds = xgb_cv.best_estimator_.predict(X_test)
# Get check scores for XGBoost mannequin
xgb_test_scores = get_test_scores('XGB check', xgb_test_preds, y_test)
# Append to the outcomes desk
outcomes = pd.concat([results, xgb_test_scores], axis=0)
outcomes

• The recall on the check set (0.181) is increased than that on the validation set (0.161), indicating a slight enchancment within the mannequin’s capacity to determine true positives (churned customers) in new and unseen information.
• Moreover, there was a slight improve in precision, and the F1 rating additionally improved.

General, the comparability between check and validation scores for the XGBoost mannequin means that the mannequin’s efficiency barely improved when evaluated on the check set.

Now, we’ll create a confusion matrix to visualise the XGBoost mannequin’s predictions on the check information.

# Generate array of values for confusion matrix
cm = confusion_matrix(y_test, xgb_test_preds, labels=xgb_cv.classes_)
# Plot confusion matrix
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=['retained', 'churned'])
disp.plot();

True Unfavorable (TN): 2228

• These are the situations appropriately predicted as not churned by the mannequin.
• In our case, the mannequin precisely recognized 2228 retained customers.

False Optimistic (FP): 125

• These are situations the place the mannequin incorrectly predicted as “churned” when the precise end result was “retained”.

False Unfavorable (FN): 415

• These situations characterize instances the place the mannequin predicted “retained”, however the precise end result was “churned”, indicating a big variety of false negatives.
• The mannequin predicted 3 times extra false negatives (415) than false positives (92).

True Optimistic (TP): 92

• These are situations the place the mannequin appropriately recognized 92 churned customers, which is considerably fewer in comparison with the variety of false negatives.

The ratio of false negatives to false positives is roughly 4.51 (415/92 ≈ 4.51), indicating that false negatives happen extra ceaselessly than false positives within the mannequin’s predictions.

The plot_importance() operate in XGBoost permits us to visualise the necessary options of our educated mannequin. This visualization signifies which options are most influential in making predictions, offering insights into the important thing components driving the mannequin’s selections.

# Plot function significance for XGBoost mannequin
plot_importance(xgb_cv.best_estimator_);

• In line with the function significance of the XGBoost mannequin, km_per_hour considerably influences predictions. In contrast to logistic regression, which closely weighted a single function (activity_days), the XGBoost mannequin makes use of many options.
• In our logistic regression mannequin, professional_driver was comparatively necessary, rating third, however within the XGBoost mannequin, it’s the least necessary function.
• This highlights that necessary options can range between fashions, emphasizing the necessity for a radical understanding of function relationships with the dependent variable. Such discrepancies usually come up resulting from complicated function interactions.

Among the many prime three most necessary options, two are engineered (km_per_hour and percent_sessions_in_last_month). Equally, among the many prime 5, three are engineered. Engineered options account for six out of the highest 10 options, underscoring the significance of function engineering.

Our mannequin’s efficiency was suboptimal, with a recall rating of 0.18 indicating that it may solely determine 18% of precise churned customers. To enhance our recall rating, one technique is to decrease the choice threshold.

By default, the edge is about to 0.5 for many classification algorithms, together with scikit-learn. Which means if our mannequin predicts {that a} consumer has a 50% chance or increased of churning, it assigns a predicted worth of 1, indicating churn. Nevertheless, in instances of imbalanced datasets the place the minority class (churned customers) is of curiosity, this threshold might not be ideally suited.

Now, let’s take a look on the precision-recall curve for the XGBoost mannequin on the check information.

# Plot precision-recall curve
show = PrecisionRecallDisplay.from_estimator(xgb_cv.best_estimator_,
X_test, y_test, identify='XGBoost')
plt.title('Precision-Recall Curve, XGBoost Mannequin');

The precision-recall curve plot illustrates that as recall will increase, the precision decreases, highlighting the trade-off between these two metrics for the XGBoost mannequin. This trade-off is typical in classification duties, the place attaining increased recall usually comes at the price of decrease precision, and vice versa.

In our Waze consumer churn prediction mannequin, false negatives pose a big subject as a result of lacking the possibility to take proactive measures to stop churn is dear, making recall essential on this context. Alternatively, false positives means sending notifications to customers who received’t really churn, which is much less dangerous.

Utilizing the precision-recall curve as a information, we discovered that aiming for a recall of roughly 50% corresponds to a precision of round 30%. Subsequently, we determined to decrease the edge from 0.5 to 0.18 to prioritize recall in our predictions. This adjustment is meant to reinforce our mannequin’s recall efficiency, and we’ll assess whether or not this results in an total enchancment in mannequin efficiency.

To do that, we’ll get hold of possibilities utilizing the predict_proba() operate, which returns a 2-D array of possibilities. Every row on this array represents a consumer, and the primary column corresponds to the chance of the unfavorable class (Retained), whereas the second column corresponds to the chance of the constructive class (churned).

# Get predicted possibilities on the check information
predicted_probabilities = xgb_cv.best_estimator_.predict_proba(X_test)
predicted_probabilities

Any customers who’ve a chance worth ≥ 0.18 within the second column (churned class) of the predict_proba() output will probably be assigned a predicted worth of 1, indicating that they’re predicted to churn.

# Create an inventory of simply the second column values (chance of goal)
probs = [x[1] for x in predicted_probabilities]
# Create an array of latest predictions that assigns a 1 to any worth >= 0.4
new_preds = np.array([1 if x >= 0.18 else 0 for x in probs])
new_preds
# Ouput
array([0, 1, 0, ..., 1, 0, 1])

# Get analysis metrics for when the edge is 0.18
get_test_scores('XGB, threshold 0.18', new_preds, y_test)

• With the lowered threshold of 0.18, we noticed notable enhancements within the recall rating, rising from 0.18 to 0.53.
• Nevertheless, precision decreased from 0.42 to 0.29 with the lowered threshold, highlighting the trade-off the place increased recall comes on the expense of precision.
• The F1 rating improved from 0.23 to 0.37, indicating a greater stability between precision and recall in comparison with the default threshold.
• Notably, the accuracy decreased from 0.81 to 0.69 when reducing the edge.

By reducing the choice threshold to 0.18, we aimed to attain a recall of round 0.50 (we achieved 0.53), which means that the mannequin now captures 53% of customers who really churn. Nevertheless, this adjustment additionally decreased precision, indicating that when the mannequin predicts a consumer will churn, it’s appropriate solely about 29% of the time. This trade-off highlights the significance of balancing recall and precision based mostly on particular enterprise aims and priorities in a classification process.

On this venture, we carried out a Random forest and XGBoost mannequin utilizing a 60/20/20 cut up methodology, dividing the info into coaching, validation, and check units. Via validation set predictions, we recognized XGBoost because the champion mannequin resulting from its good efficiency in comparison with random forest.

Subsequently, we assessed the mannequin’s efficiency on the check set, analyzed confusion metrics, and examined function significance. To boost recall, we performed experiments by reducing the choice threshold, and evaluating the mannequin’s responsiveness to this adjustment. Now, let’s proceed to reply some questions.

Would you suggest that Waze use this mannequin? Why or why not?

• Each our fashions, Random Forest and XGBoost, didn’t obtain passable efficiency scores. Whereas XGBoost outperformed Random Forest, its efficiency was nonetheless under the deployment threshold. Our try to enhance recall resulted in a rise, albeit on the expense of precision.
• The choice to make use of the mannequin is dependent upon its meant function. If the mannequin is meant to tell important enterprise selections, then neither mannequin meets the mandatory requirements as a result of low recall and average precision noticed. As an alternative, leveraging the mannequin for additional exploration and utilizing the insights gained for refinement might present helpful path.

What extra options would you prefer to have to assist enhance the mannequin?

• To enhance our capacity to foretell consumer churn, we should always collect information on utilization patterns, together with the occasions of day and days of the week when customers are most or least lively.
• We additionally want entry to location historical past to determine frequent routes, and frequent locations.
• Moreover, accumulating suggestions and evaluations will assist us assess consumer satisfaction ranges, ache factors, and perceive the explanations behind consumer dissatisfaction.

Gathering and analyzing this data will improve our understanding of consumer habits and enhance our churn prediction capabilities.

What’s the good thing about utilizing a logistic regression mannequin over an ensemble of tree-based fashions (like random forest or XGBoost) for classification duties?

• Logistic regression fashions are simpler to interpret. They assign coefficients to every predictor variable, which supplies clear insights into the affect of every function on the mannequin’s predictions.
• The magnitude and signal (constructive or unfavorable) of those coefficients immediately replicate the influence of the corresponding options on the expected chance of the goal class.
• Optimistic coefficients point out a constructive relationship with the goal variable, whereas unfavorable coefficients point out a unfavorable relationship.

What’s the good thing about utilizing an ensemble of tree-based fashions like random forest or XGBoost over a logistic regression mannequin for classification duties?

• Ensemble tree-based fashions like Random Forest or XGBoost provide a number of benefits over logistic regression for classification duties.
• These fashions are famend for his or her predictive accuracy, making them nice predictors in lots of eventualities. They’re strong to outliers and excessive values, requiring fewer assumptions in regards to the underlying information distribution.
• In contrast to logistic regression, which assumes linearity, tree-based fashions can successfully mannequin complicated non-linear relationships inside the information.
• Moreover, tree-based fashions are well-suited for giant datasets with quite a few options, overcoming the challenges of the curse of dimensionality usually encountered by logistic regression.