Project context and scope
Navigation platforms depend on active, recurrent user sessions to maintain real-time route optimization and crowdsourced traffic incident reporting. When drivers stop opening the app, overall network data quality declines.
This project examined user retention patterns using an authorized dataset provided by Waze. Structured around the PACE framework (Plan, Analyze, Construct, Execute), the workflow spanned six progressive milestones (C1 through C6), covering exploratory data analysis, statistical hypothesis testing, feature engineering, and predictive machine learning.
Exploratory data analysis and hypothesis testing
The initial analysis examined distribution differences between retained users and churned users across activity indicators:
- Driving frequency: Active users averaged noticeably higher driving days within 30-day windows. In contrast, users who churned often logged fewer driving days despite previously accumulating long individual driving distances.
- Device operating systems: A two-sample hypothesis t-test was conducted to assess whether driver churn differed between iPhone and Android users. The test yielded a p-value well above the standard 0.05 significance threshold, confirming that device type had no statistically significant relationship with user attrition.
Feature engineering and model selection
Because raw telemetry counts exhibited non-linear correlations with user attrition, several composite ratio features were engineered:
km_per_driving_day: Evaluates average daily travel intensity.percent_sessions_in_last_month: Measures recency and velocity of app engagement.total_sessions_per_day: Captures trip frequency patterns.
These engineered variables proved critical during modeling, accounting for six of the top ten features ranked by importance in tree-based algorithms.
Models were evaluated across training, validation, and held-out test splits:
| Model | Precision | Recall | F1 score | Accuracy |
|---|---|---|---|---|
| Logistic Regression (test) | 0.40 | 0.09 | 0.15 | 0.82 |
| Random Forest (validation) | 0.45 | 0.12 | 0.19 | 0.82 |
| XGBoost (validation) | 0.42 | 0.16 | 0.23 | 0.81 |
| XGBoost (test) | 0.42 | 0.18 | 0.25 | 0.81 |
Logistic regression achieved high overall accuracy (~82%), but had a recall of only ~0.09 on churned users due to heavy class imbalance.
XGBoost was selected as the champion model. Optimized with GridSearchCV (max_depth, min_child_weight, learning_rate, n_estimators), it doubled the recall of the logistic regression baseline and achieved 0.181 recall on the test dataset while maintaining 81.2% accuracy.
Confusion matrix and decision threshold analysis
Evaluating the confusion matrix under the standard 0.5 classification threshold showed that the model predicted approximately three times as many false negatives as false positives, correctly capturing 16.6% of actual churners on default settings.
For business retention campaigns, failing to detect an at-risk driver is typically far more costly than sending an unneeded retention prompt to an active driver. By analyzing the precision-recall curve, an adjusted operational threshold was identified:
- Lowering the classification decision threshold to 0.124 raised churn recall to 0.503 (capturing over half of all churned drivers) while keeping precision at approximately 30%.
- This provides an actionable tradeoff for growth and retention teams: selecting a calibrated threshold depending on whether the operational goal prioritizes notification volume or coverage.