Inaccurate demand forecasts erode margins, increase costs, and degrade customer experience. Traditional forecasting fails to scale across thousands of SKUs and store locations.
Key business consequences:
At scale: 125M+ rows of transaction data require distributed processing. Production data drift goes undetected without monitoring.
This system solves forecasting as a production ML engineering challenge. It demonstrates PySpark on AWS EMR, train-serve parity, S3 artifact management, FastAPI inference, Evidently drift monitoring, and CI/CD automation.
An end-to-end ML forecasting service that transforms raw retail data into production-ready demand predictions.
Built for scalability and reliability: PySpark feature engineering, XGBoost training, S3 storage, FastAPI inference, and automated drift detection.
Supporting operational decision-making across retail functions
Economic rationale for a mid-sized retailer with 50 stores and 3,000 SKUs:
| Cost Component | Annual Impact |
|---|---|
| Excess inventory holding costs | $2.5M |
| Stockout-related lost revenue | $1.8M |
| Waste and shrinkage from overstock | $1.2M |
| Expedited freight costs | $0.6M |
| Manual forecasting labor | $0.4M |
| Total Annual Cost | $6.5M |
| Savings Category | Annual Benefit |
|---|---|
| Inventory cost reduction | $250K |
| Stockout revenue recovery | $450K |
| Waste reduction | $144K |
| Freight cost avoidance | $120K |
| Analyst labor redeployment | $280K |
| Total Annual Benefit | $1.24M |
Payback period: ~3 months
Year 1 Net Benefit: $978K
3-Year NPV (10% discount rate): ~$2.8M
Cloud-native ML system with microservices architecture
13 sequential stages from data ingestion to deployment
Historical sales transactions, store metadata, and item attributes ingested from S3 and local storage in CSV/Parquet format.
CI/CD pipeline validates reference datasets for schema correctness, row volume sanity, and Parquet readability before deployment.
PySpark on AWS EMR constructs temporal, lag, rolling window, and store-level aggregation features with anti-leakage constraints.
XGBoost regression models trained with baseline and Optuna-tuned variants. Temporal train-validation split enforced.
MLflow tracks training runs, hyperparameters, evaluation metrics (RMSE, MAE), and model artifacts for reproducibility.
Trained models versioned via S3 key naming conventions with date-stamped filenames. Supports rollback and A/B testing.
XGBoost models serialized to JSON format for cross-language compatibility. CI/CD validates format integrity.
Containerized deployment using Docker and Compose. GitHub Actions builds, tests, and pushes to Amazon ECR.
FastAPI service loads model from S3. Includes request validation, online feature engineering, and graceful error handling.
Every prediction logged as structured JSON to date/hour partitioned S3 paths for audit trails and retraining data.
Evidently compares production logs against reference datasets. Generates HTML reports with drift share alerts.
System architecture supports manual retraining workflows. Prediction logs and drift reports provide retraining signals.
GitHub Actions executes S3 checks, model format validation, prediction quality tests, and Docker builds to ECR.
Temporal, lag, and aggregation features for seasonality and trends
Captures weekly and seasonal patterns
Explicit holiday demand uplift signals
Auto-correlation and temporal dependency
Smoothed signals and uncertainty metrics
Location and item baseline features
Prevents production feature skew
The full infrastructure code, PySpark ETL, Docker configuration, and API implementation are available on GitHub.