MTAnalytics
What do 17 years of subway data say about how New York actually moves? A Spark pipeline over the city's open data, ending in a model that predicts demand station by station.
Apache SparkPySparkXGBoostSHAPSocrata
20M+
Rows processed
472+
Stations covered
2008–2025
Span
What I built
Built a Spark/PySpark pipeline pulling five NYC Open Data feeds through the Socrata API with an adaptive sampler, processing 20M+ rows drawn from 100GB+ of source data across 472+ stations from 2008 to 2025.
Found the patterns that turn numbers into a portrait of the city: trips stay inside a borough 3–5× more often than they cross one, ridership has settled at 60–70% of pre-2019 levels, and several of the busiest stations still lack ADA access.
Ended with prediction, working with a collaborator: an XGBoost model trained on five million rows where SHAP showed that which station it is matters more to demand than what hour it is.