🚕 Uber NYC Hotspot Analysis — Demand Clustering

Geospatial demand analysis of Uber pickups in New York City using KMeans and DBSCAN clustering to identify high-demand hotspots by day and hour.

---

Overview

Business question: Where should Uber drivers position themselves to maximise pickups at different times of the week?

This project analyses ~14 million Uber pickups (Jan–Jun 2015) in New York City to:

1. Identify peak demand hours for each day of the week
2. Detect geographic hotspot clusters at peak times using two algorithms:
   • KMeans: fixed number of clusters, always produces k zones
   • DBSCAN: density-based, handles irregular shapes and noise, no fixed k
3. Compare both algorithms and provide actionable driver positioning recommendations

---

Key Findings

Day	Peak Hour	Top Hotspot	KMeans k
Monday–Friday	18:00–19:00	Midtown Manhattan	7–9
Friday	22:00–23:00	Lower Manhattan + Williamsburg	8
Saturday	23:00–00:00	Midtown + Lower East Side	9
Sunday	17:00–18:00	Midtown Manhattan	7

KMeans vs DBSCAN: KMeans gives stable, predictable zones suitable for fleet dispatch planning. DBSCAN reveals organically shaped demand clusters and correctly identifies outliers (isolated pickups far from city core).

---

Architecture

uber-raw-data-janjune-15.csv (14M rows, Jan–Jun 2015)
taxi-zone-lookup.csv (265 NYC taxi zones)
          │
          ▼
┌─────────────────────────────────────────┐
│  01_eda_exploration.ipynb               │
│  • Column inspection & type fixing      │
│  • Missing value analysis               │
│  • LocationID join key validation       │
│  • Pickup distribution by zone/borough  │
└─────────────────────────────────────────┘
          │
          ▼
┌─────────────────────────────────────────┐
│  02_hotspot_clustering.ipynb            │
│  • Geocode zones via GeoParquet         │
│  • Temporal EDA (hour/day patterns)     │
│  • KMeans (auto-k via silhouette)       │
│  • DBSCAN (haversine metric, radians)   │
│  • 7× interactive maps (one per day)    │
│  • Business insights & recommendations  │
└─────────────────────────────────────────┘

---

Project Structure

Uber/
├── notebooks/
│   ├── 01_eda_exploration.ipynb        # Data quality, column profiling
│   └── 02_hotspot_clustering.ipynb     # Main clustering analysis
├── data/
│   ├── taxi-zone-lookup.csv            # NYC taxi zone metadata (265 zones)
│   └── sample/
│       └── uber_sample_10k.csv         # 10k row sample for quick testing
├── tests/
│   └── test_clustering.py              # Unit tests for pure functions
├── requirements.txt
├── .gitignore
└── README.md

Full data: The complete uber-raw-data-janjune-15.csv (512 MB) is excluded from this repo due to GitHub's file size limit. Download it from Kaggle — Uber Pickups in New York City and place it in the data/ folder.

---

Installation

1. Clone the repository

   git clone https://github.com/anafiiliipa-dev/Uber.git
   cd Uber

2. Create and activate a virtual environment

   python -m venv .venv
   source .venv/bin/activate    # Linux / macOS
   .venv\Scripts\activate       # Windows

3. Install dependencies

   pip install -r requirements.txt

4. Download the full dataset

   • Go to Kaggle — Uber Pickups in New York City
   • Download uber-raw-data-janjune-15.csv
   • Place it in data/uber-raw-data-janjune-15.csv

5. Run the notebooks

   jupyter notebook

   Run 01_eda_exploration.ipynb first, then 02_hotspot_clustering.ipynb.

---

Running Tests

pytest tests/ -v

---

Data Sources

File	Source	Description
uber-raw-data-janjune-15.csv	Kaggle / FiveThirtyEight	14M Uber pickups Jan–Jun 2015
taxi-zone-lookup.csv	NYC TLC	265 NYC taxi zones with borough mapping
taxi_zones_4326.parquet	Source Cooperative	GeoParquet with zone polygons (auto-downloaded)

---

Algorithms

KMeans

• Auto-selects k (3–12) via silhouette score maximisation
• Coordinates scaled with StandardScaler before clustering
• Gives stable, predictable zones — ideal for fleet dispatch planning

DBSCAN

• Uses haversine metric (true geographic distance)
• Input coordinates converted to radians (required by sklearn's haversine)
• eps expressed in km, converted to radians: eps_rad = eps_km / 6371.0088
• Points labelled -1 are noise/outliers — isolated pickups far from any hotspot

---

Author

Ana Gouveia — DSFS-OD-14 cohort