spark-optimization

Workload-aware ML framework for PySpark performance prediction and configuration recommendation, validated through real benchmark execution and out-of-distribution evaluation.

Research framework for cost-aware Apache Spark self-tuning, combining Bayesian Optimization, Safe Transfer Learning, and hybrid metaheuristics for adaptive performance optimization.

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning

Apache Spark native computation engine

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

For the GitHub description field: "Hands-on PySpark performance optimization repo — broadcast joins, partitioning, caching, bucketing, AQE, and skew handling, each with runnable before/after benchmarks and correctness tests.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.