I build data platforms end to end — streaming ingestion, distributed processing, orchestration, and the AI layer on top. Most of my work lives at the point where data engineering meets applied AI: real-time pipelines feeding analytics, and retrieval systems that make that data useful to people and agents.
Everything below is a system I designed, built, and ran — not a tutorial follow-along.
Real-time e-commerce analytics on a lambda architecture. Kafka feeds four independent Spark Structured Streaming jobs into PostgreSQL and a MinIO data lake, reconciled nightly by Airflow. 27 Docker services · 36 automated tests · 9 bugs caught in live validation
|
End-to-end recruitment intelligence platform. Aggregates job offers via APIs and scrapers, extracts skills from CVs with NLP, and ranks candidate matches using embedding similarity. Medallion architecture · Airflow DAGs · FastAPI + Power BI surface Built with @mohamedamineelabidi
|
Retrieval-augmented search over an enterprise knowledge base. Hybrid pgvector and full-text retrieval with cross-encoder reranking, answers grounded in inline citations, and an abstention path that says "I do not know" rather than guessing when nothing clears the score floor. recall@5 0.49 → 0.91 · measured on a 100-query eval harness
|
An MCP server that lets AI agents understand a dataset without reading it. Returns a bounded JSON profile — types, ranges, null rates, quality flags — instead of pasted rows. Published on PyPI · 645 MB → 13 KB profile · CSV, Parquet, JSON, Excel
|
|
WhatBreaks Static breaking-change analysis for dbt. Column-level blast radius in CI, with no warehouse or credentials required. Python SQLGlot dbt
|
Doc Doctor A GitHub Action that executes the code examples in your docs and fails the PR when they break. TypeScript GitHub Actions
|
Procurement Pipeline Big-data procurement analytics pipeline built on a Hadoop and Presto stack, orchestrated with Airflow. Hadoop Presto Airflow
|
- sktime/sktime#10733 —
merged— fixed mutable default arguments in the vendored LagLlama code, where shared list and dict defaults leaked state across estimator instances. - dagster-io/dagster#34070 —
open— raise a clear error when partition-context methods are called on stub assets, instead of an opaque internal failure. - dagster-io/dagster#34069 —
open— docs fix removing duplicate op names from theproject_miniexample.
| Languages |
|
| Data Engineering |
|
| AI & ML |
|
| Storage |
|
| Infrastructure |
|
| Observability |
|