Skip to content
BranchesTags

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
config
config
 
 
dashboard
dashboard
 
 
infra
infra
 
 
notebooks
notebooks
 
 
pipeline
pipeline
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
requirements.txt
requirements.txt
 
 

Repository files navigation

SolarCast — MLOps Pipeline para Previsão de Geração Solar

🇧🇷 Português | 🇺🇸 English

Pipeline de machine learning de ponta a ponta para prever a potência gerada por um sistema fotovoltaico na hora seguinte, com rastreamento de experimentos via MLflow e infraestrutura na AWS SageMaker.

Português

Problema

Operadores de sistemas fotovoltaicos precisam saber com antecedência quanto vão gerar na próxima hora. Essa previsão permite balancear carga, negociar energia no mercado spot e reduzir desperdício. O problema é de regressão em série temporal: dado o estado meteorológico atual e o histórico recente, prever a potência gerada (W) na hora seguinte.

Dados

Item Detalhe
Fonte PVGIS — Joint Research Centre, União Europeia
Localização Sul de Minas Gerais — lat -22,3° / lon -45,0° / elevação 1336 m
Período 2015–2023 (9 anos de histórico horário)
Registros 78.888 linhas horárias, sem nulos, 100% dados diretos
Variáveis Potência (W), irradiância (W/m²), elevação solar, temperatura a 2m, vento a 10m, flag de interpolação

Estrutura do Repositório

solar-mlops/
├── data/
│   ├── raw/                  # JSON bruto do PVGIS (7,2 MB)
│   └── processed/            # Parquets intermediários e de features
├── pipeline/
│   ├── ingest.py             # Download e parsing do JSON bruto
│   ├── preprocess.py         # Engenharia de features e definição do target
│   ├── train.py              # Treinamento e rastreamento de experimentos
│   ├── evaluate.py           # Métricas e validação do modelo
│   └── deploy.py             # Deploy do endpoint e monitoramento
├── infra/
│   ├── inference.py          # Script de inferência para container SageMaker
│   ├── s3_setup.py
│   └── sagemaker_config.py
├── config/
│   └── params.yaml           # Hiperparâmetros e configurações
├── notebooks/
│   ├── 01_eda.ipynb
│   └── 02_baseline.ipynb
├── reports/
│   └── evaluation/           # Métricas, CSVs e gráficos gerados pelo evaluate.py
└── dashboard/
    └── app.py

Pipeline

1. Ingestão — pipeline/ingest.py

Lê o JSON bruto do PVGIS, normaliza o timestamp (YYYYMMDD:HHmm → datetime), renomeia as colunas para nomes legíveis e salva em Parquet.

python pipeline/ingest.py
# Entrada:  data/raw/pvgis_raw.json
# Saída:    data/processed/hourly.parquet

2. Pré-processamento — pipeline/preprocess.py

Filtra registros noturnos (irradiância = 0), constrói features temporais cíclicas, lags e médias móveis, e define o target.

python pipeline/preprocess.py
# Entrada:  data/processed/hourly.parquet
# Saída:    data/processed/features.parquet

Features geradas:

Grupo Colunas
Temporais cíclicas hour_sin/cos, doy_sin/cos, month_sin/cos
Temporais lineares hour, month, weekday
Lags (1h, 2h, 3h) power_lag_Nh, irradiance_lag_Nh
Médias móveis (3h, 6h) power_roll_Wh, irradiance_roll_Wh
Target target_power_next_h — potência da hora seguinte (W)

3. Treinamento — pipeline/train.py

Treina XGBoost ou LightGBM com split temporal (85% treino / 15% teste), validação cruzada via TimeSeriesSplit e rastreamento completo no MLflow. Registra o modelo no Model Registry se R² ≥ 0,85.

# Treino local
python pipeline/train.py --mode local

# Training Job no SageMaker
python pipeline/train.py --mode sagemaker

4. Avaliação — pipeline/evaluate.py

Avalia o modelo no hold-out temporal e gera relatório completo em reports/evaluation/.

python pipeline/evaluate.py
python pipeline/evaluate.py --model-type lightgbm
python pipeline/evaluate.py --run-id <mlflow_run_id>

Artefatos gerados:

  • evaluation_report.json — métricas globais e por hora/mês
  • predictions_vs_actual.png — série temporal dos últimos 14 dias do hold-out
  • scatter.png — real × previsto
  • residuals_histogram.png — distribuição dos resíduos
  • rmse_by_hour.png / rmse_by_month.png — análise de erros por período

5. Deploy — pipeline/deploy.py

Serve o modelo como endpoint REST.

# Servidor local (FastAPI + Uvicorn)
python pipeline/deploy.py --mode local
python pipeline/deploy.py --mode local --port 8080

# Endpoint no SageMaker
python pipeline/deploy.py --mode sagemaker --model-name solarcast-v1

Endpoints disponíveis (modo local):

Rota Descrição
GET /health Status do serviço e tipo do modelo carregado
POST /predict Predição para um único registro
POST /predict/batch Predição em lote (até 1000 registros)

Resultados

Métricas globais — hold-out temporal

Métrica Valor
RMSE 87,37 W
MAE 55,64 W
MAPE 59,87 %
R² 0,8691
Bias −0,63 W

Execute python pipeline/evaluate.py para preencher esta tabela com os resultados reais.

RMSE por hora do dia

O erro cresce no período da tarde, quando a irradiância é mais volátil:

Hora RMSE (W) MAE (W)
09h 63,8 52,9
11h 88,8 57,4
13h 101,9 67,0
15h 111,9 72,7
16h 121,7 83,0
19h 43,9 28,7
20h 19,9 14,4

RMSE por mês

O modelo tem desempenho significativamente melhor no inverno (baixa variabilidade de irradiância):

Mês RMSE (W) MAE (W)
Janeiro 106,9 76,1
Fevereiro 110,2 77,5
Junho 41,6 20,3
Julho 48,3 27,0
Outubro 100,2 70,4
Dezembro 98,3 67,3

Configuração

Todos os hiperparâmetros são centralizados em config/params.yaml:

model: xgboost          # xgboost | lightgbm
test_frac: 0.15
cv_splits: 5

mlflow:
  experiment: solarcast
  tracking_uri: sqlite:///mlflow.db

xgboost:
  n_estimators: 500
  learning_rate: 0.05
  max_depth: 6
  subsample: 0.8
  early_stopping_rounds: 30

Instalação

git clone https://github.com/seu-usuario/solar-mlops.git
cd solar-mlops
pip install -r requirements.txt

Dependências principais: xgboost, lightgbm, mlflow, fastapi, uvicorn, pandas, scikit-learn, boto3, sagemaker

Reproduzindo o pipeline completo

python pipeline/ingest.py
python pipeline/preprocess.py
python pipeline/train.py --mode local
python pipeline/evaluate.py
python pipeline/deploy.py --mode local

SolarCast — MLOps Pipeline for Solar Power Forecasting

End-to-end machine learning pipeline to forecast the next-hour power output of a photovoltaic system, with experiment tracking via MLflow and infrastructure on AWS SageMaker.

English

Problem

Operators of photovoltaic systems need to know in advance how much power they will generate in the next hour. This forecast enables load balancing, spot market energy trading, and waste reduction. The task is time-series regression: given the current meteorological state and recent history, predict the generated power (W) for the next hour.

Data

Item Detail
Source PVGIS — Joint Research Centre, European Union
Location Southern Minas Gerais, Brazil — lat -22.3° / lon -45.0° / elevation 1336 m
Period 2015–2023 (9 years of hourly history)
Records 78,888 hourly rows, no nulls, 100% direct measurements
Variables Power (W), irradiance (W/m²), solar elevation, temperature at 2m, wind speed at 10m, interpolation flag

Repository Structure

solar-mlops/
├── data/
│   ├── raw/                  # Raw JSON from PVGIS (7.2 MB)
│   └── processed/            # Intermediate and feature Parquets
├── pipeline/
│   ├── ingest.py             # JSON download and parsing
│   ├── preprocess.py         # Feature engineering and target definition
│   ├── train.py              # Training and experiment tracking
│   ├── evaluate.py           # Metrics and model validation
│   └── deploy.py             # Endpoint deployment and monitoring
├── infra/
│   ├── inference.py          # Inference script for SageMaker container
│   ├── s3_setup.py
│   └── sagemaker_config.py
├── config/
│   └── params.yaml           # Hyperparameters and settings
├── notebooks/
│   ├── 01_eda.ipynb
│   └── 02_baseline.ipynb
├── reports/
│   └── evaluation/           # Metrics, CSVs and charts generated by evaluate.py
└── dashboard/
    └── app.py

Pipeline

1. Ingestion — pipeline/ingest.py

Reads the raw PVGIS JSON, normalises the timestamp (YYYYMMDD:HHmm → datetime), renames columns to readable names and saves to Parquet.

python pipeline/ingest.py
# Input:   data/raw/pvgis_raw.json
# Output:  data/processed/hourly.parquet

2. Preprocessing — pipeline/preprocess.py

Filters nighttime records (irradiance = 0), builds cyclic temporal features, lags and rolling averages, and defines the target.

python pipeline/preprocess.py
# Input:   data/processed/hourly.parquet
# Output:  data/processed/features.parquet

Generated features:

Group Columns
Cyclic temporal hour_sin/cos, doy_sin/cos, month_sin/cos
Linear temporal hour, month, weekday
Lags (1h, 2h, 3h) power_lag_Nh, irradiance_lag_Nh
Rolling averages (3h, 6h) power_roll_Wh, irradiance_roll_Wh
Target target_power_next_h — next-hour power (W)

3. Training — pipeline/train.py

Trains XGBoost or LightGBM with a temporal split (85% train / 15% test), cross-validation via TimeSeriesSplit, and full tracking in MLflow. Registers the model in the Model Registry if R² ≥ 0.85.

# Local training
python pipeline/train.py --mode local

# SageMaker Training Job
python pipeline/train.py --mode sagemaker

4. Evaluation — pipeline/evaluate.py

Evaluates the model on the temporal hold-out and generates a full report under reports/evaluation/.

python pipeline/evaluate.py
python pipeline/evaluate.py --model-type lightgbm
python pipeline/evaluate.py --run-id <mlflow_run_id>

Generated artefacts:

  • evaluation_report.json — global metrics and per-hour / per-month breakdown
  • predictions_vs_actual.png — time series for the last 14 days of the hold-out
  • scatter.png — actual vs. predicted
  • residuals_histogram.png — residual distribution
  • rmse_by_hour.png / rmse_by_month.png — error analysis by period

5. Deployment — pipeline/deploy.py

Serves the model as a REST endpoint.

# Local server (FastAPI + Uvicorn)
python pipeline/deploy.py --mode local
python pipeline/deploy.py --mode local --port 8080

# SageMaker Endpoint
python pipeline/deploy.py --mode sagemaker --model-name solarcast-v1

Available routes (local mode):

Route Description
GET /health Service status and loaded model type
POST /predict Single-record prediction
POST /predict/batch Batch prediction (up to 1,000 records)

Results

Global metrics — temporal hold-out

Metric Value
RMSE 87,37 W
MAE 55,64 W
MAPE 59,87 %
R² 0,8691
Bias −0,63 W

Run python pipeline/evaluate.py to populate this table with real numbers.

RMSE by hour of day

Error rises during the afternoon, when irradiance is most volatile:

Hour RMSE (W) MAE (W)
09:00 63.8 52.9
11:00 88.8 57.4
13:00 101.9 67.0
15:00 111.9 72.7
16:00 121.7 83.0
19:00 43.9 28.7
20:00 19.9 14.4

RMSE by month

The model performs significantly better in winter, when irradiance variability is low:

Month RMSE (W) MAE (W)
January 106.9 76.1
February 110.2 77.5
June 41.6 20.3
July 48.3 27.0
October 100.2 70.4
December 98.3 67.3

Configuration

All hyperparameters are centralised in config/params.yaml:

model: xgboost          # xgboost | lightgbm
test_frac: 0.15
cv_splits: 5

mlflow:
  experiment: solarcast
  tracking_uri: sqlite:///mlflow.db

xgboost:
  n_estimators: 500
  learning_rate: 0.05
  max_depth: 6
  subsample: 0.8
  early_stopping_rounds: 30

Installation

git clone https://github.com/your-username/solar-mlops.git
cd solar-mlops
pip install -r requirements.txt

Main dependencies: xgboost, lightgbm, mlflow, fastapi, uvicorn, pandas, scikit-learn, boto3, sagemaker

Running the Full Pipeline

python pipeline/ingest.py
python pipeline/preprocess.py
python pipeline/train.py --mode local
python pipeline/evaluate.py
python pipeline/deploy.py --mode local

About

End-to-end MLOps pipeline for next-hour solar power forecasting — XGBoost · MLflow · FastAPI · SageMaker

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages