morfAnalytics

Read in another language: English (this document) · Français.

morfAnalytics - the analysis engine of the morfSystem ecosystem. It offloads heavy analytics (long-period statistics, sensor correlations, anomaly detection, seasonal trends, cross-device comparisons, reports…) from the embedded devices.

morfAnalytics never owns the truth of the data. It works only on a local, read-only working cache copied from the devices. The source of truth stays on the devices (e.g. MeteoHub): the device writes, morfAnalytics reads - never the opposite, which the collector guarantees by issuing GET requests only. The cache can be deleted and rebuilt in full from the device without any loss. Its presence is always optional: if no server is available, the devices keep measuring, storing, charting and exporting exactly as before; only the advanced analyses become unavailable.

It also historises GitHub traffic consolidated by SiteWatch (views, clones, release downloads). It does not query GitHub or morfCollector for these metrics.

See the ecosystem vision in ../morfSystem/docs/ARCHITECTURE.md.

Status: operational. Incremental collection, thirteen weather analyses, cache cleanup and a web page are in place. Still to be written: publishing results to morfSync, and the correlation and anomaly-detection analyses.

Architecture note. The cache is not fed through morfSync: the device is not a morfSync client, as the sync envelope (UUID, revision, origin) would weigh more than the measurement itself on an ESP32 writing one every minute. morfSync is meant to distribute the analysis results to the ecosystem.

What the service does

• Incremental collection - copies the device history, never re-requesting what the cache already holds.
• Pluggable analysis engine - analyses only ever handle a generic time series with named channels. The weather analyses are just one set: another project registers its own without touching the engine.
• Analysis web page - served by the service itself, with no external asset (usable on a LAN with no Internet access).
• HTTP API (GET + POST) - GET / (web page), GET /analyses (catalog), GET /status (morfBeacon-compatible), GET /healthz, GET /modules, GET /modules/{id}, POST /analyze (on-demand analysis) and POST /data/cleanup (local-cache cleanup - never the source).
• Cache cleanup - from the page or the API: neutralise a time range or purge everything (failed-sensor readings, 0 hPa, are rejected at import). Points excluded by the quality rules can be listed and reintegrated one by one (service state, reversible). Acts on the local copy only; the original measurements on the device are never touched, and a purged cache rebuilds itself.
• Config - JSON file with a modules list; a factory turns it into modules.
• LAN announce - morfBeacon heartbeat (bundled, no external dependency).
• Service install - scripts/linux/ (systemd) and scripts/windows/ (Task Scheduler), copying binary + config to a fixed location.

Configure collection from MeteoHub

Set the device address in the analytics module (see config/morfanalytics.example.json):

{
  "type": "analytics",
  "id": "analytics-1",
  "maintenance_ms": 60000,      // delay between two collection cycles
  "cache_dir": "cache",         // working cache directory
  "source_url": "http://192.168.1.42"
}

• source_url - without it, no collection runs: the service only exposes the cache it already holds. Use that mode to analyse an already-copied history while the device is offline.
• No altitude here. MeteoHub publishes pressure already reduced to sea level, using the altitude of each of its sensors (hub System page). morfAnalytics analyses it as is and knows nothing of the installation: moving the probe is set in MeteoHub only. A leftover altitude_m parameter is ignored, with a warning at startup.

Then open http://<server-address>:8799/ to follow the collection. The first cycle copies the whole history stored on the SD card and may take several minutes; later cycles only transfer new measurements.

The cache is a plain SQLite file in cache_dir. It can be deleted at any time: it gets rebuilt from the device without loss, since MeteoHub remains the source of truth.

SiteWatch summaries are stored separately in /var/lib/morfsystem/morfanalytics/sitewatch-history.sqlite. This historical database contains only calculated summaries: SiteWatch remains the sovereign owner of the source access logs.

Photo library (morfPhoto)

The /photo space reads the photo library indexed by morfPhoto (the source of truth): camera bodies, lenses, focal lengths (bucketed into usual values), years, and the per-folder photographic context (context and subject, two independent dimensions qualified in PhotoHub). Presets cross these dimensions - "Focale naturelle" = DECOUVERTE + GENERAL, spontaneous vs prepared wildlife, events, shows - to compare them; they are only filter shortcuts, never a rule baked into the data. morfAnalytics only polls morfPhoto's aggregate endpoints at a regular interval (never the file list) and keeps a snapshot. Owned camera bodies are saved from the Configuration tab (POST /photo/practice) so the selection can be recalled and edited later. Each analysed station is listed once, by hostname when known (IP only as a fallback). Configure the source through the photo module of the configuration (source_url, e.g. http://127.0.0.1:8793); if morfPhoto is unreachable or the module is missing, the page says so explicitly.

Machine history (morfMonitor)

The /monitor space historises the machine metrics reported by morfMonitor (CPU, memory, temperature, load, active services) and charts them over time: an overview, CPU / RAM / temperature / load series, and a who-consumes-what per-service breakdown (top CPU and RAM consumers, plus per-service averages and maxima over the period), with a machine and period selector (1 h - 30 d). morfMonitor stays the probe ("now"); this domain provides the memory over time. A Supervision section reads morfMonitor's own temporal memory (contract morfhistory/1, read-through and cached, GET /monitor/history?machine=): overall availability, a per-service table for the current day (incidents by cause, downtime), a 24 h event chronology, availability/incident trend charts and a quarterly table. That memory is owned by morfMonitor; morfAnalytics only reads and represents it. Samples are stored in /var/lib/morfsystem/morfanalytics/monitor.sqlite, with a configurable raw-sample retention (retention_days, 90 by default; 0 = keep forever) as a first step before tiered compaction. Activities are historised too: any component that knows what it is doing posts one to POST /api/monitor/activity. Compilations are the first case: morfDeploy emits one per build (set MORFANALYTICS_ACTIVITY_URL on the build machine). JSON integer timestamps (start / end from Python) are accepted. The page shows per-project build stats (count, success rate, total/average/min/max duration, including seconds) plus the system cost measured around each build. Baseline and anomalies come next. Configure it through the monitor module (sources, one entry per morfMonitor to poll, plus interval_ms and retention_days).

Available analyses

Open http://<server-address>:8799/, or query one analysis directly:

curl -X POST -H 'Content-Type: application/json'      -d '{"type":"zambretti"}' http://localhost:8799/analyze

GET /analyses exposes the full catalog.

Conditions and local forecast

Analysis	type	What it gives
Current conditions	current	Dew point, absolute humidity, humidex, sea-level pressure
Pressure tendency	pressure_trend	1 h and 3 h change, WMO code, rapid-fall warning
Local forecast	zambretti	12-24 h text forecast derived from pressure
Fog risk	fog_risk	Dew-point spread and how fast it narrows
Frost risk	frost_risk	Projected minimum at dawn

Climatology

Analysis	type	What it gives
Daily normal	normals	Departure from the rolling normal for the day of year
Degree days	degree_days	Heating (base 18) and cooling (base 26), per month
Diurnal range	diurnal_amplitude	Max-min spread, mean and extremes
Records	records	Dated absolute minima and maxima
Notable days	streaks	Frost, heat, tropical nights, consecutive runs
Daily cycle	daily_cycle	Mean temperature per hour, extreme hours
Completeness	data_quality	Complete days, partial days and collection gaps

Comfort, building and forecast (context-aware)

Each analysis carries an intrinsic context - Outdoor (weather), Indoor (comfort), or Both (indoor/outdoor relation) - overridable per request with ctx=in|out|both, never with a silent cross-fallback. Indoor and outdoor are two separate caches; the forecast has its own.

Analysis	type	Context	What it gives
Indoor comfort	indoor_comfort	Indoor	Temperature/humidity comfort zones, dew point, mould-risk cue
Building thermal behaviour	thermal_behaviour	Both	Indoor/outdoor gap, damping, inertia lag
Indoor inertia model	indoor_inertia_model	Both	Fits indoor from lagged outdoor (gain, offset, R²/RMSE), predicted-vs-actual
Forecast vs observed	forecast_vs_observed	Outdoor	Day-ahead forecast vs observed min/max: bias and mean absolute error

Optional parameters: days (window depth), window_days (normals half-window), heating_base / cooling_base (degree days), ctx (context override).

Beyond the analyses, a Graphiques tab (/meteohub/graphs) plots the series over time - temperature, humidity, pressure, source indoor / outdoor / both - as the visual counterpart to the analyses. The window is either sliding (6 h to 30 d, ending now) or free: a "Période libre" button picks a start and end day + time, in 5-minute steps, to look back at a past moment. /meteohub/series and /meteohub/events accept from/to (epoch seconds) as well as hours.

Measurement quality (0.58.0). Measurements are never modified in the cache: they are qualified on read. A point outside physical bounds, an isolated spike (off from both neighbours, same direction, then back) or an outdoor measurement taken during a period annotated "sonde hors conditions" (probe brought indoors, on the bench) is kept out of the analyses and event detection. The Graphs tab shows it as a grey cross, with its reason on hover and a summary under the chart. Thresholds are cautious: better let a small artefact through than drop a real fast change.

An analysis short of history does not return an HTTP error: it answers ok: false with the reason and the required depth. The service did answer; it is the result that cannot be computed - better said plainly than published as an average over three measurements.

Acknowledged limits

These analyses rest on three quantities only - temperature, humidity, pressure. With no wind or sunshine data, Zambretti and the fog and frost risks remain local indications, not forecasts. Each result carries the matching note, displayed as-is on the page.

HTTP API

Every page of the interface is served by the service itself, and each one reads its data from a JSON route. The table lists them all, so a script (or another component) can use them directly.

Route	Role
GET /healthz · GET /status	Liveness, and the rich report of the morfSystem contract
GET /modules · GET /modules/<name>	Declared modules and their state
GET /, /meteohub, /meteohub/graphs, /sitewatch, /photo, /monitor, /github	HTML pages (shared dark / light theme, remembered by the browser; optional ?back=<url>&back_label=<name> adds a return link to the calling app)
GET /analyses	Catalogue of available analyses (the page builds itself from it)
POST /analyze	Run an analysis on demand
GET /meteohub/series · GET /meteohub/events	Down-sampled series and time events (crossings, trends, regimes) for the Graphs tab
GET·POST /meteohub/annotations, POST /meteohub/annotations/delete	Human weather observations
POST /data/cleanup	Purge the local working cache (never the device's measurements)
POST /sitewatch/ingest · GET /sitewatch/reports	SiteWatch summaries: reception, then history
POST /github/ingest · GET /github/data?repo=&from=&to=	GitHub traffic consolidated by SiteWatch
GET /photo/data?sources= · GET /photo/sources · GET·POST /photo/practice	Photo page data, known morfPhoto hosts, owned-camera scope
GET /monitor/data · GET /monitor/history?machine=	Machine metrics and supervision history (morfhistory/1)
POST /api/monitor/activity	Report an activity (build, indexing...) to historise
POST /api/monitor/forget	Forget a machine and its whole history ({"machine": "..."})

Build

Only needs Qt 6 (Core, Network, Sql). morfBeacon is vendored under third_party/morf/beacon.

cmake --preset mingw        # or linux / linux-arm64
cmake --build --preset mingw

Run

./build-mingw/service/morfanalytics.exe --config config/morfanalytics.example.json
curl http://127.0.0.1:8799/analyses

Without --config, the service looks for a configuration in the current directory, next to the binary, then in /etc/morfsystem/morfanalytics/; failing that it starts with a default analytics module, with no source, hence no collection.

Install as a service

# Any platform: Linux, Windows, Raspberry Pi
sudo ./service.py install      # build if needed, install, start
sudo ./service.py update       # rebuild, replace the binary, restart
sudo ./service.py uninstall    # deregister, keeping your configuration
./service.py status            # what the system says about it

One entry point everywhere. What this service is - its name, its directory, its configurations - is declared in service.json beside it. The four install steps live once for the whole parc; only the service manager differs by platform.

To redeploy the configuration to /etc/morfsystem/morfanalytics/ after editing it (a MeteoHub / morfPhoto / morfMonitor source of the monitor module...), without a full rebuild, use the parc's unified command (timestamped backup, then restarts; Linux and Windows alike):

sudo ./service.py config push --force     # or, from the parc root: morf config deploy morfAnalytics

Unlike service.py update (which only adds missing keys, never overwrites), config push replaces the deployed file with the one from the repo - keep a real config/morfanalytics.json in your clone, with your sources, as the deployed reference. (Use service.py config with no mode to only add a new version's keys while keeping your settings.)

Updating never replaces values already present in the configuration, but adds the parameters introduced since installation and reports them. Without that, a new feature would stay silently inactive for want of its parameter. Uninstalling removes the service and its installation directory (binary, configuration, cache); --keep-config backs the configuration up on the way out, --dry-run shows what would be removed. The git clone is never touched.

Documentation

French documentation in docs/fr/: architecture, execution flow, and the design decisions with their rationale. See also CHANGELOG · ROADMAP · CONTRIBUTING (including how to add an analysis).

License

GPL-3.0-only - © 2026 morfredus (Frédéric Biron).