feat(serial_plotter): binary telemetry service + WebUSB transport (Phase 2)

.github/workflows/build.yml

@@ -284,6 +284,9 @@ jobs:
           target: esp32s3
         - path: 'components/seeed-studio-round-display/example'
           target: esp32s3
+        - path: 'components/serial_plotter/example'
+          target: esp32s3
+          command: 'IDF_COMPONENT_MANAGER=0 idf.py build'
         - path: 'components/serialization/example'
           target: esp32
         - path: 'components/smartpanlee-sc01-plus/example'

components/serial_plotter/CMakeLists.txt

@@ -1,8 +1,11 @@
-# serial_plotter currently ships only a self-contained browser webapp
-# (web/serial_plotter.html), auto-hosted by the docs "web apps" pipeline and
-# usable offline over file://. There is no firmware source yet: a telemetry
-# service (headers + example) built on stream_frame / dispatcher is a planned
-# follow-up. This registration keeps the directory a valid ESP-IDF component so
-# repo tooling and the component registry stay consistent; it gains INCLUDE_DIRS
-# and REQUIRES when the telemetry service lands.
-idf_component_register()
+# The serial_plotter component ships:
+#  - web/serial_plotter.html: the browser data plotter (Web Serial + WebUSB),
+#    auto-hosted by the docs "web apps" pipeline and usable offline over file://.
+#  - include/telemetry_service.hpp: espp::Telemetry, a binary telemetry emitter
+#    (dispatcher module 3 over stream_frame) that the web app plots over WebUSB.
+# The header only depends on base_component + stream_frame; an app pairs it with
+# a Dispatcher and a transport (e.g. usb_device) — see the example.
+idf_component_register(
+  INCLUDE_DIRS "include"
+  REQUIRES base_component stream_frame
+)

components/serial_plotter/README.md

@@ -1,18 +1,19 @@
 # Serial Plotter
 
-A self-contained browser tool for reading columnar serial data and plotting it
-efficiently — modeled on [esp-cpp/uart_serial_plotter](https://github.com/esp-cpp/uart_serial_plotter),
-but running entirely in a Chromium-based browser over the Web Serial API. No
-install, no CDN, no network access.
+A self-contained browser tool for reading data and plotting it efficiently —
+modeled on [esp-cpp/uart_serial_plotter](https://github.com/esp-cpp/uart_serial_plotter),
+but running entirely in a Chromium-based browser. No install, no CDN, no network
+access. Two transports feed the same plot:
+
+- **Web Serial (text / CSV)** — auto-parses columnar output (a header line plus
+  numeric rows) from any device that prints it.
+- **WebUSB (binary telemetry)** — an espp device streams typed float channels
+  via `espp::Telemetry` (see `include/telemetry_service.hpp` and the
+  [example](example/)) for higher rate and device-accurate timestamps.
 
 - **Hosted:** <https://esp-cpp.github.io/espp/apps/serial_plotter.html>
 - **Offline:** open `web/serial_plotter.html` directly via a `file://` URL.
 
-> This component currently ships the webapp only. A firmware-side binary
-> **telemetry** transport (a `stream_frame` / `dispatcher` module for
-> higher-bandwidth, typed channels) is a planned follow-up; the same webapp will
-> gain a WebUSB transport that feeds the same plot.
-
 ## Screenshots
 
 The demo data below is a Lorenz attractor (`time,x,y,z`) loaded via **Load CSV**.
@@ -60,12 +61,19 @@ rest, with the per-series filter bar:
   (or any matching CSV) to view it offline with no device connected.
 - **Serial controls.** Baud selector, pause / resume, clear, and a DTR/RTS device
   reset.
+- **Binary telemetry over WebUSB.** Connect with **USB** to an espp device
+  running `espp::Telemetry`: the app reads the channel schema and plots the
+  device-timestamped sample stream (decoded from the `stream_frame` framing,
+  dispatcher module 3) into the same plot. Requests the schema on connect and
+  can pause/resume the device stream.
 
 ## Requirements
 
-Web Serial is available only in Chromium-based browsers (Chrome, Edge, Opera) and
-needs a secure context — it works from `https`, `http://localhost`, or `file://`.
-In an unsupported browser the app still loads and can **Load CSV** for viewing.
+Web Serial and WebUSB are available only in Chromium-based browsers (Chrome,
+Edge, Opera) and need a secure context — they work from `https`,
+`http://localhost`, or `file://`. In an unsupported browser the app still loads
+and can **Load CSV** for viewing. Native USB telemetry needs an ESP32-S3 (also
+S2 / P4) device; see [`example/`](example/).
 
 ## Third-party
 

components/serial_plotter/example/CMakeLists.txt

@@ -0,0 +1,50 @@
+# The following lines of boilerplate have to be in your project's CMakeLists
+# in this exact order for cmake to work correctly
+cmake_minimum_required(VERSION 3.20)
+
+# Build the example (and the espp components it pulls in) as C++20 (the repo
+# standard; the telemetry service needs no C++23 feature). Must be set before
+# the project.cmake include / project() call so IDF picks it up.
+set(CMAKE_CXX_STANDARD 20)
+
+# NOTE: the IDF component manager is intentionally left ENABLED here (unlike most
+# espp examples) so that it can fetch the managed `espressif/esp_tinyusb`
+# dependency declared by the usb_device component's idf_component.yml. To avoid
+# the component manager scanning every espp component manifest (some board
+# components declare target-specific constraints that would fail on esp32s3),
+# EXTRA_COMPONENT_DIRS is narrowed to just the components this example uses; the
+# in-repo espp components there satisfy the `espp/*` dependencies locally.
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+
+# add only the component directories that we want to use
+set(EXTRA_COMPONENT_DIRS
+  "../../../components/base_component"
+  "../../../components/dispatcher"
+  "../../../components/format"
+  "../../../components/logger"
+  "../../../components/serial_plotter"
+  "../../../components/stream_frame"
+  "../../../components/task"
+  "../../../components/usb_device"
+)
+
+# With the component manager disabled (IDF_COMPONENT_MANAGER=0, e.g. in CI so the
+# build does not need the as-yet unpublished espp/* components in the registry),
+# esp_tinyusb/tinyusb are not fetched; add the vendored submodule copies under
+# external/ to the search path. esp_tinyusb's CMakeLists adds `tinyusb` to its
+# REQUIRES when the manager is off, so both directories must be discoverable.
+if(DEFINED ENV{IDF_COMPONENT_MANAGER} AND "$ENV{IDF_COMPONENT_MANAGER}" STREQUAL "0")
+  list(APPEND EXTRA_COMPONENT_DIRS
+    "../../../external/esp-usb/device/esp_tinyusb"
+    "../../../external/tinyusb"
+  )
+endif()
+
+set(
+  COMPONENTS
+  "main esptool_py base_component dispatcher format logger serial_plotter stream_frame task usb_device esp_tinyusb"
+  CACHE STRING
+  "List of components to include"
+  )
+
+project(telemetry_example)

components/serial_plotter/example/README.md

@@ -0,0 +1,46 @@
+# Serial Plotter — USB telemetry example
+
+Streams synthetic float channels from an ESP32-S3 to the browser **Serial
+Plotter** web app over USB, using `espp::Telemetry` (a binary telemetry emitter
+carried on the `stream_frame` framing, dispatcher module 3).
+
+The hosted app — <https://esp-cpp.github.io/espp/apps/serial_plotter.html> —
+connects on the **vendor (WebUSB)** interface, reads the channel **schema**, and
+plots the live **sample** stream. It is the binary, higher-rate,
+device-timestamped counterpart to the app's text/CSV Web Serial transport.
+
+## What it does
+
+- Declares four channels — `sine`, `cosine`, `noise`, `ramp` — as the schema.
+- A producer task emits one sample (a `float` per channel) every ~10 ms (100 Hz),
+  timestamped with the device clock.
+- Exposes the stream over the USB **vendor (WebUSB)** interface; a `Dispatcher`
+  routes module-3 frames to the emitter and serves capability discovery so the
+  browser **Device Hub** lists this device and links to `serial_plotter.html`.
+- The web app can pause/resume the stream and request a rate (`SET_STREAM`), and
+  requests the schema on connect (`GET_SCHEMA`).
+
+`espp::Telemetry` itself is transport-agnostic (the `stream_frame` framing works
+over CDC / UART / a socket too); this example streams over WebUSB because that is
+what the web app's binary path consumes.
+
+Swap the synthetic generator for your real signals: build a `std::array<float, N>`
+in channel order and call `telemetry.emit(...)`.
+
+## Build & run
+
+```sh
+idf.py -p /dev/ttyACM0 flash monitor   # target esp32s3 (set in sdkconfig.defaults)
+```
+
+Then open the Serial Plotter web app, click **Connect (USB)**, and pick the
+"espp Serial Plotter" device. The system console/logs go to the separate
+built-in USB-Serial-JTAG.
+
+## Notes
+
+- Native USB (vendor / WebUSB) needs an ESP32-S3 (also S2 / P4) — not the
+  classic ESP32. `sdkconfig.defaults` pins `esp32s3` and enables the TinyUSB
+  vendor class.
+- WebUSB / Web Serial are Chromium-only and need a secure context (`https`,
+  `http://localhost`, or `file://`).

components/serial_plotter/example/main/CMakeLists.txt

@@ -0,0 +1,5 @@
+idf_component_register(
+  SRC_DIRS "."
+  INCLUDE_DIRS "."
+  REQUIRES serial_plotter dispatcher stream_frame task usb_device esp_tinyusb
+)

components/serial_plotter/example/main/telemetry_example.cpp

@@ -0,0 +1,168 @@
+// USB telemetry -> Serial Plotter web app example.
+//
+// Streams a few synthetic float channels from an ESP32-S3 to the browser over
+// USB using the espp::Telemetry emitter (a small binary protocol carried on the
+// stream_frame framing, dispatcher module 3). The hosted `serial_plotter.html`
+// web app connects on the vendor (WebUSB) interface, reads the SCHEMA (channel
+// names), and plots the SAMPLE stream live — the binary, higher-rate,
+// device-timestamped counterpart to the app's text/CSV Web Serial transport.
+//
+// Replace the synthetic generator below with your real signals: build a
+// std::array<float, N> in channel (schema) order and call telemetry.emit(...).
+// The system console/logs go to the separate built-in USB-Serial-JTAG.
+
+#include <array>
+#include <chrono>
+#include <cmath>
+#include <condition_variable>
+#include <cstdint>
+#include <cstdlib>
+#include <deque>
+#include <mutex>
+#include <span>
+#include <thread>
+#include <vector>
+
+#include "dispatcher.hpp"
+#include "logger.hpp"
+#include "stream_frame.hpp"
+#include "task.hpp"
+#include "telemetry_service.hpp"
+#include "usb_device.hpp"
+
+using namespace std::chrono_literals;
+namespace sf = espp::stream_frame;
+
+extern "C" void app_main(void) {
+  espp::Logger logger({.tag = "Telemetry", .level = espp::Logger::Verbosity::INFO});
+  logger.info("Starting USB telemetry (Serial Plotter) example");
+
+  // --- the telemetry emitter: one float per named channel, in schema order ---
+  espp::Telemetry telemetry({
+      .channels = {"sine", "cosine", "noise", "ramp"},
+      .stream_on_start = true, // stream immediately; the web app can pause via SET_STREAM
+      .period_ms = 10,         // default 100 Hz (the web app may request another rate)
+      .log_level = espp::Logger::Verbosity::WARN,
+  });
+
+  // --- USB: the vendor (WebUSB) interface carries the telemetry protocol ------
+  // The browser Serial Plotter plots binary telemetry over WebUSB. (The
+  // Telemetry service itself is transport-agnostic - CDC / UART / a socket work
+  // too - but the web app's Web Serial path parses text/CSV, so this example
+  // streams the binary protocol over WebUSB only.) The console/logs go to the
+  // separate built-in USB-Serial-JTAG.
+  espp::UsbDevice::Config usb_cfg;
+  usb_cfg.manufacturer = "espp";
+  usb_cfg.product = "espp Serial Plotter";
+  usb_cfg.log_level = espp::Logger::Verbosity::WARN;
+  espp::UsbDevice::VendorFunction vendor;
+  vendor.interface_name = "espp Telemetry (WebUSB)";
+  vendor.webusb = true;
+  vendor.landing_page_url = "esp-cpp.github.io/espp/apps/serial_plotter.html";
+  usb_cfg.vendor = vendor;
+  espp::UsbDevice usb(usb_cfg);
+
+  // Every device->host write (samples, request replies, discovery replies) goes
+  // through this one tx_mutex-guarded helper so the sample-producer task and the
+  // RX worker never write the TinyUSB vendor FIFO at the same time.
+  std::mutex tx_mutex;
+  auto send = [&](std::span<const uint8_t> bytes) {
+    std::lock_guard<std::mutex> lock(tx_mutex);
+    if (!usb.write_vendor(bytes))
+      logger.warn_rate_limited("dropped a {}-byte frame (USB TX backpressure or disconnect)",
+                               bytes.size());
+  };
+  telemetry.set_send(send);
+
+  // --- dispatcher: route module-3 frames to the telemetry service + discovery -
+  espp::Dispatcher dispatcher;
+  const espp::Dispatcher::ModuleInfo info{.name = "Serial Plotter",
+                                          .app = "serial_plotter.html",
+                                          .description = "Live device telemetry plotting"};
+  dispatcher.register_module(
+      espp::Telemetry::kModule, [&](const sf::Frame &frame) { telemetry.handle(frame); }, info);
+  dispatcher.set_device_info(usb_cfg.product);
+  dispatcher.serve_discovery(send);
+
+  // --- USB RX plumbing: queue in the TinyUSB callback, dispatch from a worker -
+  std::mutex rx_mutex;
+  std::condition_variable rx_cv;
+  std::deque<std::vector<uint8_t>> rx_queue;
+  size_t rx_queued_bytes = 0;
+  bool rx_overflow = false;
+  static constexpr size_t kMaxQueuedRxBytes = 8 * sf::kMaxFrameSize;
+  usb.set_vendor_receive_callback([&](std::span<const uint8_t> data) {
+    {
+      std::lock_guard<std::mutex> lock(rx_mutex);
+      if (rx_queued_bytes + data.size() > kMaxQueuedRxBytes) {
+        rx_queue.clear();
+        rx_queued_bytes = 0;
+        rx_overflow = true;
+      } else {
+        rx_queue.emplace_back(data.begin(), data.end());
+        rx_queued_bytes += data.size();
+      }
+    }
+    rx_cv.notify_one();
+  });
+
+  std::error_code usb_ec;
+  if (!usb.initialize(usb_ec)) {
+    logger.error("Failed to initialize USB device: {} - no host transport available; aborting",
+                 usb_ec.message());
+    return;
+  }
+
+  espp::Task rx_task({.callback = [&](std::mutex &, std::condition_variable &) -> bool {
+                        std::deque<std::vector<uint8_t>> chunks;
+                        bool overflowed = false;
+                        {
+                          std::unique_lock<std::mutex> lock(rx_mutex);
+                          rx_cv.wait_for(lock, 100ms,
+                                         [&] { return !rx_queue.empty() || rx_overflow; });
+                          std::swap(chunks, rx_queue);
+                          rx_queued_bytes = 0;
+                          overflowed = rx_overflow;
+                          rx_overflow = false;
+                        }
+                        if (overflowed) {
+                          dispatcher.reset();
+                          return false;
+                        }
+                        for (const auto &chunk : chunks)
+                          dispatcher.feed(chunk);
+                        return false;
+                      },
+                      .task_config = {.name = "telemetry_rx", .stack_size_bytes = 8192}});
+  rx_task.start();
+
+  // --- synthetic signal producer: emit one SAMPLE per period ------------------
+  // Replace this with your real signals. Values are in schema (channel) order.
+  const auto t0 = std::chrono::steady_clock::now();
+  espp::Task gen_task(
+      {.callback = [&](std::mutex &m, std::condition_variable &cv) -> bool {
+         const float t =
+             std::chrono::duration<float>(std::chrono::steady_clock::now() - t0).count();
+         const float two_pi = 6.2831853f;
+         const float noise = static_cast<float>(std::rand()) / RAND_MAX * 0.4f - 0.2f;
+         const std::array<float, 4> values = {
+             std::sin(two_pi * 1.0f * t), // sine @ 1 Hz
+             std::cos(two_pi * 0.5f * t), // cosine @ 0.5 Hz
+             noise,                       // uniform noise
+             std::fmod(t, 2.0f) - 1.0f,   // -1..1 sawtooth ramp
+         };
+         telemetry.emit(values); // no-op while streaming is paused
+         std::unique_lock<std::mutex> lock(m);
+         cv.wait_for(lock, std::chrono::milliseconds(std::max<uint16_t>(1, telemetry.period_ms())));
+         return false; // keep running
+       },
+       .task_config = {.name = "telemetry_gen", .stack_size_bytes = 4096}});
+  gen_task.start();
+
+  logger.info("Telemetry ready. Open the Serial Plotter web app and connect over WebUSB "
+              "(channels: sine, cosine, noise, ramp).");
+
+  while (true) {
+    std::this_thread::sleep_for(1s);
+  }
+}

components/serial_plotter/example/sdkconfig.defaults

@@ -0,0 +1,31 @@
+# This example uses the native USB-OTG peripheral (vendor / WebUSB), which is
+# only available on the ESP32-S3 (also S2 / P4) -- NOT the classic ESP32. Pin the
+# target here so a bare `idf.py build` does not fall back to esp32.
+CONFIG_IDF_TARGET="esp32s3"
+
+# Common ESP-related
+CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
+
+# Enable the TinyUSB vendor-specific class (THE key enablement for the vendor /
+# WebUSB interface). esp_tinyusb gates CFG_TUD_VENDOR behind
+# CONFIG_TINYUSB_VENDOR_COUNT; setting it > 0 compiles in the vendor class
+# driver so espp::UsbDevice's vendor function (bInterfaceClass 0xFF + WebUSB)
+# works. The telemetry web app plots over this WebUSB interface.
+CONFIG_TINYUSB_VENDOR_COUNT=1
+
+# CDC + HID are enabled only to keep the usb_device component compiling (it
+# includes the TinyUSB CDC/HID headers unconditionally). This example creates
+# NO CDC or HID interface -- it exposes the vendor (WebUSB) interface only, so
+# the device enumerates with just that one telemetry interface.
+CONFIG_TINYUSB_CDC_ENABLED=y
+CONFIG_TINYUSB_CDC_COUNT=1
+CONFIG_TINYUSB_HID_COUNT=1
+
+# Telemetry frames are small (a SAMPLE is a 4-byte timestamp + 4 bytes/channel),
+# but give the vendor TX FIFO headroom so a burst of batched samples is never
+# truncated by TinyUSB backpressure. The vendor RX FIFO only carries short
+# requests (GET_SCHEMA / SET_STREAM), so 512 bytes is plenty there.
+CONFIG_TINYUSB_VENDOR_RX_BUFSIZE=512
+CONFIG_TINYUSB_VENDOR_TX_BUFSIZE=4096
+
+CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y