Repository navigation
Conversation
Three ESPHome profiles (microphone, camera, camera and microphone) on generic ESP32 DevKit and ESP32-CAM boards, configured from VisionState and installed through ESPHome, found and paired through Home Assistant; a mute switch that can be wired in; sending only when loud left for later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
esphome/: the component visionstate_stream (the microphone as an endless 16 kHz WAV and the camera as a fresh snapshot, over HTTP behind a user name and password) and ESPHome packages for a generic ESP32 DevKit with an INMP441 and an ESP32-CAM (camera; camera and microphone), with a Microphone switch, the gain, an optional mute pin, a light while someone listens, and the flash as a light. CI compiles all three for the oldest and the newest ESPHome. The ESP32 page in the app (linked from the sound and image source): pick the board and what it is, see the wiring, and get the device's whole ESPHome file with the Wi-Fi and password filled in in the browser; then check the device answers and make the sensor from it. docs/ESP32_PLAN.md rewritten: what is decided, built, proposed and not checked yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (no copy), the guide on the esp32 branch Copying is not allowed on a page opened over plain http://, as most Home Assistants are; the file is downloaded and imported, and shown folded under What is in it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t be added and shows its key Step 4 looks for the device every few seconds: found but not added (Discovered) it says so, links to Devices & services and shows the encryption key Home Assistant asks for; added, its address is filled in. The guide's file fetches the firmware on every build while it is a branch, and the snapshot logs each step (a picture asked for never came on the first board). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for and a listener comes or goes The camera hands pictures out in the main loop, which sleeps while nothing happens, and it is woken only for a picture taken while one is waited for. With two buffers one is ready already, so the fresh snapshot's second picture never came on a device nobody listened to (it did while a listener's sound kept the loop busy). Found on the first ESP32-CAM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed for One wake was answered too soon (the camera gives at most one picture per 1/max_framerate) and the loop slept again: the fresh snapshot still failed on a device nobody listened to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The camera's XCLK runs on LEDC channel 0 and timer 0; the flash was given channel 0 and set it to 1 kHz, so the camera took no new pictures: a snapshot got the one it had ready, never a fresh one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The camera clears its requests after handing a picture out, and the snapshot runs on the other core: the fresh picture's request, made right after the first, was wiped out before the camera saw it. It only showed with nobody listening (a listener keeps both cores busy, so the request came later). Device log on the first ESP32-CAM: the first picture after 5 ms, the second never. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, its flash offered From the guide the wizard loads the device's picture right away, says nothing about a camera's low-resolution stream (an ESP32-CAM has one size), names the sensors "<device> sound" and "<device> camera" (the device's own name gave switch.x_enabled_2 next to the device's entities), and offers an ESP32-CAM's flash as the light, never switched on unasked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cs brought up to date A stream of nothing but zeros (an ESP32 without its microphone, or muted; a camera with its audio off) looked like a quiet room: the sensor said "ok" and was deaf. After SOUND["dead_silence_s"] it now says "Pure silence" on the dashboard and the Live tab, and logs it once. Docs: esphome/README.md describes the guide's download and import and what to do when something is wrong; ARCHITECTURE.md maps esphome/ and the guide's files; RELEASING.md says how the firmware is released; ESP32_PLAN.md ticks what was tried on the boards; SOUND_PLAN.md and DOCS.md follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ection The server has a socket per listener and few to spare. A client that keeps its connection open after an answer (HTTP keep-alive: a Google Cast speaker after its 401, VisionState's and Home Assistant's snapshots) held the spare one, and then no new connection was taken at all, while the streams already open went on: found on both boards after a media player tried the stream. Now a snapshot, 401, 404, 405, 409 and 500 say "Connection: close" and the session is closed after the answer, and there are two sockets to spare instead of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its camera and PDM microphone are on the board, so nothing is soldered (a microphone on the ESP32-CAM's pins leaves no room for its USB board). New packages for the board, its camera and a PDM microphone; the Microphone switch and gain move to microphone-controls.yaml, which both microphone packages include, so existing device files keep their package lists. The guide offers the board (nothing to wire), CI compiles it. Compiled, not yet tried on a board. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An add-on package for each, added to the device's own voice assistant configuration: the stream on the microphone the wake word uses (Voice PE: XMOS channel 1, noise suppressed but without automatic gain), one listener, and silence while the assistant hears a command and answers (visionstate_stream gets set_paused; a switch turns the pause off). Valid next to Voice PE 26.9.0 and M5Stack's configuration; not built in CI or tried on a device yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compiled next to both devices' own firmware; the Atom's test device gets a shorter name, since a long one takes its build paths over Windows' 260 characters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
This creates a downloadable yaml file to flash to certain ESP32 boards to use with Visionstate. It sets it up, so its shares IP address in home assisatnt, so Visionstate can connect directly easeily instead of trough home assistant, this way even if IP changes, it get it updated trough home assistant. Also I want to have it working on voice assistant devices, which have been compiled but is also untested. Feedback, and help is wanted. |
…t, not 100 The first request after a picture is handed out is lost, so the fresh one came on the second ask, WAKE_EVERY_MS (50) later at least: every fresh snapshot waited 100 ms on an ESP32-CAM whose camera had it long before. Asked every 10 ms now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0 ms Measured on the maintainer's ESP32-CAM: the fresh picture came 102 ms after the one kept ready, every time, whatever the request interval. ESPHome's camera gives a picture no sooner than 1 / max_framerate after the last, and its default is 10 fps. The comment on WAKE_EVERY_MS (previous commit) blamed the request interval; it was the frame rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at 25 fps too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A choice in Home Assistant, kept over restarts, that sets the picture size, its JPEG quality and the exposure on the camera: as before for still things; every pixel (1600x1200) and less compression for meters and displays; a short fixed exposure with automatic gain for people, cars and animals (the exposure a number too). The camera starts at its largest size so each choice fits its buffers. Compiled for the ESP32-CAM and the XIAO; tried on the ESP32-CAM next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For a camera, the guide asks whether it looks at still things, a meter or display, or things that move, and writes camera_use into the device's file (settings.ESPHOME["camera_uses"], kept equal to the device's options by a test). Docs: the table of what each choice does, why it is per device and not per picture, and what was measured on the ESP32-CAM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore its pictures The firmware takes /snapshot.jpg?use=states|meter|movement: it sets the camera up so first (the pictures before it settled thrown away), keeps it so, and says how it is set up in an X-VisionState-Camera header; Camera use in Home Assistant follows. VisionState knows a camera of ours by that header and asks for what its sensors need: by kind (a reading a meter, objects things that move, states still things) or the sensor's own choice (triggers.camera_use), the most demanding of the sensors on one camera, so it is not switched back and forth. The choice is in a sensor's settings only for a camera of ours. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cameras of ours and what their sensors want now live in sources.CameraUses on the runtime (the grabber reads it), so a stand-in grabber in tests or elsewhere no longer breaks the checks. The log says when a camera answers as ours and when it is asked to switch, its address redacted. The fake camera has an ESP32 of ours (/esp32/snapshot.jpg), and an e2e test chooses the set-up in a sensor's settings and sees the camera follow, and no such choice for any other camera. Docs: VisionState sets the camera up itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hi. Unfortunately, I can't find the Seeed Studio ESP32-S3 Cam, so I had an AI generate a list of boards that I can test. Available ESP32 Development Boards for TestingI have several ESP32 development boards available for testing VisionState, including camera integration, microphone input, voice assistant compatibility, audio streaming, and two-way intercom functionality. 1. Waveshare ESP32-S3 Audio BoardAvailable hardware:
Current status: Previously configured with a working camera, display, and intercom. 2. Elecrow CrowPanel Advance ESP32-S3Potential tests:
Current status: Home Assistant Voice Assistant firmware installed. Potential tests:
Current status: Available for testing; microphone integration required. 4. Waveshare 3.5-inch ESP32 Display BoardAvailable hardware:
Potential tests:
Current status: Home Assistant Voice Assistant firmware installed. 5. ESP32-P4 Camera / Video Doorbell PlatformAvailable hardware:
Potential tests:
Current status: Available as a potential intercom and video doorbell testing platform. Proposed Testing Priorities
Testing ApproachI can test configurations on physical hardware and provide compilation results, runtime logs, and feedback. The preferred approach is to reuse existing working firmware configurations and avoid unnecessary reflashing or hardware modifications. Note: The proposed VisionState integrations have not yet been validated. Hardware capabilities depend on the exact board revisions and installed firmware. |
|
I’m also sending a link to a working ESP32-S3 LCD 2 setup, which includes a microphone added specifically for VisionState. I’ve also managed to set up an automation where, whenever the speech sensor detects speech, it begins recording into the VoiceBox. If you’re interested in utilizing these speech recordings, I’d be happy to help. |
…icrophone) For #75: a user had wired an INMP441 to a Waveshare ESP32-S3-Touch-LCD-2 with firmware of their own; our firmware now builds for it with their wiring (SCK/WS on GPIO43/44, the log on USB; SD on GPIO12, or GPIO18 with a camera), and for a generic ESP32-S3 DevKit (RGB LED). The light that shows someone listens moves from common.yaml into each board package, so a board without a plain LED gives its own (the S3 DevKit's RGB LED, nothing on the Waveshare). The guide offers both boards; a board with no mute pin takes no mute switch. Compiled, not tried on a board. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Thanks a lot, both for the list and for your own setup!
|
|
"Your repo: only the README and LICENSE are in it so far. microphone-http.yaml, components/ and profiles/ seem to be missing, could you push them"-You can download them in the v1 releases.
|
… its light For #75: the same user wired an INMP441 to a Waveshare ESP32-C5-WIFI6-KIT (GPIO8, 9, 10); our firmware now builds for it, with its RGB LED (GPIO27) as the light and a mute switch on GPIO4 (not a strapping pin, not the battery's ADC on GPIO6). The guide offers it, and says the Wi-Fi band from the board: 2.4 or 5 GHz on the C5. From their own firmware we took one idea: the Waveshare ESP32-S3-Touch-LCD-2 has no LED, so its display is now the light, a red dot while someone listens and dark otherwise. Not taken: a token in the address, a fixed WAV length, level sensors every second (docs/ESP32_PLAN.md says why). rgb_order is deprecated in ESPHome: channel_colors on the S3 DevKit too. Compiled with ESPHome 2026.10.2 (C5, LCD-2, S3 DevKit), not tried on a board. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kept backups hold history_days 9 and history from the day they were made; once that is older, the clean-up after a restore removes it and the test fails by date (schema15 now). The test is about the format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kept open while the ESP32 work is built and tried on real boards, so the app's CI runs on it. Not to be merged until the maintainer says so.
esphome/: thevisionstate_streamcomponent (sound as an endless WAV, a fresh snapshot, user name and password) and packages for an ESP32 DevKit + INMP441 and an ESP32-CAM; its own CI (.github/workflows/esphome.yml).docs/ESP32_PLAN.md.🤖 Generated with Claude Code