Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0c380d0
feat: hardware-in-the-loop runtime, target CLI and browser adapters
carlvon17 Oct 23, 2026
e8b8185
feat: control CLI for the generic Control API v1
carlvon17 Oct 23, 2026
28e72d4
feat: single-source the Control v1 wire contract
carlvon17 Oct 24, 2026
3e8e5c1
feat: remote Control middleware over HTTPS with enrolled identity pins
carlvon17 Oct 24, 2026
0ac18ab
feat: web HIL device chain with outbound gateway relay (H1)
carlvon17 Oct 24, 2026
540e2f8
feat: ship an agent handoff prompt with control connection output
carlvon17 Oct 24, 2026
e40f49f
fix: anchor agent_prompt to the CLI-side project path
carlvon17 Oct 24, 2026
2c2b7a5
feat(hil): gateway text-frame authentication for browser coordinators
carlvon17 Oct 24, 2026
3d92dbc
control: move agent prompt to shared prompts module and add unpaired …
carlvon17 Oct 24, 2026
ee400bc
docs: drop the unimplemented agent-prompt extension endpoint
carlvon17 Oct 24, 2026
bd8d09a
control: drop the CLI-side agent_prompt field
carlvon17 Oct 24, 2026
17b1e7a
docs: add AI-assistant HIL deployment runbook for the device-gateway …
carlvon17 Oct 25, 2026
d0a274a
docs: rerun scenario B over a loopback gateway and ssh -L tunnel
carlvon17 Oct 25, 2026
0198e45
docs: make offline install the canonical board setup path
carlvon17 Oct 25, 2026
d9b3bc1
fix: keep the agent dispatch loop alive across idle read timeouts
carlvon17 Oct 25, 2026
7173550
feat: accept {en, zh} tables for parameter and key mapping labels
cmjang Oct 28, 2026
ada4a2c
sdk: mirror BotLab config-split SDK (read_config + commands parameters)
carlvon17 Oct 29, 2026
4b3b944
wip: 归拢本地未提交工作(control serve/descriptor、打包面、CI、docs)
carlvon17 Oct 29, 2026
041b273
Merge local WIP: control serve/descriptor, HIL, packaging and docs
cmjang Nov 4, 2026
8a29c16
release: 0.4.0 — mirror BotLab config-split SDK with localized labels
cmjang Nov 4, 2026
96535e0
Merge #11: accept {en, zh} tables for parameter and key mapping labels
cmjang Nov 4, 2026
da6880e
test: give the TLS fixture CA keyUsage so Python 3.13 strict verifica…
cmjang Nov 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
python-version: ["3.11", "3.12", "3.13"]
env:
# Fresh clones have no src/botlab/ui display frontend (gitignored,
# regenerated with `botlab ui sync`); tests must pass without it. The
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,7 @@ src/botlab/ui/

# Local verification artifacts (screenshots, logs, scratch scripts)
tmp/

# Native library logs and stray session artifacts
MUJOCO_LOG.TXT
:memory:.ses
104 changes: 94 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
BotLab's Python SDK and command-line interface. Declare robot projects with
`project()`, `Robot`, `Model`, `Sensor`, `Motion`, and `Scene` in
`controller.py`, express control logic as composable signals, and run them
natively on your machine with one command — real MuJoCo physics and ONNX
Runtime inference in local CPython, with the browser as a read-only viewer.
in the browser with one command — MuJoCo WASM physics and ONNX Runtime Web
inference. Native CPython execution is available with `--native`.

```bash
pip install "botlab-python[runtime]"
pip install botlab-python
cd my-robot-project
botlab run
```
Expand All @@ -25,8 +25,8 @@ Watch, and Metrics panels, and the browser itself executes the project
3. Assemble an integrity-checked release in memory and serve it to the
browser without writing an archive to disk.
4. Start a local static server and open the browser. The Vite-style startup
banner prints clickable `Local` and `Network` links; the server binds to
`0.0.0.0` by default so devices on the same network can connect.
banner prints a clickable `Local` link; browser runs bind to `127.0.0.1`
by default. Use `--host 0.0.0.0` for ordinary LAN access (HIL is disabled).

The editor opens at `/`. Its localized `index.html` includes the
`window.BOTLAB_LOCAL_PROJECT` flag, so it loads the current directory from
Expand All @@ -52,12 +52,42 @@ The CLI is project-agnostic: it works with any BotLab project that follows the
two-file `main.py` + `controller.py` contract and contains no robot- or
policy-specific behavior.

## Hardware-in-the-loop

Run `botlab run`, open **Target manager**, and add a Linux target over SSH.
BotLab can inspect a clean system, help configure a supported network interface,
and install its Agent plus matching Linux inference dependencies. The board needs
a booted Linux system, Python 3.10+ and reachable SSH; it does not need BotLab,
robot-specific programs, pip or Internet access preinstalled. New SSH host keys
require explicit verification; passwords are transient.

Once prepared, choose the target and **Run on Target**. BotLab builds, deploys and
loads the current project's ONNX models before running them on the board, while
browser MuJoCo and the controller graph remain on the host. The separate run menu
provides Build → Deploy → Load project → Run: deployment uses the last successful
build, Load project directly connects that exact version after it is deployed to
the connected target, and Run starts only that connected version with matching
local source and models. Choose other projects or versions explicitly in Target
manager's Deployed projects tab. Stop and Unload remain separate actions. The
target manager keeps multiple projects and versions under `~/botlab/projects/`; one session executes
at a time. The CLI also supports `botlab target discover/add/inspect/provision`,
`botlab deploy --target <name>`, and `botlab run --hil <name>`.

This unreleased implementation uses ONNX Runtime CPU on compatible Linux
ARM64/x86_64 targets over verified SSH forwarding. It is model-service HIL, not
full-controller deployment or hard real-time scheduling. RDK X5 is an example;
BPU/MCU support and per-project dependency environments are not implemented.
Install the matching SDK and browser runtime. See the [HIL guide](docs/hil.md)
for first connection, CLI commands, deployment layout and current limits.

## Project structure

```text
my-robot-project/
├── main.py # Fixed four-line entry point
├── controller.py # Project declarations and control expressions
├── config/ # Optional: declaration data split into yaml files
│ └── runtime.yaml # referenced canonically via read_config("config/runtime.yaml")
├── robot.preset.yaml # Joint order, PD gains, initial state, and calibration
└── assets/ # Project-relative MJCF, ONNX, meshes, and other assets
```
Expand Down Expand Up @@ -99,6 +129,24 @@ See the BotLab repository's `docs/api/` directory for the complete API
contract, including signal expressions, `policy_bank`, state machines, and
resource preset fields.

Declaration data (runtime settings, parameters, IO bindings, sensor configs,
control profiles) may live in `config/*.yaml` files referenced from
`controller.py` through `read_config("config/<name>.yaml")` — the file is the
declaration data, the source is structure and wiring. The CLI collects them
recursively into the release `files` (same path rules as the browser editor,
so `botlab build` and a browser publish produce identical project digests):

```python
from botlab import Model, PolicyController, Robot, concat, project, read_config

runtime = read_config("config/runtime.yaml")
project(
runtime["name"],
control_hz=runtime["control_hz"],
physics_hz=runtime["physics_hz"],
)
```

## Commands

| Command | Description |
Expand Down Expand Up @@ -241,7 +289,7 @@ installation and make no runtime download through the CLI.
For offline machines, import the exact matching archive first:

```bash
botlab ui fetch --archive botlab-browser-runtime-0.3.0.zip
botlab ui fetch --archive botlab-browser-runtime-0.4.0.zip
```

`botlab ui fetch --url <url>` allows a mirror. The default cache is
Expand All @@ -259,7 +307,7 @@ npm run build
npm run build:viewer
# In botlab-python:
botlab ui sync --botlab ../BotLab
botlab ui export --botlab ../BotLab --out tmp/botlab-browser-runtime-0.3.0.zip
botlab ui export --botlab ../BotLab --out tmp/botlab-browser-runtime-0.4.0.zip
uv build --wheel --out-dir tmp/core-wheel
```

Expand Down Expand Up @@ -338,9 +386,9 @@ its real synchronized display frontend. Verify the separate browser runtime
archive used for website assembly too. From this checkout, with `uv` on PATH:

```bash
python scripts/verify_wheel.py dist/pypi-0.3.0/botlab_python-0.3.0-py3-none-any.whl \
--browser-runtime tmp/botlab-browser-runtime-0.3.0.zip \
--out tmp/wheel-acceptance-0.3.0
python scripts/verify_wheel.py dist/pypi-0.4.0/botlab_python-0.4.0-py3-none-any.whl \
--browser-runtime tmp/botlab-browser-runtime-0.4.0.zip \
--out tmp/wheel-acceptance-0.4.0
```

Use a new output directory for each run. The command installs that wheel with
Expand Down Expand Up @@ -372,3 +420,39 @@ only; this verification command does not upload an sdist or any other artifact.
## License

Proprietary — same as the main BotLab repository.

## Browser adapters for hardware-in-the-loop projects

Projects that provide a browser controller and a hardware bridge can use the
standard launcher with an explicit project adapter:

```bash
botlab run /path/to/project --browser-adapter hardware.web:run \
--inference-endpoint 192.168.128.10:50051
```

The selected callable receives `botlab.browser.BrowserRunContext` and returns an
exit code. The context contains `project_dir`, `ui_dir`, `host`, `port`,
`open_browser`, and optional `inference_endpoint`. The adapter runs with the
project on the Python import path and as the working directory. Selecting an
adapter executes its Python code; it is an explicit alternative to the usual
controller compilation/release path. It cannot be combined with native,
headless, release export, `--no-server`, or native report options.

The SDK supplies its versioned, integrity-checked browser runtime in `ui_dir`;
the project serves its own controller, assets, and inference transport. Adapter
runs default to loopback and port 0 (choose an available port); adapters must use
the actual bound port in URLs. Physics still runs in the browser. The SDK has no
robot models, board addresses, or robot-specific protocol implementation.

Browser-only fixes belong in BotLab's frontend source. Rebuild and distribute
the browser runtime archive alongside the compatible SDK release; Python SDK
source does not duplicate TypeScript fixes. For local/offline verification:

```bash
botlab ui export --botlab /path/to/fixed/BotLab --out /tmp/runtime.zip
botlab ui fetch --archive /tmp/runtime.zip
```

These commands use a local build; they do not publish a release. Existing users
must install the corrected runtime as well as any required CLI changes.
103 changes: 103 additions & 0 deletions docs/control-assistant.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# BotLab Control CLI 部署 runbook(AI 助手专用)

本文件是 BotLab 网页版「CLI 控制」面板提示词的**目标文档**:面板提示词只需一句话指向本文件。
读者是用户委托的 AI 助手。按本文件执行即可把「CLI → 通用中间件 → 浏览器页面」链路跑通。

**目标**:让用户正在使用的 BotLab 网页(编辑器或托管页)连上本地通用中间件,
此后通过 `botlab control` 命令驱动页面里的机器人仿真。

## 0. 链路与前提

- 用户已有浏览器页面,浏览器侧**无需安装任何东西**;本地要安装的只有
`botlab-python`(提供 `botlab` CLI 与通用中间件)。
- 中间件是**通用的**:`botlab control serve` 一条命令启动,不含任何项目代码。
项目的能力声明(capabilities)、动作表与控制经验由**页面在配对时上报**,
中间件按描述符构建——因此**不需要检出被控制项目的源码**。
- 中间件是 WS 服务端,浏览器作为客户端拨入,因此必须跑在用户本地(或用户可达的机器上)。
- 动手前向用户确认一件事:从哪个地址打开的 BotLab 页面(决定第 3 步的 `--origin`)。
- 项目自带中间件(如 microduck 的 `python -m microduck_control serve`)仍受支持:
按项目自己的说明启动即可,后续步骤不变;本文件默认走通用中间件。

## 1. 组件检查(只读)

- `which botlab`;若存在,看脚本 shebang 定位所属 Python 环境,用对应方式查版本
(`python3 -m pip show botlab-python`)。
- 版本号不能区分是否含 `control`;以 `botlab control --help` 是否存在为准。

## 2. 安装 / 更新

- 源:`git+https://github.com/OpenLegged/botlab-python.git@feat/control-protocol-core`
- ⚠️ **git 源构建必须前缀 `BOTLAB_ALLOW_MISSING_UI=1`**:UI 快照(`src/botlab/ui`)是
生成产物不进 git,否则构建钩子拒绝出包。CLI 控制用不到 UI,此为官方支持用法。
- 本地 checkout 若含 `src/botlab/ui`,可直接从该目录安装(同一提交、免开关、保留 UI,严格更优)。
- 安装带 `serve` extra(中间件需要 aiohttp):`pip install "botlab-python[serve]"`;
控制客户端本身标准库 only。缺 extra 时 `botlab control serve` 报
`missing_dependency`(exit 2),按提示补装即可。
- 系统 pip 受限先建 venv;Windows 用 `py -3 -m pip`。

## 3. 启动中间件并配对

```bash
botlab control serve --port 8776 --origin <本页 origin>

# 取配对信息(websocket 地址 + 令牌):
botlab control connection --show-token --json
```

- 工作目录决定中间件的项目锚点(discovery 的 `project_key`);与页面项目无关,
在哪个目录启动,之后就在哪个目录(或加 `--project` 指向它)调用其余命令。
- 中间件默认只监听 127.0.0.1 且只接受 allowlist 的 origin。**网页版页面的 origin 不是
127.0.0.1**,必须加 `--origin <页面地址的 scheme://host[:port]>`(如
`--origin https://botlab.example.com`),否则浏览器的 WS 连接被拒。
- 每个中间件只接受一个浏览器运行实例;不要对同一浏览器同时启动两套服务。
- 把 `connection --show-token` 输出的 websocket 地址与令牌**在对话中**交给用户,
提醒其粘贴到浏览器「CLI 控制」面板并点「连接并启用外部控制」。
连接只启用外部指令,不会让机器人移动。

## 4. 验证

```bash
botlab control state --json
```

`connected` 与 `external_enabled` 均为 `true`、`run_state` 为 `running`、`ready` 为
`true` 时链路就绪。`ready=false` 常见于页面隐藏或仿真暂停——保留配对,恢复后即可发令。

## 5. 控制规则(接管后)

1. 控制前先拉项目经验与能力表(均由页面上报):
`botlab control experience --json` 与 `botlab control capabilities --json`;
经验记录实测参数(如触发行走的最低速度)与坑,capabilities 声明不含这些。
2. 每个动作:读 `state` → 发命令 → `wait --id` 直到 `terminal:true` → 核对物理反馈,
再发下一条。
3. 工具输出解析失败不等于命令未执行:先查 `state` 的 `active_command_id`,
确认未接受后才可重发。
4. 参数按 `capabilities` 声明的范围发送;动作别名来自页面声明,通用 CLI 不内置;
定时移动不是精确导航,实际效果以位置与姿态反馈为准。
5. 网络超时只表示结果未知:先 `wait` 原请求,必要时对同一 request-id 显式 `retry`;
新的运动决定用新 ID。
6. 退出码:`0` 为请求成功(含 `pending`)、`2` 为请求/连接错误、`3` 为任务失败/取消/过期。

## 6. 安全规则

令牌只在用户对话中转述,绝不写入文件、日志或提交;令牌经 `BOTLAB_CONTROL_TOKEN`
或隐藏输入传入时不入 argv、不落 shell 历史。请求收据目录(默认
`~/.local/state/botlab/control/receipts/`)保持 0700/0600,不复制进项目日志,也不要
删除收据来绕过冲突。

## 7. 出错纪律

贴完整报错、分析根因后再行动,不盲目换端口或重装。已知坑速查:

- 浏览器面板连不上、中间件日志显示 origin 拒绝 → 缺 `--origin`,按第 3 步补页面 origin;
- `state` 显示 `connected:false` → 用户尚未在面板完成配对,重新走第 3 步交付流程;
- `capabilities` / `experience` / 发令返回 `no_project_paired` → 页面还没配对上报描述符,
或页面过旧不支持下发;确认用户用的是新版页面并重新连接;
- `serve` 报 `missing_dependency` → 缺 serve extra,按第 2 步补装 aiohttp;
- 中间件重启后旧命令全部被拒 → 实例身份(instance_id)已更换,属预期;重新配对即可,
不要重放旧 session / episode / request ID;
- CLI 安装报 `src/botlab/ui is missing` → git 源缺 UI 快照,安装命令加前缀
`BOTLAB_ALLOW_MISSING_UI=1`;
- 端口相关报错 → 先 `lsof -nP -iTCP:8776` 查占用再决定,不要直接换端口。

协议与托管背景见 [control-protocol.md](control-protocol.md) 与 [control-hosting.md](control-hosting.md)。
Loading
Loading