Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions apps/api/src/opendirector_api/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from opendirector.agent.citations import referenced_sources, with_trace_sources
from opendirector.agent.runtime import AgentRuntimes, runtime_available, configured_runtimes, runtime_spec, runtime_catalog, runtime_status
from opendirector.agent.skills import skills
from opendirector.agent.tools import model_parameter_choices
from opendirector.domain.projects import ProjectError
from opendirector.application.model_availability import requests_for
from opendirector.domain.rule_facts import MAX_REFERENCES, PROMPT_MAX_LENGTH
Expand Down Expand Up @@ -446,12 +447,21 @@ async def turn(project_id: UUID, value: Turn, x_opendirector_session: UUID = Hea
if value.runtime == 'jiuwenswarm' and value.thinking != 'auto':
raise ProjectError('JiuwenSwarm 当前使用原生默认推理强度。', 409)
config = await asyncio.to_thread(configs.resolve, str(value.modelId), 'agent') if spec.modelSource == 'settings' else None
if config is not None:
prepared = await asyncio.to_thread(generation.availability.prepare, config)
current = await asyncio.to_thread(configs.resolve, str(value.modelId), 'agent')
generation.availability.require(current, _assistant_request(current), prepared)
config = current
available = await skills(projects.local.repository, directory) if spec.supportsSkills else []
pending_skills = (asyncio.create_task(skills(projects.local.repository, directory))
if spec.supportsSkills else None)
try:
# Probe the model alongside skill discovery, while checking model errors first.
if config is not None:
prepared = await asyncio.to_thread(generation.availability.prepare, config)
current = await asyncio.to_thread(configs.resolve, str(value.modelId), 'agent')
generation.availability.require(current, _assistant_request(current), prepared)
config = current
available = await pending_skills if pending_skills is not None else []
finally:
if pending_skills is not None:
with CancelScope(shield=True):
pending_skills.cancel()
await asyncio.gather(pending_skills, return_exceptions=True)
if set(value.skills) - {item['name'] for item in available}:
raise ProjectError('技能已失效,请重新选择。', 409)
snapshot = await asyncio.to_thread(projects.snapshot, project_id)
Expand All @@ -462,9 +472,14 @@ async def turn(project_id: UUID, value: Turn, x_opendirector_session: UUID = Hea
model = config['modelId'] if config else value.runtimeModel
model_is_cli_default = spec.modelSource == 'cli' and not model
model_selections = dict(value.modelSelections)
model_names = {row['id']: row['modelId'] for row in configs.list()}
preferences = {kind: {'modelConfigId': model_id, 'modelName': model_names.get(model_id)}
model_rows = {row['id']: row for row in configs.list()}
preferences = {kind: {'modelConfigId': model_id, 'modelName': model_rows.get(model_id, {}).get('modelId')}
for kind, model_id in model_selections.items() if model_id}
if value.runtime == 'default':
for preference in preferences.values():
row = model_rows.get(preference['modelConfigId'])
if row and (choices := model_parameter_choices(row)):
preference['supportedParameterValues'] = choices
events = RunEvents()
def validate_connection():
if config is not None:
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ This directory is the entry point for the repository's shared documentation. The
| [Public website](../website/README.md) | Local preview, installer links, and GitHub Pages deployment | Website deployment or installer publishing changes |
| [Release tags and daily cuts](development/release.md) | Public SemVer tags and the daily patch cut | The tag scheme, daily cut gate, or version-field set changes |
| [UI performance design](development/ui-performance.md) | Interaction design principles, latency budgets, and measurement | A UI performance principle, target, or measurement method changes |
| [Agent image latency benchmark](development/agent-image-ordinary-benchmark.md) | Live browser workload and metrics for single-image requests | The workload, timing boundaries, or result checks change |
| [macOS installation validation](development/macos-install-validation.md) | Installation and usage scenarios verified on Apple Silicon | The corresponding acceptance checks are repeated |
| [Bilingual interface](development/i18n.md) | Language selection, message organization, and checks | Language policy, message structure, or checks change |
| [Python installation and packaging](development/packaging.md) | Installation, platform wheel builds, and isolated verification | Packaging, resource paths, or build process changes |
Expand Down
6 changes: 5 additions & 1 deletion docs/architecture/agent-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ English summary: a Job created by an Agent `generation.submit` records `agentRun

| Harness | 执行约束 |
| --- | --- |
| OpenCode | 工具权限配置开放 Shell、文件读取、已发现技能和业务 CLI |
| OpenCode | 工具权限配置开放 Shell、文件读取、已发现技能、结构化业务工具和业务 CLI |
| Codex | workspace-write 沙箱,开放 Shell 和业务 CLI |
| Claude Code | 明确允许 Skill、Bash、Read 和业务 CLI |
| JiuwenSwarm | 明确允许 skill_tool、list_skill、read_file、bash 和业务 CLI |
Expand All @@ -100,6 +100,10 @@ On Windows, Codex explicitly sets `windows.sandbox="elevated"` so ignoring user

CLI 通过环境变量中的本轮本地 Socket 地址连接业务网关,不使用连接令牌或凭据文件。网关继续校验工程范围、轮次生命周期及业务输入。

内置 OpenCode 通过每轮专用的回环 MCP 连接调用结构化业务工具。结构化工具和 CLI 共用工程范围校验、业务命令、回执和幂等规则。`read_project` 可一次读取工程快照、模型选择和指定业务命令的契约,也可按指定连接收窄参数选项。创建生成节点时,`create_generation_node` 用统一接口解析生成类型、模型和布局,并按请求创建节点或继续提交生成。工具在顶层声明各媒体类型的可用设置。发起生成时,创建事务先按共享规则校验模型、参数和引用。正式提交会按当前状态重新校验。若节点已创建但提交失败,工具返回原节点的回执,供该轮继续处理。内置 OpenCode 不导入宿主 Claude 提示文件。监听启动失败时,内置 OpenCode 仍可通过 CLI 执行业务操作。其他 Harness 继续使用各自的 CLI 接入方式。

Bundled OpenCode invokes structured project tools through a turn-scoped loopback MCP connection. The structured tools and CLI share project scoping, business commands, receipts, and idempotency. `read_project` can return the project snapshot, model choices, and selected command contracts in one call, with parameter choices scoped to the selected connection when requested. `create_generation_node` resolves the generation kind, model, and layout through one interface and can submit the generation on request. It exposes the applicable media settings as top-level tool arguments. When submission is requested, the creation transaction validates the proposed model, parameters, and references with the shared generation rules. Submission validates against current state again. If it fails after creation, the tool returns the original node receipt so the turn can continue from it. Bundled OpenCode excludes host Claude prompt files. It retains CLI access if the listener fails to start. Other harnesses continue to use their existing CLI integration.

## 对话轨迹页 / Conversation trace page

轨迹页 `/?traceProject=…&conversation=…&turn=…` 在一页内按顺序列出一个对话的全部已执行轮次,每轮可独立展开,展开时才读取该轮日志与关联 Job。轮次索引来自只读接口 `GET .../assistant/conversations/{conversation_id}/trace`,它解析当前对话、已归档对话和 `reusedCreateIds` 别名,只返回轮次元数据(序号、用户输入摘要、运行时、状态),不读取日志正文,不激活对话,不获取编辑租约。轮次由持久化的身份关系确定:用户消息与 id 为 `uuid5(用户消息 id, 'assistant')` 的助手消息成对才算已执行的一轮;没有该助手记录的用户消息(例如排队中、未执行)不出现。每轮使用自己记录的运行时选择日志源和解析器,仅 Default(opencode)显示模型原始请求。`GET .../traces/{turn_id}/conversation` 供旧的单轮链接查找所属对话;找不到时保持原单轮页面。
Expand Down
39 changes: 39 additions & 0 deletions docs/development/agent-image-ordinary-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Agent image latency / Agent 单图交互耗时

The manual browser benchmark sends ordinary English and Chinese single-image requests on a fresh canvas and on a canvas with an imported image. Run both prompt sets. The browser checks that the current turn created exactly one real image node and one job, that the adopted preview has the stored asset's bytes, and that pre-existing nodes and assets are preserved. Ordinary prompts leave output size and encoding open. The browser compares the decoded dimensions with the saved asset and job's technical check and logs the executed settings.

手动浏览器基准在空画布和已有导入图片的画布上发送中英文单图请求。运行两组不同措辞。浏览器核对本轮只创建一个真实图片节点和一个任务,采纳后预览与保存的素材字节一致,已有节点和素材保留。普通请求未指定尺寸和编码。浏览器核对解码尺寸与素材、任务技术检查记录一致,并记录实际执行参数。

- Count samples within the [3-second interaction budget](ui-performance.md#interaction-budgets) from Send to the new node's first visible frame. Report the time from the same click to the decoded result appearing in the node separately.
- 按[交互预算](ui-performance.md#interaction-budgets)统计从点击发送到新节点首次可见不超过 3 秒的样本。成图耗时从同一次点击到已解码图片在节点内可见,单独报告。

Build with `npm run build` and install the browser dependencies described in the [validation guide](validation.md). Use an empty data directory and an existing workspace with the named model connections. Live runs consume model quota.

用 `npm run build` 构建,并按[验证指南](validation.md)准备浏览器依赖。使用空数据目录和包含指定模型配置的现有工作区。真实运行消耗模型额度。

```sh
OPENDIRECTOR_ROOT="$(mktemp -d)" \
PYTHONPATH="$PWD/packages/core/src:$PWD/apps/api/src" \
python -m scripts.agent_image_benchmark_server \
--model-root /path/to/configured/workspace \
--agent-model MiniMax-M3.1-Flash --image-model MiniMax-H3-Image-Highspeed \
--port 18261
```

```sh
OPENDIRECTOR_API_URL=http://127.0.0.1:18261 npm run preview -- --port 15261 --strictPort
```

```sh
export CREATION_API_URL=http://127.0.0.1:18261 CREATION_WEB_URL=http://127.0.0.1:15261
node tests/browser/agent_image_ordinary.mjs
AGENT_IMAGE_PARAPHRASES=1 node tests/browser/agent_image_ordinary.mjs
```

To compare shared Agent runtime changes on the same input, run `AGENT_IMAGE_CASE=fixed AGENT_IMAGE_RUNS=5 node tests/browser/agent_image_ordinary.mjs`. This repeats the fixed 2048×2048 PNG request on a fresh canvas and checks the exact saved generation prompt, decoded size, and PNG encoding in addition to the node, job, and asset checks.

对比共享 Agent 运行时改动时,可用 `AGENT_IMAGE_CASE=fixed AGENT_IMAGE_RUNS=5 node tests/browser/agent_image_ordinary.mjs` 在空画布上重复发送同一条 2048×2048 PNG 请求。脚本校验保存的生图提示词、解码尺寸和 PNG 编码,同时核对节点、任务和素材。

The runner emits raw samples and per-case summaries as JSON Lines. Check `withinBudget` alongside successful generation and the two duration metrics.

脚本以 JSON Lines 输出逐次记录和各场景汇总。结合生成成功数和两项耗时核对 `withinBudget`。
2 changes: 2 additions & 0 deletions docs/development/lessons.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ When work exposes a consequential risk or a recurring problem, add a lesson or u

- **Reference merge follow-up:** Combining [#308](https://github.com/ThinkFlowLab/OpenDirector/pull/308) with #266 could merge cleanly while selecting capabilities from followed references and counting limits from followed plus pinned references. Use the same prepared reference list for both checks; verify pinned-only, mixed and duplicate-asset inputs after reopening. / **参考素材整合补充:** 能力选择和数量限制共用一次计算的请求素材,检查重开后的固定素材、混合素材及相同资产的去重;Git 自动合并成功不能证明两处使用相同输入。

- **Agent composition follow-up:** [PR #426](https://github.com/ThinkFlowLab/OpenDirector/pull/426) found that general parameter types exposed values the selected connection could reject. Creating the node first could leave an extra item when the Agent retried by creating another. Check proposed node settings and references with the same rules as submission before publishing a composed create-and-submit node. Preserve the original item identity when a later submission fails. / **Agent 组合操作补充:** 通用参数类型可能包含所选连接不支持的值。对于组合创建并提交,先用共享提交规则检查拟创建节点的参数和引用。如果创建后提交仍失败,用原节点身份继续处理。

## Operations across boundaries can lose identity or context

- **Trigger:** Importing a project, referencing assets across conversation turns, or switching projects during a run.
Expand Down
4 changes: 4 additions & 0 deletions docs/development/ui-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ Use these default budgets for routine UI actions such as opening a panel, naviga

Start both timers at the user action. Define the visible feedback and the usable result for each workflow, and calculate P90 using the nearest-rank method. For long-running operations such as generation or export, apply the feedback budget to acknowledging the action and measure completion separately against a workload-specific target.

When the user clicks Send in AI chat with an explicit request to generate one image, the image node created for that request must become visible on the canvas within **3 seconds of the click**. Measure separately from the same click until the generated image is visible in the node.

用户在 AI 对话中点击发送明确的单图生成请求后,本次请求创建的图片节点必须在 **3 秒内**在画布上可见。成图耗时单独测量,从同一次点击到生成的图片在节点内可见。

## Measurement

Use browser measurements for user-facing latency. Record first feedback and readiness separately, and split slow paths into client work, request queueing, server time, and rendering. Use comparable builds, fixtures, and interaction sequences. Keep initial load, warm navigation, direct manipulation, and long-running operations separate when setting budgets and reporting results.
Expand Down
5 changes: 5 additions & 0 deletions docs/guides/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,18 @@ opendirector canvas create-generation --help
```bash
opendirector project list
opendirector --project PROJECT_UUID project show
opendirector --project PROJECT_UUID project show --with-models
opendirector model list --kind video --search H3-Lite
opendirector model show MiniMax-H3-Lite
opendirector --project PROJECT_UUID asset import ./after.png --kind image
opendirector --project PROJECT_UUID canvas add-material --asset ASSET_UUID --x 0 --y 0
opendirector --project PROJECT_UUID video create --model MODEL_CONFIG_UUID --prompt "人物缓缓转头" --duration 5 --first-frame SOURCE_CANVAS_ITEM_UUID
```

`project show --with-models` 同时返回工程快照、模型目录和可用业务命令。在 Agent 的执行环境中,这些数据及本轮模型选择通过一次业务网关读取返回。

`project show --with-models` returns the project snapshot, model catalogue, and available business commands. In an Agent execution environment, one gateway read also includes the turn's model selections.

`canvas add-material` 和 `video create` 通过必填的 `--x`、`--y` 明确创建位置。Agent 先读取画布对象的位置、尺寸及分组,再安排新元素布局。`video create` 创建可编辑的生成节点与参考关系,返回 `specId` 和 `itemId`。所有画布修改命令以及批量事务通过共用返回层提供 `canvasItems`,包含元素身份、版本、位置、尺寸、父级、坐标系及可见性。位置未变化的操作也返回目标元素的几何信息,供后续布局判断。几何信息读取失败时,回执保留已提交状态并返回 `geometryReadError`。模型可通过配置 UUID 或唯一匹配的完整模型名指定。重名配置返回候选 UUID 和端点,由调用者明确选择。模型的输入模式必须支持所选参数。`--reference-item` 支持重复指定参考节点。提交生成时使用创建回执中的对象版本,后台任务由服务端继续执行。

```bash
Expand Down
Loading
Loading