Repository navigation
Add a twin loop that lifts simulator observations back onto the model - #54
Open
haphantran wants to merge 1 commit into
Open
haphantran wants to merge 1 commit into
haphantran wants to merge 1 commit into
Conversation
Generation fidelity compares the model with the generated files. The twin loop compares the model with what the generated scene does when it runs. - The Smart Warehouse Omniverse templates stamp every layout prim and layout record with the ID of its model element (spatialDsl:elementId, id). - observe.py records the world pose of every stamped prim as a simulator-neutral observation log. The Isaac Sim bridge writes it with --observe-out; run_standin_sim.py writes the same log with no simulator. - benchmarks/twin-loop.ts lifts each sample onto a copy of the model (design pose plus the residual between observed and authored scene pose) and evaluates the design-time spatial rules on it. It reports realization offsets, runtime drifts, rule verdicts that differ between model and run, and a reconciliation proposal, each traced to the model element, prim, generated file, and template.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can delete observation inputs and produce incorrect drift, rule-verdict, and reconciliation results through incomplete scene fingerprints and mishandled pose observations.
7 open issues
What changed in this PR
Adds a twin loop that extends generation fidelity checks to runtime behavior, mapping simulator observations back onto the Smart Warehouse model and reusing its spatial rules.
Changes:
- Stamps generated layout records and USD prims with model element IDs.
- Adds observation logging, Isaac Sim recording support, and a simulator-free stand-in with injectable disturbances.
- Adds pose lifting, rule comparisons, reconciliation reports, and three self-checking scenarios.
| File | Description |
|---|---|
frontend/src/examples/data/smart-warehouse-omniverse-project.json |
Adds element ID stamps to bundled app templates. |
examples/omniverse-assets/warehouse_sim/test_observe.py |
Tests logging, stand-in disturbances, and USD snapshots. |
examples/omniverse-assets/warehouse_sim/run_warehouse_sim.py |
Adds runtime observation recording options. |
examples/omniverse-assets/warehouse_sim/run_standin_sim.py |
Adds kinematic simulation and injected disturbances. |
examples/omniverse-assets/warehouse_sim/README.md |
Documents recording and observation tests. |
examples/omniverse-assets/warehouse_sim/observe.py |
Defines the log format, recorder, and pose snapshots. |
examples/codegen-projects/smart-warehouse-omniverse-project.json |
Adds element ID stamps to standalone generation templates. |
docs/user-guide/omniverse-code-generation.md |
Documents generated element IDs and runtime tracing. |
benchmarks/twin-loop.ts |
Generates bundles, runs scenarios, and analyzes external logs. |
benchmarks/README.md |
Documents twin-loop setup, usage, and outputs. |
benchmarks/package.json |
Registers the twin-loop command. |
benchmarks/lib/twin.ts |
Implements pose lifting, deviation analysis, and reconciliation reports. |
Review effort: Balanced
Comment on lines
+267
to
+269
| const traced = model.elements.filter(element => ( | ||
| element.presentation?.position3D && authored.has(element.id) && first.poses.has(element.id) | ||
| )); |
| }; | ||
| traced.filter(element => !controlled.has(element.id)).forEach(element => { | ||
| const start = first.poses.get(element.id)!; | ||
| if (exceeds(residual(authored.get(element.id)!, start), tolerances)) { |
Comment on lines
+296
to
+297
| const end = last.poses.get(element.id); | ||
| if (onset && end) deviations.push(finding('runtime-drift', element, onset.t, start, end)); |
Comment on lines
+342
to
+343
| ...[...designIssues.entries()].filter(([key]) => !seen.has(key)) | ||
| .map(([, issue]) => ruleFinding('design-violation-not-observed', issue)), |
| templateId: templates[index].id, | ||
| templateName: templates[index].name, | ||
| })); | ||
| fs.rmSync(outDir, { recursive: true, force: true }); |
| const authored = readObservationLog(authoredPath); | ||
|
|
||
| const analyse = (name: string, title: string, observations: ObservationLog): TwinReport => { | ||
| if (observations.header.layoutSha256 !== layoutSha256) { |
| pose["x"] += disturbance["dx"] | ||
| pose["y"] += disturbance["dy"] | ||
| pose["z"] += disturbance["dz"] | ||
| pose["yaw"] += disturbance["dyaw"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Generation fidelity (
benchmarks/fidelity) compares the model with the generated files. The twin loop continues from there: it compares the model with what the generated scene does when it runs, and brings the result back in the model's own terms.spatialDsl:elementId) and everywarehouse_layout.jsonrecord (id).observe.pyrecords the world pose of every stamped prim as a simulator-neutral JSON Lines log. The Isaac Sim bridge writes it with--observe-out.run_standin_sim.pywrites the same log with no simulator: the MAS brain drives the robots kinematically and--disturbdisplaces an element at a chosen time.benchmarks/twin-loop.tslifts each sample onto a copy of the model (design pose plus the residual between observed and authored scene pose) and evaluates the design-time spatial rules, unchanged, on the lifted model.The report lists realization offsets, runtime drifts, rule verdicts that differ between the model and the run, and a reconciliation proposal (the model changes that accepting the observed state would make, and the rule issues that model would gain or lose). Each finding names the model element, the prim, the generated file, and the template that produced it. The log records the hash of the layout it ran against, and the script refuses a log recorded against a different generation of the model.
Try it
It runs three stand-in scenarios in about 5 seconds and exits non-zero if the findings differ from what was injected: an undisturbed run, a storage rack pushed into the aisle at t = 20 s (one drift, and the rack aisle rule fails at runtime only), and a dock resting 150 mm above its authored pose (one realization offset).
benchmarks/README.mdhas the details and the command for analysing a log recorded in Isaac Sim.Verified
npm run twin-loop: all three scenarios report exactly what was injected.npm run fidelity: 105,927 values compared, 0 mismatches, byte-identical regeneration, USD validation passes with the stamped templates.python3 -m unittest test_nav test_brain test_runtime_modes test_layout test_observe: 23 tests pass (the pxr snapshot test runs when usd-core is installed).Not verified, and known limits
--observe-outhas not been run inside Isaac Sim yet. The recording code path inrun_warehouse_sim.pyis untested there.Deployment note
Merging to
maindeploys to production. The only change that ships in the app is the bundled Smart Warehouse Omniverse example project, whose generated output gains the two ID fields. Everything else is underbenchmarks/,examples/, anddocs/.