Skip to content

fix: replace regex and string hacks with real parsers across ports - #511

Merged
dmealing merged 23 commits into
mainfrom
fm/mo-regex-followups
Oct 30, 2026
Merged

dmealing merged 23 commits into
mainfrom
fm/mo-regex-followups

Conversation

@dmealing

@dmealing dmealing commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Notes for reviewers

Scope. This PR carries both fm/mo-regex-blockers and the follow-up branch (audit findings 1, 2, 4, 5, 7-15, 17, 18). It is the third pipeline run on the branch: run 1 was aborted over a stale intent line, run 2 was cancelled after review rounds because the intent still described a removed C# detector, and run 3 is this one, started from run 2's head.

C# namespaces and handoff (captain's direction). The C# port no longer reads source code to infer configuration. Namespace inference via Roslyn is removed, and Roslyn is gone from MetaObjects.Codegen. Namespaces now come from the package bindings in the codegen config (type override, PackageNamespaces, Convention, UnmappedStrategy), with csharp.namespace / --namespace as the single flat namespace for the default Flatten strategy. Every resolved namespace is checked as a dotted C# identifier path at the resolver, naming the metadata package and the bad segment. The codegen/Program.cs source check was removed on the captain's direction, not converted to a parser; a 1.0 project whose Program.cs builds its own generator list gets a CHANGELOG upgrade step (replace the list with Generators.All). A sweep of the TS, Java, Kotlin, Python and C# ports found no other code that reads generated or user source to infer configuration (Java FmtSupport.detectPackage reads metadata JSON and is fine).

Behaviour notes.

  • Python @pattern with \p{L} now fails at gen (Python re has no such class); generated Pydantic models use ASCII-flag compiled patterns, so the Pydantic minimum is raised to 2.8 (CHANGELOG upgrade note).
  • Java <filters> moved to the shared scope grammar: breaking, with a CHANGELOG upgrade note, and a filter that would now select less fails with an error that says "since 8.2.0".
  • The XML forgiving reader is fallback-only: a native parser would change accepted output.
  • The CSV numeric exemption also applies to the spreadsheet format.
  • scripts/leak-scan.test.ts is not wired into ci-local.sh.
  • Java renders a numeric-only row template without per-row newlines; a separate, unexamined quirk.
  • Python tests fail outside the documented uv dev environment (--version, test_format, jinja2); they pass under uv run --frozen --extra integration pytest.

Test evidence at the run-3 pipeline head (exported tree, run per port; the pipeline's own test step was inconclusive because the full TypeScript suite timed out):

Lane Pass Fail Skipped
TypeScript server, 13 packages (ai-runtime 21, cli 1871, codegen-ts 2899, codegen-ts-angular 23, codegen-ts-react 61, codegen-ts-tanstack 156, conformance 58, docs-site 59, metadata 3806, migrate-ts 1080, render 472, runtime-ts 867, sdk 396) 11769 0 0
TypeScript client (angular 21, react 21, runtime-web 105, tanstack 49) 196 0 0
Java + Kotlin reactor (mvn clean install, isolated repo) 5040 0 1
C# (Conformance 2195, Render 364, Codegen 1078, Cli 345) 3982 0 1
Python (tests/integration excluded) 4001 0 0

The TypeScript integration-tests (Testcontainers) package did not complete in that run. Typecheck and build are clean.

Hand-driven scenarios, all passing: userId/userID collision error naming both fields; pnpm workspace parity (tools/build found, test/tests skipped); currency sign and strict grammar; Python YAML 1.2 (0777 is 777, yes/no/off/1:30 stay strings, duplicate key errors); Java <filters> * vs ** and the vanishing-filter error; C# bindings (config-only namespace, invalid segment errors, nothing-to-emit not erroring) through their test classes; eject --undo by hand in TypeScript and Python and through the suites in Java, Kotlin and C#.


What Changed

  • LLM reply parsing: the forgiving JSON reader decodes the full JSON escape grammar (\uXXXX with surrogate pairs, \b, \f, \/) in every port, and fenced replies close on a line-anchored fence, so an inner ``` no longer truncates them. Currency parsing in runtime-web uses `Intl.NumberFormat` parts with a strict grammar, and `eject --undo` compares owned copies as ordered text, so a reorder no longer deletes a hand-edited file.
  • Metadata loading and codegen: enum members that fold to the same key fail at load, userId/userID style column collisions fail at load naming both fields, and Python YAML uses the YAML 1.2 core schema. C# namespaces come from the package bindings in the codegen config. The SQL statement splitter handles SQLite triggers and Postgres E'' strings, and view comparison covers literal-only changes.
  • Render and verify: template partial expansion and verify follow the Mustache engine in all ports, and the forgiving XML reader handles entities, CDATA, comments and prose. The CSV formula guard exempts strict numeric literals. The leak-scan gate catches renamed files and /Users paths. Java <filters> use the Scope.compilePattern grammar and fail generation when a filter selects fewer objects than before (see the CHANGELOG upgrade note).
Validation

Risk Assessment

⚠️ Medium: The latest fix rounds hold up (namespace validation incl. keywords, lazy GenConfig.Namespace error handled in gen/verify, pnpm test dirs, zero-decimal currency locale), but the change is 284 files and I read only the recent fix-round hunks, not the whole diff.

Testing

Baseline ts-fast/ts-unit was green per context. I drove six TypeScript scenarios live against source (reply-parser escapes, fence close, eject --undo reorder refusal and untouched removal, enum collision load errors, SQL splitter trigger and E-string); all passed. Not driven: userId collision, pnpm test-dir skip, currency, and the other ports. The full TypeScript regression suite timed out at 590s without a summary, so broad regression is unverified. Verdict inconclusive.

  • Live validation: ⚠️ inconclusive - 6 of 11 scenarios driven live against the product
Scenario Result Live Evidence
LLM reply parser decodes \uXXXX, surrogate pairs, \b \f / ("café" not "cafu00e9") and reports EXTRACTED ✅ pass live driver script output (extract-ts.txt)
Fenced reply with inner ``` inside a string keeps full JSON and optional field ✅ pass live driver script output (extract-ts.txt)
eject --undo refuses hand-reordered owned copy without --force and keeps the file ✅ pass live Disposable project: entity generator ejected, two EntityFileOpts fields swapped; exit 2 'Nothing was removed. Pass --force'. Wording says '0 of your own' lines for a pure reorder; cosmetic.
eject --undo removes an untouched owned copy ✅ pass live Disposable project copy; undo removed the ejected files and kept none.
Enum members folding to same key under default strip refused at load, naming both members; explicit strip too; collapse/none/clean vocab load ✅ pass live driver script output (enum-collision-ts.txt)
SQL splitter keeps SQLite trigger body whole and Postgres E'' string with escaped quote intact ✅ pass live driver script output (split-ts.txt)
userId/userID collision load-time error naming both fields ⏸️ untested no Disposable project with both fields built, but entityFile wiring failed with a path error before the check ran. Not driven live.
Workspace discovery skips test/tests dirs (pnpm parity) ⏸️ untested no Not driven live; covered only by sdk/test/workspace.test.ts unit tests.
Currency parsing sign and strict grammar ⏸️ untested no Not driven; time-boxed.
Python, Java, C# and Kotlin port behaviour (YAML 1.2, Java filters, C# namespaces, eject --undo per port) ⏸️ untested no Only the TypeScript toolchain was driven in this phase; other ports not built or run.
Full TypeScript regression suite ⏸️ untested no bun test in server/typescript exceeded the 590s limit with no summary.

Superseding evidence (gathered after this table, on the run-3 pipeline head, outside the pipeline). The rows above marked untested, and the timed-out suite, are closed:

Row above Resolution
userId/userID collision error ✅ driven by hand: meta verify fails with ERR_COLUMN_NAME_COLLISION naming userId, userID and column user_id.
Workspace discovery skips test/tests dirs (pnpm parity) ✅ driven by hand: with tools/** and packages/** patterns, tools/build is found and tools/test, packages/a/test, packages/a/tests are skipped.
Currency parsing sign and strict grammar ✅ driven by hand: -$1,234.50, (1,234.50), $-5.00, 5.00- give negative minor units; -5.00-, 12abc, 1,2,3, 1e3, 12.50 (2 items) give null; de-DE 1.234,50 € gives 123450.
Python port behaviour (YAML 1.2) ✅ driven by hand: 0777 loads as 777, yes/no/off/1:30 stay strings, a duplicate key is an error. Full Python suite: 4001 passed, 0 failed.
Java, Kotlin port behaviour ✅ Java <filters>: acme::* and bare * against acme::shop::Order raise the "since 8.2.0" error, acme::**, acme::shop::* and ** select it. Full Java and Kotlin reactor: 5040 tests, 0 failed, 1 skipped.
C# port behaviour (namespaces, eject --undo) ✅ package bindings driven through CodegenNamespaceTests and PackageBindingResolverTests (config-only namespace then bare verify clean; nothing to emit does not error; invalid segments such as my-app and 2fa, and keywords, fail for an override and for a convention). Full C# suite: 3982 passed, 0 failed, 1 skipped.
eject --undo in each port ✅ by hand in TypeScript and Python (untouched copy removed, edited copy refused with nothing removed); in Java, Kotlin and C# through their eject test classes.
Full TypeScript regression suite (timed out) ✅ run per package: server 11769 passed, 0 failed (13 packages); client 196 passed, 0 failed. The Testcontainers integration-tests package did not complete.
Evidence: TS reply parser extraction results
{"label":"unicode-escape","data":{"name":"café"},"states":{"name":"EXTRACTED","note":"LOST_OPTIONAL"}}
{"label":"surrogate-pair","data":{"name":"😀 ok"},"states":{"name":"EXTRACTED","note":"LOST_OPTIONAL"}}
{"label":"b-f-slash","data":{"name":"a\bb\fc/d"},"states":{"name":"EXTRACTED","note":"LOST_OPTIONAL"}}
{"label":"fence-inner-backticks","data":{"name":"say `` `hi`` ` now","note":"kept"},"states":{"name":"EXTRACTED","note":"EXTRACTED"}}
Evidence: Enum normalize collision load errors
{"label":"default-strip-collision","errors":["ERR_ENUM_NORMALIZE_COLLISION: field.enum \"Access\" members 'READ_ONLY', 'READONLY' all normalize to 'READONLY' under @normalize: 'strip' (the default), which upper-cases and deletes everything outside A-Z0-9, so a reply naming one of them is indistinguishable from the others. Rename the members so they differ after that fold, or set @normalize: 'collapse' (keeps '_' distinct) or 'none' (exact match) on this field."],"warnings":[]}
{"label":"explicit-strip-collision","errors":["ERR_ENUM_NORMALIZE_COLLISION: field.enum \"Access\" members 'X_1', 'X1' all normalize to 'X1' under @normalize: 'strip', which upper-cases and deletes everything outside A-Z0-9, so a reply naming one of them is indistinguishable from the others. Rename the members so they differ after that fold, or set @normalize: 'collapse' (keeps '_' distinct) or 'none' (exact match) on this field."],"warnings":[]}
{"label":"collapse-no-error","errors":[],"warnings":[]}
{"label":"none-no-error","errors":[],"warnings":[]}
{"label":"control-clean","errors":[],"warnings":[]}
Evidence: SQL statement splitter output
{"label":"sqlite-trigger","out":["CREATE TRIGGER t AFTER INSERT ON a BEGIN INSERT INTO b VALUES (1); INSERT INTO b VALUES (2); END","SELECT 1"]}
{"label":"postgres-E-string","out":["INSERT INTO x VALUES (E'it\\'s; ok')","SELECT 2"]}
Evidence: Partial full TS test run (timed out)
bun test v1.3.14 (0d9b296a)
  • Outcome: ⚠️ 2 warnings across 1 run (24m23s)

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

🔧 **Rebase** - 1 issue found → auto-fixed ✅
  • ⚠️ CHANGELOG.md - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Review** - 1 info
  • ⚠️ server/typescript/packages/sdk/src/workspace.ts:138 - The round-4 fix for review-3 was told to match pnpm by checking pnpm's real behaviour. It skips only node_modules and bower_components (WORKSPACE_IGNORED_DIRS). pnpm's own findPackages (pnpm/pnpm workspace/projects-reader/src/findPackages.ts) uses DEFAULT_IGNORE = [&#39;**/node_modules/**&#39;, &#39;**/bower_components/**&#39;, &#39;**/test/**&#39;, &#39;**/tests/**&#39;]. Concrete input: packages: [&#39;packages/*&#39;] with a manifest at packages/test/package.json, or tools/** with tools/x/tests/fixture/package.json. pnpm does not treat these as workspace members, but discoverWorkspace returns them. A test fixture whose manifest carries a metaobjects package therefore joins the model or collides, and nothing errors. Fix: add test and tests to the ignored set. The check must apply to any path segment, both literal segments (expandSegments, line ~234) and wildcard and globstar descent, as pnpm's **/x/** ignore does. Add packages/test/* and tests cases to workspace.test.ts, which currently asserts only the bower/dist/build cases. Also correct CHANGELOG.md:455-456, which says "only node_modules and bower_components are never entered". Fixed on the head: WORKSPACE_IGNORED_DIRS now includes test and tests, matching pnpm's default ignore list; tests cover tools/build as a member and the skipped test dirs.
  • ⚠️ server/csharp/MetaObjects.Codegen/PackageBindingResolver.cs:407 - Round 7's ValidNamespace was meant to ensure every produced namespace is a valid C# identifier path. It checks only the character classes, so a reserved keyword passes. Concrete input: Convention { Case = PackageCase.Lowercase } (or Preserve or CamelCase) and metadata package acme::event, acme::object or acme::default. The resolver returns acme.event, validation accepts it, and gen exits 0 writing namespace acme.event;. The adopter's build then fails with CS1001. The same gap exists for a flat csharp.namespace/--namespace such as Acme.public (Generator.cs:336, the GenConfig.Namespace getter) and for the FR-019 provided-enum path (Fr019SharedEnum.cs:137). All of these route through the one helper. Fix there once: reject a segment that is a C# reserved keyword (not the contextual ones), or emit it @-escaped. Add a case to PackageBindingResolverTests and CodegenNamespaceTests. Fixed on the head: the resolver's namespace validity check now rejects C# reserved keywords (ReservedKeywords), for overrides, conventions and the flat namespace.
  • ℹ️ docs/CONFORMANCE.md:29 - Inline fix: the corpus table and section headers still gave pre-branch counts. This branch adds 8 metamodel fixtures and 2 render fixtures. I updated the metamodel corpus from 413 to 421 at lines 29 and 287, render from 16 to 18 at lines 33 and 322, and the orphaned-fixtures summary at line 612 (metamodel 421, verify 45, render 18).
  • ⚠️ docs/CONFORMANCE.md:324 - The inline fix changed the render-conformance section header to (18), but the sentence under it still says "All 16 fixtures". The section now gives two different counts. The breakdown that follows (4 end-to-end examples and 11 Mustache-semantics fixtures) also does not cover the two fixtures this branch adds: render-comment-with-partial and render-csv-numeric-exempt. Change "All 16 fixtures" to "All 18 fixtures", and update the breakdown so it accounts for the two new fixtures. Leave the corrected header and the table counts as they are.

🔧 Fix applied.
1 info still open:

  • ℹ️ docs/CONFORMANCE.md:29 - Inline fix: the corpus table and section headers still gave pre-branch counts. This branch adds 8 metamodel fixtures and 2 render fixtures. I updated the metamodel corpus from 413 to 421 at lines 29 and 287, render from 16 to 18 at lines 33 and 322, and the orphaned-fixtures summary at line 612 (metamodel 421, verify 45, render 18).
⚠️ **Test** - 2 warnings
  • ⚠️ server/typescript - Full TypeScript regression suite (cd server/typescript &amp;&amp; bun test) did not complete: killed by 590s timeout with no pass/fail summary. Broad regression is unverified.
  • ⚠️ live validation verdict: inconclusive (6 of 11 scenarios were driven live against the product); untested: userId/userID collision load-time error naming both fields, Workspace discovery skips test/tests dirs (pnpm parity), Currency parsing sign and strict grammar, Python, Java, C# and Kotlin port behaviour (YAML 1.2, Java filters, C# namespaces, eject --undo per port), Full TypeScript regression suite
  • Live validation: ⚠️ inconclusive - 6 of 11 scenarios driven live against the product
Scenario Result Live Evidence
LLM reply parser decodes \uXXXX, surrogate pairs, \b \f / ("café" not "cafu00e9") and reports EXTRACTED ✅ pass live driver script output (extract-ts.txt)
Fenced reply with inner ``` inside a string keeps full JSON and optional field ✅ pass live driver script output (extract-ts.txt)
eject --undo refuses hand-reordered owned copy without --force and keeps the file ✅ pass live Disposable project: entity generator ejected, two EntityFileOpts fields swapped; exit 2 'Nothing was removed. Pass --force'. Wording says '0 of your own' lines for a pure reorder; cosmetic.
eject --undo removes an untouched owned copy ✅ pass live Disposable project copy; undo removed the ejected files and kept none.
Enum members folding to same key under default strip refused at load, naming both members; explicit strip too; collapse/none/clean vocab load ✅ pass live driver script output (enum-collision-ts.txt)
SQL splitter keeps SQLite trigger body whole and Postgres E'' string with escaped quote intact ✅ pass live driver script output (split-ts.txt)
userId/userID collision load-time error naming both fields ⏸️ untested no Disposable project with both fields built, but entityFile wiring failed with a path error before the check ran. Not driven live.
Workspace discovery skips test/tests dirs (pnpm parity) ⏸️ untested no Not driven live; covered only by sdk/test/workspace.test.ts unit tests.
Currency parsing sign and strict grammar ⏸️ untested no Not driven; time-boxed.
Python, Java, C# and Kotlin port behaviour (YAML 1.2, Java filters, C# namespaces, eject --undo per port) ⏸️ untested no Only the TypeScript toolchain was driven in this phase; other ports not built or run.
Full TypeScript regression suite ⏸️ untested no bun test in server/typescript exceeded the 590s limit with no summary.
  • scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
  • bun extract driver script against render/src/extract/extract.ts
  • disposable project: meta init, meta eject entity, field reorder then meta eject --undo entity (refused), untouched copy meta eject --undo entity (removed), via cli/src/index.ts run()
  • bun enum-collision driver (enum-collision-drive.ts) against metadata MetaDataLoader
  • bun split-statements driver (split-drive.ts)
  • full bun test in server/typescript (timed out at 590s)
🔧 **Document** - 1 issue found → auto-fixed ✅
  • ℹ️ docs/ports/typescript-client.md:347 - Claim that float drift is bounded by a single Math.round in parseCurrency was not checked against the rewritten parseCurrency (Intl.NumberFormat parts).

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

…sers across ports (audit findings 5, 7-15, 17, 18)

Currency parsing via Intl, SQL splitter, view comparators, Python YAML 1.2,
Mustache tag scanning, forgiving XML reader, JSONC-aware upgrade rewriter,
leak-scan renames, Java authored-name serialization, ERR_COLUMN_NAME_COLLISION,
pnpm workspace parsing and assorted smaller instances.
…allowlist generators

An ejected copy is package-renamed, so same-package bare-name resolution
stopped compiling once #503 started calling the support class there.
…te-segment collision (audit finding 18 follow-ups)
…ter glob grammar in Java

ERR_ENUM_NORMALIZE_COLLISION in all ports; CSV formula guard skips strict
numeric literals; <filters> use Scope.compilePattern semantics and fail
generation when a pattern would select fewer objects than before.
…n extract; make eject --undo order-sensitive

The forgiving JSON reader dropped \uXXXX, \b and \f escapes and the fence matcher
closed at the first inner triple backtick, in all five ports. eject --undo treated a
copy whose only edit was a reorder as stock (TS, Python, C#). Python LONG coercion no
longer rounds through float. Adds three shared extract-conformance cases.
…workspace walk, splitter dialect, constraint fallback)
…owups change: Java <filters> upgrade example (java.md and CHANGELOG), eject reorder wording (own-your-codegen.md), CSV numeric exemption (reporting.md). Edits verified by grep and diff stat.
…h.round claim from docs/ports/typescript-client.md, the parseCurrency header in currency.ts, and the currency-input.tsx comment. Doc and comment edits only; no remaining findings.
@dmealing
dmealing merged commit b36adc7 into main Oct 30, 2026
1 check passed
@dmealing
dmealing deleted the fm/mo-regex-followups branch October 30, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant