Skip to content

feat: add CSV, JSON and XML export to report list route - #471

Merged
dmealing merged 7 commits into
mainfrom
fm/mo-report-export
Oct 28, 2026
Merged

dmealing merged 7 commits into
mainfrom
fm/mo-report-export

Conversation

@dmealing

Copy link
Copy Markdown
Member

What Changed

  • Add ?format=csv|json|xml to the report list route in every port. A new report-export catalog generator (TypeScript, C#, Java, Kotlin, Python) emits a per-report export module and a shared writer. Unknown formats return 400 invalid_format, and exports over maxRows (default 100,000) return 400 export_too_large before any bytes are written.
  • Route handlers gain a format branch only when an export generator is wired. The ?format= parsing, the two error envelopes and Content-Disposition live in each port's existing runtime HTTP helper. The change adds no new dependency for adopters.
  • Document the export contract: ADR-0031 amendment, api-contract.md, reporting.md, per-port docs, and the CHANGELOG. The report/ api-contract corpus gains six export scenarios (CSV, JSON, XML, invalid format, too large, with count).
Validation

Risk Assessment

✅ Low: The fix round (e7ace3f) cleanly resolves F1 through F5 across the TS, C#, Java, Kotlin and Python ports. It adds independent-oracle tests, and I found no defect I could substantiate with a concrete failing path.

Testing

Drove the shared export-* corpus scenarios (csv, json, xml, invalid-format, too-large, with-count) against each port's generated API over HTTP, plus the Java/Kotlin/C#/Python/TS codegen and compile-and-run tests. All passed. The only incomplete item is the --quick mutation gate, which timed out under my own time limit; every step before it was green, and the configured baseline (ts-fast, ts-unit) had already passed. No UI surface, so no screenshots.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
GET report list with ?format=csv returns RFC 4180 body with text/csv and Content-Disposition attachment, in every port ✅ pass live export-csv.yaml corpus run over HTTP: bun api-contract-report*, pytest test_api_contract_report, ReportOmdbSqlite, ReportGeneratedApiContractConformanceTest, dotnet IntegrationTests Report*
?format=json returns today's array body plus Content-Disposition; ?format=xml returns the row XML with application/xml ✅ pass live export-json.yaml and export-xml.yaml corpus scenarios, all five ports (XML compared by infoset)
Adversarial: unknown/unwired ?format=pdf returns 400 invalid_format with the expected list ✅ pass live export-invalid-format.yaml corpus scenario, all five ports
Adversarial: export over the maxRows cap is refused with 400 export_too_large before any body byte ✅ pass live export-too-large.yaml corpus scenario, all five ports
Adversarial: withCount with csv/xml returns 400 export_with_count; with json returns the envelope plus attachment headers; without format unchanged ✅ pass live export-with-count.yaml (r1-r4) corpus scenario, all five ports
Without ?format the report list route is byte-identical to before; export branch absent when report-export is not wired ✅ pass live existing list/pagination/filter/sort corpus scenarios pass; ReportingInertTest (Java), test_unwired_routes_carry_no_export_branch (Python), ReportingInertTests (C#)
Generated export code compiles and runs, with wire-aware decimals and temporal cells on JVM ✅ pass live GeneratedReportExportCompileRunTest (12 pass), ReportExportTest, C# ReportExportGeneratorTests, Python test_report_export.py

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ server/typescript/packages/runtime-ts/src/drizzle-fastify/report-export.ts:86 - The intent says "withCount with a non-JSON format returns 400", and it says "json is today's body plus Content-Disposition". The change refuses withCount with every format, JSON included: 400 export_with_count, documented as "An export has no { rows, total } envelope, JSON included." Under the intent's wording, ?format=json&withCount=1 would return the {rows,total} envelope with an attachment header. The same rule appears in every port: TS Fastify mount-read-only.ts:211 and Hono mount-read-only.ts:180 (if (exportSpec && format !== undefined && isExportCountFlag(...)) throw exportWithCountError()), C# ReportExportHttp.Refuse, Python router_generator.py _report_list_with_export (if is_truthy_flag(with_count)), Java SpringControllerGenerator appendReportExportHandler (ReportExport.isTruthy(withCount)), Kotlin appendReportExportHandler. The corpus pins it in fixtures/api-contract-conformance/report/scenarios/export-with-count.yaml r2, and api-contract.md, reporting.md and ADR-0031 Amendment 1 describe it. Confirm that refusing JSON is the intended contract, or narrow the refusal to csv and xml.
  • ⚠️ server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/SpringReportExportGenerator.java:346 - The intent requires CSV values "in wire form (currency as integer minor units, dates YYYY-MM-DD, decimals per wire...)", and api-contract.md and reporting.md say "a decimal as the wire setting spells it". The Java and Kotlin writers always write a decimal in canonical form (stripTrailingZeros().toPlainString()), whatever the wire setting. Under the default wire.decimals: native, the JVM list route and the JSON export write 12.50, but the CSV and XML exports of the same row write 12.5. docs/ports/java.md calls this deliberate ("under either wire setting"), which contradicts the intent and the cross-port contract text. Same rule in Kotlin: KotlinReportExportGenerator.kt:296 fun decimal. C# (DecimalsAsString from ctx.Root.WireFormat), TS (wire-form rows) and Python (render_wire) do follow wire. Decide whether JVM export should read loader.getRoot().getWireFormat() the way SpringDtoGenerator does, or whether the contract docs should carve out a JVM exception.
  • ⚠️ server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/SpringControllerGenerator.java:158 - The intent says the format branch is generated by the route generator "only when an export generator is wired". On Java and Kotlin the branch is controlled by a separate routes arg, reportExport=true, and the presence of report-export in the run has no effect. Two incorrect setups follow. (a) Set reportExport=true without wiring report-export: the controller references <Report>Export.MAX_ROWS, .csv(rows) and .xml(rows), which nobody emitted, so the project does not compile. (b) Wire report-export without the arg: the build emits orphan <Report>Export modules, and ?format= is silently ignored. TS (emitsReportExport), C# (GenContext.ReportExportMaxRows) and Python (REPORT_EXPORT_GENERATOR in ctx.generators) all detect the generator automatically. Same knob in Kotlin: KotlinSpringControllerGenerator.kt:130 reportExport(). docs/ports/java.md justifies it: "the generator list is not visible to a sibling generator". Detecting the sibling may need new plumbing in the Maven plugin or SPI, and that remedy is what needs approval.
  • ⚠️ server/csharp/MetaObjects.Codegen/Generators/ReportExportGenerator.cs:190 - Some temporal cells in CSV and XML are not in the list route's wire form, so the exported file and the JSON export disagree for the same row without any error. C#: DateTimeOffset is formatted yyyy-MM-dd'T'HH:mm:ss.FFFFFFF'Z', but Iso8601TimestampConverter (the list route's wire form) writes whole seconds or exactly .fff. An instant at 10:00:00.120 exports as .12Z against wire .120Z, and a Postgres microsecond value exports as .123456Z against wire .123Z. Java: SpringReportExportGenerator.java:341 raw() uses String.valueOf. LocalTime (field.time) and LocalDateTime (@localTime timestamp) toString() drop zero seconds (10:00 and 2026-01-01T10:00), while Jackson's JSR-310 serializers write 10:00:00 and 2026-01-01T10:00:00. Kotlin: KotlinReportExportGenerator.kt:293 raw() uses value.toString(), same defect. Reachable by a report with a timestamp or time dimension, or by a min/max measure on one. Fix: use the converter's format in C#, and use DateTimeFormatter.ISO_LOCAL_TIME / ISO_LOCAL_DATE_TIME (or the ObjectMapper's value) for java.time types on JVM.
  • ⚠️ server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportExportGeneratorTest.kt:66 - Several new tests check substrings of generated source as a stand-in for behaviour, which the test-quality rule forbids. One cannot fail: "const val SEGMENT: String = \"store_totals\"" in module || "SEGMENT" in module passes as soon as the word SEGMENT appears anywhere. Others pin incidental text rather than behaviour: C# ReportExportGeneratorTests.cs:75 asserts the exact ReportExportHttp.Csv(http, "line_totals", w => LineTotalsExport.WriteCsv(w, rows)) call string, and C# ReportExportGeneratorTests.cs lines 64–66 and Python test_report_export.py:153 ("format" not in router.replace("information", "")) prove absence by grepping for a word. Kotlin test lines 341–345 have the same problem. Each of these breaks on a behaviour-preserving rename and passes on dead code. Better oracles already exist next to them: for the "no export wired" case, compare output byte-for-byte with and without the generator (as the Kotlin test does at line 347). For the wired case, rely on the compile-and-run tests and the api-contract export-* corpus, which exercise the HTTP behaviour against independently written expected bodies.
  • ℹ️ docs/features/reporting.md:805 - Inline doc fix. The per-port table, and ADR-0031 Amendment 1 at spec/decisions/ADR-0031-serialization-vs-data-download-boundary.md:88, said the Python XML writer is xml.sax.saxutils.XMLGenerator. The generated Python writer (report_export_generator.py _WRITER_SOURCE) builds the XML itself with xml.sax.saxutils.escape/quoteattr and never uses XMLGenerator. Both lines now say it is a generated writer over xml.sax.saxutils escaping.

🔧 Fix applied.
1 info still open:

  • ℹ️ docs/features/reporting.md:805 - Inline doc fix. The per-port table, and ADR-0031 Amendment 1 at spec/decisions/ADR-0031-serialization-vs-data-download-boundary.md:88, said the Python XML writer is xml.sax.saxutils.XMLGenerator. The generated Python writer (report_export_generator.py _WRITER_SOURCE) builds the XML itself with xml.sax.saxutils.escape/quoteattr and never uses XMLGenerator. Both lines now say it is a generated writer over xml.sax.saxutils escaping.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
GET report list with ?format=csv returns RFC 4180 body with text/csv and Content-Disposition attachment, in every port ✅ pass live export-csv.yaml corpus run over HTTP: bun api-contract-report*, pytest test_api_contract_report, ReportOmdbSqlite, ReportGeneratedApiContractConformanceTest, dotnet IntegrationTests Report*
?format=json returns today's array body plus Content-Disposition; ?format=xml returns the row XML with application/xml ✅ pass live export-json.yaml and export-xml.yaml corpus scenarios, all five ports (XML compared by infoset)
Adversarial: unknown/unwired ?format=pdf returns 400 invalid_format with the expected list ✅ pass live export-invalid-format.yaml corpus scenario, all five ports
Adversarial: export over the maxRows cap is refused with 400 export_too_large before any body byte ✅ pass live export-too-large.yaml corpus scenario, all five ports
Adversarial: withCount with csv/xml returns 400 export_with_count; with json returns the envelope plus attachment headers; without format unchanged ✅ pass live export-with-count.yaml (r1-r4) corpus scenario, all five ports
Without ?format the report list route is byte-identical to before; export branch absent when report-export is not wired ✅ pass live existing list/pagination/filter/sort corpus scenarios pass; ReportingInertTest (Java), test_unwired_routes_carry_no_export_branch (Python), ReportingInertTests (C#)
Generated export code compiles and runs, with wire-aware decimals and temporal cells on JVM ✅ pass live GeneratedReportExportCompileRunTest (12 pass), ReportExportTest, C# ReportExportGeneratorTests, Python test_report_export.py
  • scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
  • cd server/typescript/packages/integration-tests && bun test test/api-contract-report-corpus.test.ts test/api-contract-report-sqlite.test.ts test/api-contract-report.test.ts (130 pass; generated Fastify/Hono servers booted over HTTP)
  • uv run --extra integration --extra dev pytest tests/integration/test_api_contract_report.py tests/codegen/test_report_export.py (134 pass; generated FastAPI app, Postgres/MySQL containers)
  • mvn -pl codegen-spring -am test -Dtest=GeneratedReportExportCompileRunTest,ReportExportTest,ReportingInertTest (23 pass)
  • mvn -f integration-tests/pom.xml test -Dtest=ReportOmdbSqliteApiContractConformanceTest (38 pass; Tomcat-hosted generated Spring controller)
  • mvn -f integration-tests-kotlin/pom.xml test -Dtest=ReportGeneratedApiContractConformanceTest (113 pass)
  • dotnet test MetaObjects.IntegrationTests --filter FullyQualifiedName~Report (112 pass; generated ASP.NET routes against Testcontainers)
  • dotnet test MetaObjects.Codegen.Tests (1037 pass, 1 skipped)
  • scripts/ci-local.sh --quick (hygiene, parity gates and TS unit suites green; the stryker mutation gate hit my 590s timeout, no failures before it)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…all five ports

Adds the standalone report-export catalog generator (importable and ejectable in
every port), the ?format= branch on report list routes, the invalid_format,
export_too_large and export_with_count envelopes, a maxRows cap that never
truncates, a CSV formula-injection guard, and six shared api-contract scenarios.
ADR-0031 Amendment 1 records that JSON and XML come from each port's standard
serializer for now. Excel (xlsx) is reserved by name only.
… and generator counts. Export contract, ports, ADR-0031 Amendment 1, roadmap and changelog already match code at target. Edited four files: docs/CONFORMANCE.md, docs/features/api-contract.md, docs/ports/kotlin.md, README.md. No code or tests touched.
@dmealing
dmealing merged commit de4f9f6 into main Oct 28, 2026
1 check passed
@dmealing
dmealing deleted the fm/mo-report-export branch October 28, 2026 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant