Skip to content

Add PyWayang DataQuanta.collect() support - #874

Open
Kaileshwar16 wants to merge 3 commits into
apache:mainfrom
Kaileshwar16:feat/pywayang-collect
Open

Kaileshwar16 wants to merge 3 commits into
apache:mainfrom
Kaileshwar16:feat/pywayang-collect

Conversation

@Kaileshwar16

Copy link
Copy Markdown
Contributor

This PR follows up on #870 and implements the DataQuanta.collect() part of #869. It lets PyWayang users execute a pipeline and get the results directly as a Python list without writing to a text file. It preserves supported value types and nested structures, handles empty results, and reports backend errors while keeping existing text-file sinks and REST responses compatible. It also fixes collecting an earlier pipeline branch after another branch or sink is attached.

Four existing ML/DL tests remain skipped.

Return supported in-memory results through the existing JSON execution path,
preserving terminal Python values and legacy REST/text-sink behavior.

Refs apache#869; stacked on apache#870.

Generated-by: OpenAI Codex (GPT-6)
Derive serialized output connections from the selected plan's inputs so
collecting an earlier branch still works after another consumer is attached.
Add client, worker, Scala result conversion, and real backend regression
coverage, and document how to run the collection tests.

Refs apache#869; follow-up to apache#870.
@Kaileshwar16 Kaileshwar16 changed the title Feat/pywayang collect Add PyWayang DataQuanta.collect() support Nov 8, 2026
@zkaoudi
zkaoudi requested a review from juripetersen November 11, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant