Conversation
Bundle mode embeds every local module in one container command-line argument. Linux rejects any single argument longer than MAX_ARG_STRLEN (128 KiB) with E2BIG, so a component whose bundle crosses that size fails at exec before any Python runs, and nothing warns at generation time. Encode the payload with bz2 instead of zlib, and Base85 instead of Base64. Both are in the standard library, so generated components gain no dependency. On a real 32-module, ~400 KB bundle, the argument drops from 132,638 bytes (over the limit) to 94,952 bytes, with identical module sources that decode and execute. Base85 contains no quotes or backslashes, so the blob stays safe inside the emitted Python string literal, and the program reaches the shell only as $0 written out by printf, so it is never shell-interpreted. Every generated component embeds its own decoder, so previously generated components keep working unchanged. Very small bundles grow by a few dozen bytes of bz2 header, which does not matter for the limit. The bundled_modules_b64 keyword keeps its name because callers pass it by keyword; only its documentation changes.
Jflick58
marked this pull request as ready for review
September 30, 2026 19:28
Collaborator
|
I suggest not modifying an existing bundling mode, but adding a new one. Base85 allows things like {{ and {%, which might break jinja templates used during hydration. So the choice of this highly efficient bundling should be optional. alternatively you might look into escaping these symbols safely, so that jinja doesn't break |
Volv-G
requested changes
Sep 30, 2026
Volv-G
left a comment
Volv-G
left a comment
Collaborator
There was a problem hiding this comment.
See the comment about jinja templates
Jflick58
changed the title
Author
|
Fixed. |
Volv-G
approved these changes
Oct 3, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this change?
Large bundled Python components can exceed Linux's per-argument size limit and fail with
argument list too longbefore Python starts. Keep the more compact format opt-in so existing callers retain zlib/Base64 behavior. Base85 can also contain Jinja delimiters, which need escaping before hydration.Summary of changes
bundleon zlib/Base64 and add opt-inbundle-bz2for bz2/Base85. The new mode works through component generation,@task, hydration, and version regeneration.{{,{%, and{#remain inert during Jinja hydration, including repeated rendering._bz2extension. Both modes still use one command-line argument, so this reduces payload size rather than removing the size limit.Testing
Automated Testing
Tophatting
Offline CLI generation, hydration, and generated-wrapper execution passed for both modes using a synthetic helper. Both printed
manual:value-510as expected. No remote pipeline runs were submitted.