Conversation
…hema
- SchemaField.pack/unpack handed a nested schema a context of only
{'__packet__': packet}, so a callback written the ordinary way --
length=lambda pkt: pkt['length'] -- raised KeyError as soon as a schema
was nested. Measured casualty: CGA Parameters (mh.py
CGAParameter.extensions), unparsable via the public API.
- Fix: nested_packet_context() replaces that literal with a two-level
collections.ChainMap, so a name absent locally falls through to the
enclosing schema, while __packet__ still reaches it explicitly.
ChainMap.__setitem__/__delitem__ always act on the nested map, so a
write never reaches the parent and a shadowed name is read locally.
- mh.py needed no change: CGAParameter.extensions's pkt['length'] now
resolves via the fallback. CGA Parameters still does not parse -- it
now reaches FieldValueError: Field parameters has invalid length
(#446, ForwardMatchField vs. Schema.__len__), which is not fixed here.
- pcapng.py's two __packet__ consumers are untouched: real callers hand
them a plain {'__packet__': {...}} dict rather than going through
SchemaField, so their hand-rolled fallback still has to handle that
shape and is not redundant with this change.
- Documented the __packet__ contract in nested_packet_context, and added
it to Schema.unpack's reserved-key list alongside __length__ and
__option_padding__.
- The same latent defect affected six HTTP/2 frame schemas (reading the
header's 'flags' the same way); updated
tests/protocols/test_option_roundtrip_unit.py's EXPECTED_FAILURES:
removed PUSH_PROMISE and PING (now round-trip cleanly), and
re-attributed Multi_Prefix, DATA, HEADERS, CONTINUATION and SETTINGS to
the distinct defects this fix newly exposes underneath.
Ran PYTHONSAFEPATH=1 python -m pytest tests -q at baseline e2d8ed6
(960 passed, 17 skipped, 1264 subtests) and after (see PR description).
33c7413 to
46fe59a
Compare
|
Reviewing on behalf of Copilot (out of tokens). Head CISettled at 19 SUCCESS / 2 FAILURE / 2 SKIPPED:
The design: chained lookup via
|
- The previous NestedPacketContext used collections.ChainMap directly. ChainMap is itself a collections.abc.MutableMapping, and constructing one was enough to disturb the shared _abc_impl cache that every Schema subclass uses on CPython <= 3.10 (#439): a later, unrelated isinstance(some_dict, Schema) check in test_pcapng_remaining_constructor_branches_and_custom_dispatch flipped from False to True, raising AttributeError: 'dict' object has no attribute 'to_dict' from Schema.to_dict (schema.py:520). - Measured directly: with everything else unchanged, reverting only the ChainMap call back to a plain {'__packet__': packet} literal makes that test pass again on Python 3.10; restoring it reproduces the failure. Confirmed both in an isolated single-test run and via a local Python 3.10 venv (943 passed, 0 failed after this commit, against a failure before it). - Fix: NestedPacketContext is now a plain class implementing __getitem__/__setitem__/__delitem__/__contains__/__iter__/__len__/ get/update/copy/keys/values/items by hand, with none of it deriving from collections.abc. Same two-level fallback, same __packet__ reachability, same write isolation as before -- only the mechanism changes, not the contract. All existing tests pass unchanged. - Not a fix to #439 itself, which remains filed and untouched; this only stops this PR's own code from being the thing that trips it. Verified: PYTHONSAFEPATH=1 pytest tests -q on Python 3.14 (981 passed, 17 skipped, 1267 subtests, 0 failed) and on a local Python 3.10 venv (943 passed, 55 skipped, 1237 subtests, 0 failed -- the skip count differs only because that venv lacks some optional runtime deps).
…ation - NestedPacketContext was a hand-written class implementing the mapping protocol from scratch, deriving from nothing -- correct, but it left Schema.pack/unpack's "packet: dict[str, Any]" annotation unsatisfied, which is what every other field class's pack/unpack still declares. Widening those annotations to admit the new type cascades through every field class that forwards packet along (ListField, ConditionalField, FieldBase.__call__, pre_process/post_process, ...), well outside this fix's file list, for seven new mypy errors net. - Fix: NestedPacketContext now subclasses dict directly. dict's own metaclass is plain "type", not ABCMeta, so subclassing or instantiating it never touches the shared _abc_impl cache that #439 is about -- confirmed again on the Python 3.10 venv (this test passes both alone and in the full pcapng module: 37 passed, 139 subtests). Being a real dict also nominally satisfies every existing "dict[str, Any]" annotation, so no other file needs to change. - __missing__ gives the enclosing-schema fallback for free, since dict.__getitem__ calls it automatically when a key is absent locally. - __contains__ and get are overridden because the dict built-ins for both bypass __missing__ entirely. - __iter__/__len__/keys/values/items are overridden for the union-of-both- levels semantics the design promises; dict's own versions would only see this instance's local keys. - copy is overridden because dict.copy() always returns a plain dict, even for a subclass, which would silently drop the fallback. - Plain assignment/deletion need no override: dict's own __setitem__/ __delitem__ already only touch this instance's own storage. - Removed the now-genuinely-unused "type: ignore[has-type]" this rewrite exposed on SchemaField.length -- confirmed present and already-unused on a clean origin/main (da24227) checkout too, so this is a pre-existing, unrelated mypy hygiene gap this rewrite happened to touch, not something introduced by it. Net mypy count: 123 (down from main's own 124), with no new errors from this branch's own code. - Fixed a stale citation: the EOFError-on-zero-length Gap entries pointed at decorators.py:222; the actual "raise EOFError" is at :228 (prepare itself starts at :177, and #450 shifted lines since the citation was written). Verified: mypy pcapkit -> 123 errors/40 files (main: 124; the diff is the one pre-existing unused-ignore above, not a new error). pytest tests -q unaffected on Python 3.14. Local Python 3.10 venv: test_pcapng_remaining_constructor_branches_and_custom_dispatch passes alone and as part of the full misc/test_pcapng_unit.py module.
…-context # Conflicts: # tests/protocols/test_option_roundtrip_unit.py
- main merged #437 (MH registry completion, including the four mh-extension codes and the _make_ext_multiprefix arithmetic fix) and #456/#446 (the ForwardMatchField double-count in Schema.__len__) since this branch's last merge. Combined with this PR's own fix, all four mh-extension/{Multi_Prefix,Exp_FFFD,Exp_FFFE,Exp_FFFF} cases now round-trip cleanly -- verified directly against the round-trip harness (all four return 'OK'), not assumed from the PR descriptions. Deleted their EXPECTED_FAILURES entries; a stale PARSE/KeyError expectation would otherwise have failed this module outright, per its own two-way assertion. - The issue's own 40-octet CGA Parameters reproduction now parses completely end to end (confirmed directly: MH(raw, len(raw), extension=True) returns a populated CGAParametersOption, no exception). Rewrote test_cga_parameters_option_reaches_the_446_boundary _not_a_keyerror, which asserted the (now stale) FieldValueError boundary, as test_cga_parameters_option_now_parses_end_to_end, asserting the parsed fields directly. - #437 had pinned the pre-fix KeyError as test_mh_cga_parameters_option_is_unparsable_upstream, explicitly so that "whoever fixes it finds out here" -- and it did: this run turned that test red once the merge above landed. Replaced it with test_mh_cga_parameters_option_now_parses, asserting the option parses and its fields are what the wire says, and fixed the now-stale cross-reference and claim in test_mh_pmipv6_options_round_trip_byte_for_byte's docstring (CGA_Parameters is still excluded from that test's cases, but no longer because it cannot be parsed -- that is now a separate, deliberate scope decision for whoever adds its full round-trip identity). - Merged origin/main (0283a6d) with one conflict, in this exact region of tests/protocols/test_option_roundtrip_unit.py, resolved by re-deriving the correct entries from the actual post-merge behaviour rather than picking either side. Verified: mypy pcapkit -> 123 errors/40 files (a fresh main, 0283a6d, is 124 -- unchanged from before this merge). Round-trip harness: 7 passed, 363 subtests passed, 0 failed (up from 299 subtests before #437 grew the mh-extension family to four codes). tests/protocols/ internet/test_mh_unit.py: 35 passed, 266 subtests passed, 0 failed. Full local suite result to follow in the PR description.
| return self._field.unpack(buffer, packet) | ||
|
|
||
|
|
||
| class NestedPacketContext(dict): |
There was a problem hiding this comment.
what about use an Info subclass? and im not sure why must we use a dedicated class for this.
There was a problem hiding this comment.
Measured rather than argued from the abc angle (that angle turned out to be a
dead end -- see below), by actually building an Info-based nested context
and running it: it does not break Python 3.10. Swapped NestedPacketContext
for an Info subclass on disk, ran it on a local Python 3.10.21 venv (matching
CI's exact patch), and tests/protocols/misc/test_pcapng_unit.py -- the module
that broke under the ChainMap version -- came back clean: 37 passed, 139
subtests passed, 0 failed, both for the one previously-failing test alone and
for the whole module.
So the earlier "avoids a Mapping tie" reasoning I gave for the dict choice
doesn't hold up, and I'm not using it any more: Info is itself Mapping-based
and gets its own _abc_impl regardless (its metaclass has no CPython-version
bypass), so a Mapping tie alone was never the risk. I'm not citing a
mechanism I haven't personally re-verified, so I'll leave the actual cache
mechanics to whoever measured that -- what I can say directly is that Info
passes the test that mattered.
That still leaves the real question: not "is it safe" but "does it fit". I
built the Info version to answer this honestly rather than guess, and here
is what it would and wouldn't give for free:
__missing__fallback: no saving.Info.__getitem__is
self.__dict__[self.__map__.get(name, name)]-- a from-scratch
implementation with no__missing__-style hook (that's adictC-level
feature, not a generalMappingone). Whether I subclassdictorInfo,
I have to write the fallback lookup myself; there's no version whereInfo
saves me this code.__contains__/get: a genuine point inInfo's favour.Mapping
supplies mixin implementations of both that delegate to__getitem__, so
once__getitem__has the fallback,inand.get()inherit it for free.
dict's own__contains__/getare C-level and bypass__missing__
entirely, so myNestedPacketContext(dict)has to override both by hand.
Infowould save that code.- Writes must land on the nested instance only, never on the enclosing
schema -- this is where it breaks down.Infois deliberately immutable:
__setattr__raisesUnsupportedCall, and it has no__setitem__at all
(it inherits read-onlyMapping, notMutableMapping). Field callbacks
write into the packet dict constantly during parsing --
Schema.unpack's own per-field loop doespacket[field.name] = valueonce
per field, for every field of every nested schema. MyInfoscratch class
only supports this by writing intoself.__dict__directly from a custom
__setitem__, bypassing the immutabilityInfo's own docstring promises
("Info objects are immutable, thus cannot set or delete attributes after
initialisation"). It works, but it's a subclass that quietly defeats the
guarantee the class exists to provide -- not an extension ofInfo's
contract, a contradiction of it. - Extra bookkeeping to filter out:
Infoinstances carry__map__and
__map_reverse__inself.__dict__for its builtin-name-collision
handling, which then show up in naive iteration alongside the real packet
keys (I hit this directly --set(packet.keys())came back with_parent,
__map__and__map_reverse__mixed in with the actual field names, and
I'd have needed to filter them the wayInfo.__iter__filters
self.__excluded__). A plaindictsubclass has no such baggage: its own
storage holds only what's explicitly put there. - Purpose mismatch, not just mechanics: every real
Infosubclass in this
codebase declares a fixed, type-annotated field set and is built once as a
stable snapshot (that's whatinfo_final/__new__are for). A nested
packet context is the opposite: one shared, generic type instantiated fresh
per parse, holding whatever field names that schema happens to declare,
mutated field-by-field as parsing proceeds. Reusing the bareInfoclass
works because it happens to accept arbitrary kwargs, not because that's
what it's for.
So: two of the four things this needs (__contains__/get) Info gives for
free; one (__missing__) is a wash; and the write-isolation requirement is a
real contract conflict, not just extra code, because Info advertises
immutability as a feature and a nested context needs exactly the opposite.
That's why I kept the small dedicated class rather than switching -- "least
code that gets the contract right without fighting another class's
guarantees," not "because that's how I already did it." If the immutability
conflict is judged acceptable to paper over (my scratch class shows it's
mechanically possible), I'll switch; I don't think it should be, given the
class's own docstring says the opposite of what the subclass would then do.
PR #462 merged (bind HTTP.make to a real instance, and wrap SETTINGS' item schema, closing #459) since this branch's last merge, and pulled in via the fast-forward to 0e7abbe. SettingsFrame.settings now wraps its item_type in SchemaField(schema=SettingPair) instead of passing the raw class, so the AttributeError this entry recorded no longer happens. Verified directly: tests/protocols/test_option_roundtrip_unit.py's httpv2-frame/SETTINGS case now returns 'OK'. This is what failed CI on Python 3.12 and Integration Python 3.15 at 0e7abbe -- not an interpreter-dependent or order-dependent failure, the same stale-entry mismatch on every interpreter; those two jobs simply reported first. Verified: mypy pcapkit -> 123 errors/40 files (unchanged). Round-trip harness, this file's own tests, and tests/protocols/internet/ test_mh_unit.py: 46 passed, 629 subtests passed, 0 failed.
main gained #461 (closing #458: prepare's @prepare decorator now distinguishes a declared zero length from a derived one, raising StreamEOFError only for the latter) since this branch's previous merge, on top of #462 (closing #459, already handled). Between the two, all three remaining httpv2-frame entries this PR's own fix had exposed -- DATA, HEADERS, CONTINUATION -- now round-trip cleanly too. Verified directly against the round-trip harness: all three return 'OK'. Rewrote the HTTP/2 section's comment block to summarise all six frames' history (PUSH_PROMISE/PING via #445 itself, DATA/HEADERS/ CONTINUATION via #461, SETTINGS via #462) now that none of them need an entry. Also merged origin/main (fa12895, #461) -- clean auto-merge, no conflicts, confirmed against #464's changes to tests/protocols/internet/test_mh_unit.py (different hunks, and its own test run clean: 62 passed, 272 subtests, 0 failed). Verified: mypy pcapkit -> 124 errors/40 files (main at fa12895: 125, one more than its previous 124 -- unrelated to this branch, and this branch stays one fewer than whatever main's own count is, from the same pre-existing type: ignore cleanup as before). Round-trip harness: 7 passed, 363 subtests passed, 0 failed. mh-extension/* re-verified 'OK' again on this merge (all four).
|
Reviewing on behalf of Copilot (out of tokens). This supersedes the stale CI
Full suite and mypy, run myself, not taken from the PR bodyGenerated fixtures with
The six
|
Summary
Closes #445.
A nested schema's field callbacks got a packet dict holding the enclosing
schema only under
__packet__, so a callback written the ordinary way --length=lambda pkt: pkt['length'], exactly as every top-level schema writesit -- raised
KeyErrorthe moment the schema was nested. The measuredcasualty is CGA Parameters:
CGAParameter.extensions(
pcapkit/protocols/schema/internet/mh.py:515-521) sizes itself frompkt['length'], which belongs to the enclosingCGAParametersOption, and awell-formed 40-octet option raised
KeyError: 'length'inSchemaField.unpack(pcapkit/corekit/fields/misc.py:619, the{'__packet__': packet}literal).Design chosen: chained lookup (option 1 of the three in the issue)
nested_packet_context()(pcapkit/corekit/fields/misc.py) replaces theliteral with a
NestedPacketContext, adictsubclass (see "A Python3.10-only regression" below for why it is a
dictsubclass rather thancollections.ChainMap, which is where this started). A name the nestedschema does not declare falls through to the enclosing schema; a name it
does declare, or
__packet__itself, is found locally first. This designwas chosen over the other two because:
callback site, including ones outside this PR's scope, to get the same
fix
mh.py's CGA Parameters needed. The chained lookup needed zerochanges to
mh.py:CGAParameter.extensions's existingpkt['length']just resolves once the mapping falls through. Verifiedby reproducing the issue's exact 40-octet packet before and after.
names: it lets an inner field silently shadow an outer one. The mapping
keeps them apart -- see the shadowing test below.
__packet__contract had exactly one piece of documentation (adocstring on an unrelated pcapng.py helper) and zero mentions in
Schema.unpack's own reserved-key list. Both are fixed: the contract isnow documented on
nested_packet_contextitself, andSchema.unpack'sdocstring names
__packet__alongside__length__and__option_padding__.Write path (explicitly checked, since a mapping that writes through or
drops writes would be worse than today): plain
dictassignment anddeletion (
pkt[key] = value,del pkt[key]) always act on this instance'sown storage -- that is what a
dictsubclass gives for free, with nooverride needed. So a field a nested schema sets -- including one that
shadows a parent field name -- is never visible to the parent, and never
silently dropped either.
inand.get()are overridden (thedictbuilt-ins for both bypass
__missing__and would otherwise miss thefallback entirely), and so are
__iter__/__len__/keys/values/items,for the union-of-both-levels iteration the design promises. All of this is
covered by
test_nested_schema_reads_enclosing_field_by_name_and_does_not_leak_writes.pcapkit/protocols/schema/misc/pcapng.py's two existing__packet__consumers (
packet_byteorder,BlockType.post_process) are untouched.Both are called directly, in tests and in
pcapkit/foundation/engines/pcapng.py:192, with a plain{'__packet__': {...}}dict rather than throughSchemaField, so theirhand-rolled fallback has to keep handling that shape regardless of this
design -- simplifying them to rely on the chain would have broken those
direct callers. Covered by
test_pcapng_byteorder_consumer_still_works_with_both_shapesandtest_pcapng_block_type_mismatch_consumer_still_works_with_both_shapes,each exercising both the hand-built dict and
NestedPacketContext.A Python 3.10-only regression, and why
NestedPacketContextis adictsubclassThe first pushed version of this PR used
collections.ChainMap({'__packet__': packet}, packet)directly, and CI went red on exactly two legs:Python 3.10andIntegration Python 3.10, both ontests/protocols/misc/test_pcapng_unit.py::PCAPNGUnitTests::test_pcapng_remaining_constructor_branches_and_custom_dispatch,with
AttributeError: 'dict' object has no attribute 'to_dict'fromSchema.to_dict(schema.py:520,isinstance(value, Schema)wrongly truefor a plain
dict).That test has nothing to do with CGA Parameters, HTTP/2, or MH -- it
exercises fifteen unrelated PCAP-NG option constructors. The actual cause,
confirmed with a local Python 3.10 venv:
isolation (no other test file loaded, so none of this PR's new dynamic
Schemasubclasses are even created) as soon as this PR'smisc.pychange alone is applied.
collections.ChainMap(...)call back to a plain{'__packet__': packet}literal -- nothing else changed -- makes it passagain. Restoring the
ChainMapcall reproduces the failure.Diagnosis (full version posted on #439):
Schemainheritscollections.abc.Mapping(schema.py:245), and on CPython <= 3.10SchemaMeta.__new__bypassesABCMeta.__new__, so noSchemasubclassever gets its own
_abc_impl-- every one of them sharesSchema's.Asking
Mappinga question before askingSchemaone therefore poisonsthe cache for the whole family.
collections.ChainMapis itself acollections.abc.MutableMapping, and constructing one pulledMappinginto the question order, flipping the unrelated, later
isinstance(some_dict, Schema)check in the PCAP-NG test fromFalsetoTrue. This is not a fault in the chained-lookup design; it is #439 (notthis PR's to fix) surfacing through an implementation detail of this PR's
own code.
Because the failing test is a plain
unittestmethod, not one oftest_option_roundtrip_unit.py's per-code cases,INTERPRETER_GAPScannotexpress it -- that table only overrides a
case.labellookup, and there isno table to add this to. Rather than leave a real, reproducible CI failure
in place,
NestedPacketContextnow subclassesdictdirectly instead ofChainMap/Mapping.dict's own metaclass is plaintype, notABCMeta-- constructing or using adictsubclass never asksMappinganything, so the shared cache is never touched. Being a real
dictalsonominally satisfies every existing
packet: 'dict[str, Any]'annotation onthe rest of the field classes, so no other file needed to change (see the
mypy section below for why that mattered).
__missing__gives the enclosing-schema fallback for free:dict's own__getitem__calls it automatically the moment a key is not foundlocally.
__contains__andgetare overridden, since thedictbuilt-ins forboth bypass
__missing__entirely and would otherwise never fallthrough.
__iter__/__len__/keys/values/itemsare overridden for theunion-of-both-levels iteration the design promises;
dict's own versionswould only see this instance's local keys.
copyis overridden becausedict.copy()always returns a plaindict,even for a subclass, which would silently drop the fallback.
dict's own__setitem__/__delitem__already only touch this instance's ownstorage, which is exactly the write isolation the design requires.
Same fallback, same
__packet__reachability, same write isolation as theChainMapversion -- confirmed by the unchanged test suite (allpre-existing tests pass unmodified). Verified after the change: the Python
3.10 venv runs
tests/protocols/misc/test_pcapng_unit.pyclean (37 passed,139 subtests, 0 failed), and the previously-failing test passes both alone
and as part of that module.
mypy: two errors, from the first
NestedPacketContext, now zero netAn intermediate version of this fix (after moving off
ChainMap, beforesettling on
dict) implemented the same two-level mapping from a bareobject deriving from nothing at all. That is a legitimate way to avoid
collections.abcentirely, but it does not satisfy the existingpacket: 'dict[str, Any]'annotationSchema.pack/unpackdeclare, somypy flagged
Argument 1 to "pack" of "Schema" has incompatible type "NestedPacketContext"; expected "dict[str, Any] | None", plus anSchemaField.lengthtype: ignore[has-type]that the restructuring madenewly unused.
Widening
Schema.pack/unpack's annotation to admit the new type directlywas the obvious fix and the wrong one:
packetflows from there intopre_unpack/pre_pack/post_process,FieldBase.__call__,ListField.pack,ConditionalField.testand more, each with its ownnarrowly-typed
packet: 'dict[str, Any]'signature. Widening only the twoSchemamethods produced seven new errors at those call sites; doing itproperly would mean widening every field class's own signature, well
outside this PR's file list.
Making
NestedPacketContextadictsubclass (previous section) sidestepsthis too: it satisfies the existing annotation nominally, everywhere,
because it is one. The
type: ignore[has-type]was simply deleted ratherthan replaced -- and checked: it is already flagged unused on a clean
da2422728checkout (no changes at all), so it is a pre-existing,unrelated mypy hygiene gap this restructuring happened to touch, not
something this PR introduced.
mypy pcapkiton this PR's head: 123errors in 40 files (
main: 124) -- one fewer than baseline, and no newerror anywhere.
One stale citation
The
httpv2-frame/{DATA,HEADERS,CONTINUATION}Gap entries'defectstringpointed at
pcapkit/utilities/decorators.py:222; the actualraise EOFErroris at:228(prepareitself starts at:177). Corrected.#437's three
Exp_FFFD/Exp_FFFE/Exp_FFFFfailuresYes, same defect -- and #437 has since merged (
489eef651), which let thisbe checked directly rather than by reading its diff.
ExperimentalExtensiondeclares
data: 'bytes' = BytesField(length=lambda pkt: pkt['length']), butlengththere isCGAExtension's own field (parsed locally, no nestingproblem). The
KeyErrorthose three cases hit happened beforeExperimentalExtensionwas even selected:CGAParameter.extensions'sOptionFieldhas to size the whole extensions area first, via the samepkt['length']this PR fixes, regardless of which extension type ends upinside it. #437 itself registered all three codes and attributed all four
mh-extension/*entries (includingMulti_Prefix) to #445 with exactlythat reasoning. See "Second merge" below for what happens to those four
entries once this branch also carries #437 and #446/#456.
CGA Parameters:
#446merged too, so it now parses end to endThis PR alone does not unblock CGA Parameters: with only #445 fixed, it
reaches
FieldValueError: Field parameters has invalid length.-- theseparate, already-filed
ForwardMatchField/Schema.__len__defect (#446)-- instead of the
KeyError. That was true when this PR was first opened.#446 has since merged as #456 (
0283a6d59), and with both fixes on thesame tree, the issue's own 40-octet reproduction parses completely: no
exception, a populated
CGAParametersOptionwith oneCGAParameter. See"Second merge" below for the test updates this required.
EXPECTED_FAILURESfallout in the round-trip harnessFixing the
KeyErrorunblocks the same latent defect in six HTTP/2 frameschemas (they read the header's
flagsthe same way, on the pack side).Running
tests/protocols/test_option_roundtrip_unit.pyafter the fixturned seven cases red against the old table, each hitting a distinct,
unrelated, previously-unreachable defect underneath:
httpv2-frame/PUSH_PROMISE,httpv2-frame/PING: now round-trip cleanly-- entries deleted.
httpv2-frame/DATA,HEADERS,CONTINUATION: nowEOFError--pcapkit/utilities/decorators.py:222'sprepareunconditionally treatsa zero-length nested unpack as end-of-file, which is wrong for a frame
that legitimately has no payload. Not fixed here (that file is owned by
open PR utilities: let @prepare read length/packet by keyword, not position #450).
httpv2-frame/SETTINGS: nowAttributeError: 'SettingPair' object has no attribute 'length'--SettingsFrame.settings(
pcapkit/protocols/schema/application/httpv2.py:283-285) passes theSettingPairclass toListField'sitem_typeinstead of wrapping itin a
SchemaField, silenced by a# type: ignore[arg-type]. Filed asSettingsFrame.settings passes a Schema class to ListField's item_type instead of a SchemaField #459; not fixed here.
mh-extension/Multi_Prefix: at the time, nowFieldValueError: Field prefixes has invalid length--_make_ext_multiprefix(
pcapkit/protocols/internet/mh.py:3864) wrotelength=1 + len(prefixes) * 16(17 octets for one prefix) instead of4 + len(prefixes) * 8(12). Not a new defect: open PR protocols: complete the Mobility Header registry -- all 24 message types, 70 of 71 options, all 4 CGA extensions #437 alreadyfixed exactly this. protocols: complete the Mobility Header registry -- all 24 message types, 70 of 71 options, all 4 CGA extensions #437 has since merged -- see "Second merge" below for
where this entry ends up.
Two more defects surfaced the same way, filed rather than fixed here:
#458 (
@prepareraises a bareEOFErrorfor any zero-length schema,decorators.py:227-228-- the mechanism behind theDATA/HEADERS/CONTINUATIONentries above) and #459 (above). Both were open whenthis was written and have since merged -- see "Third merge" below.
All entries re-attributed with the new status, fragment and
file:line,verified against the actual exception text.
Second merge:
maingained #437 and #446/#456 mid-reviewmainmoved again while this PR was in review -- #437 (MH registrycompletion) and #456 (issue #446, the
ForwardMatchFielddouble-count)both merged. Merged
origin/main(0283a6d59) a second time, oneconflict in the same
mh-extensionregion ofEXPECTED_FAILURES(resolved by re-deriving each entry from the actual post-merge behaviour,
not by picking a side):
mh-extension/{Exp_FFFD,Exp_FFFE,Exp_FFFF}-- alongsideMulti_Prefix,all four attributed to A nested schema cannot reach the enclosing packet's fields by name, so CGA Parameters raises KeyError: 'length' #445 with the identical
KeyError: 'length'.Ran each of the four individually against the merged tree rather than
assuming they behave alike: all four now return
'OK'. protocols: complete the Mobility Header registry -- all 24 message types, 70 of 71 options, all 4 CGA extensions #437 bothregistered the three experimental codes and fixed
_make_ext_multiprefix's bogus arithmetic; schema: a forward match consumes nothing, so bill it nothing in len(schema) #456 fixed theForwardMatchFielddouble-count that stoppedCGAParametersOption.parametersfrom sizing correctly. Combined withthis PR's own fix, nothing blocks any of the four any more. All four
entries deleted.
above).
test_cga_parameters_option_reaches_the_446_boundary_not_a_keyerror,which asserted the now-stale
FieldValueErrorboundary, is rewritten astest_cga_parameters_option_now_parses_end_to_end, asserting the parsedfields directly.
KeyErrorastests/protocols/internet/test_mh_unit.py::test_mh_cga_parameters_option_is_unparsable_upstream,explicitly so that "whoever fixes it finds out here" -- and it did: that
test went red the moment this merge landed, for exactly the reason its
own docstring named. Replaced it with
test_mh_cga_parameters_option_now_parses, asserting the option parseswith the fields the wire says it should have, and corrected the now-false
claim in
test_mh_pmipv6_options_round_trip_byte_for_byte's docstring(CGA_Parameters is still excluded from that test's round-trip cases, but
no longer because it cannot be parsed -- that is a separate, deliberate
scope decision for whoever adds it, not implied by parsing alone).
Re-verified after this second merge:
mypy pcapkit-> 123 errors/40files (a fresh
origin/mainat0283a6d59: 124, unchanged from thefirst merge's baseline). Round-trip harness: 7 passed, 363 subtests
passed, 0 failed (up from 299 before #437 grew the
mh-extensionfamilyto four codes).
tests/protocols/internet/test_mh_unit.py: 35 passed, 266subtests passed, 0 failed. Full suite (Python 3.14,
PYTHONSAFEPATH=1 python -m pytest tests -q): 1001 passed, 17 skipped,1547 subtests passed, 0 failed (883s).
NestedPacketContext: from a hand-written mapping to adictsubclass, and back to semanticsThe
ChainMap-vs-dictdiagnosis above turned out to be only half right onreview.
#462merged mid-review and, independently, an owner review threadasked why
nested_packet_context()doesn't just reuseInfo(
pcapkit/corekit/infoclass.py) instead of a dedicated class. Rather thanargue from the (now-corrected) ABC-cache mechanism, I built an actual
Info-based nested context on disk and ran it: on a local Python 3.10.21venv (matching CI's exact patch),
tests/protocols/misc/test_pcapng_unit.py-- the module that broke under
ChainMap-- came back clean, 37 passed,139 subtests, 0 failed. So
Infois not unsafe here; the real question issemantics, not the ABC cache:
__missing__-style fall-through: no saving either way.Info.__getitem__is a from-scratch
self.__dict__[...]lookup with no such hook (that's adictC-level feature), so it needs the same custom code whichever baseis used.
__contains__/.get(): a genuine point forInfo--Mappingsuppliesmixins for both that delegate to
__getitem__, so fixing__getitem__once gets both for free, where
dict's own versions bypass__missing__and need separate overrides.
Infoisdeliberately immutable (
__setattr__raises, no__setitem__at all),but field callbacks write into the packet dict throughout parsing
(
Schema.unpack's per-field loop:packet[field.name] = value, once perfield). Supporting that on an
Infosubclass means writing intoself.__dict__directly from a custom__setitem__, bypassing ratherthan extending the immutability
Info's own docstring promises.Infoinstances carry__map__/__map_reverse__inself.__dict__(for its builtin-name-collisionhandling), which then show up in naive iteration alongside the real
packet keys -- measured directly, not assumed.
Full reasoning posted on the owner's thread
(#457 (comment));
left unresolved for the owner to close.
Third merge:
maingained #461 (closing #458) and #462 (closing #459)Two more of the newly-surfaced defects this PR had filed (#458, #459) were
fixed and merged while this was in review, as
#461and#462. Mergedorigin/main(fa128959e, thene7004191aafter reconciling with aduplicate parallel merge already pushed to this branch) -- no conflicts;
#464's changes totests/protocols/internet/test_mh_unit.pyland in adifferent region and were confirmed non-interacting by running that file
(62 passed, 272 subtests, 0 failed).
#461makes@preparedistinguish a declared zero length (a nestedschema legitimately sized to zero) from a derived one (genuine
end-of-stream), raising only for the latter. Verified directly:
httpv2-frame/{DATA,HEADERS,CONTINUATION}all now return'OK'.#462wrapsSettingsFrame.settings's item type inSchemaField(schema=SettingPair). Verified directly:httpv2-frame/SETTINGSnow returns'OK'too.All six
httpv2-frameentries this PR's own fix had exposed are now gone:PUSH_PROMISE/PINGclosed by #445 itself,DATA/HEADERS/CONTINUATIONby #461,
SETTINGSby #462. Rewrote the section comment to summarise allsix rather than describe five stale gaps.
Final numbers, at this branch's current head:
mypy pcapkit-> 124errors/40 files (a fresh
origin/mainatfa128959e: 125 -- one morethan its own earlier count, unrelated to this branch, and this branch
stays one fewer than whatever
main's own count is, from the samepre-existing
type: ignorecleanup as before). Round-trip harness: 7passed, 363 subtests, 0 failed. Full suite (Python 3.14): 1010 passed,
17 skipped, 1553 subtests passed, 0 failed (906s). CI is green: all
22 checks pass (2 skip by design -- the docs gate and the single-version
full-suite gate).
Testing
New file
tests/corekit/test_fields_misc_packet_context.py:test_nested_schema_reads_enclosing_field_by_name_and_does_not_leak_writes-- the load-bearing case. Before:
KeyError: 'length'(matches theissue exactly). After: passes, and also checks shadowing, the write
path,
in,.get()and iteration in one pass.test_pcapng_byteorder_consumer_still_works_with_both_shapes,test_pcapng_block_type_mismatch_consumer_still_works_with_both_shapes-- the two existing consumers, with a hand-built dict and with the new
chain. Before:
ImportError(the helper does not exist yet).After: pass.
test_cga_parameters_option_reaches_the_446_boundary_not_a_keyerror--the issue's own 40-octet reproduction. Before:
KeyError: 'length'at
mh.py:516. After:FieldValueError, confirming the fix workedand CGA Parameters correctly still does not parse.
Baseline at
e2d8ed6d1(this branch's original merge-base, confirmed bytemporarily reverting the changed files back to that commit's content in
this worktree, not quoted second-hand):
PYTHONSAFEPATH=1 python -m pytest tests -q-> 960 passed, 17 skipped, 1264 subtests passed (617s). Afterthis PR's original commit, same command: 964 passed, 17 skipped, 1264
subtests passed, 0 failed (605s).
mainthen moved four commits (#449, #450, #451, #453) while this PR was inreview, so it was merged (
git merge --no-ff origin/main, one cleanauto-merge in the
EXPECTED_FAILUREStable -- see the heads-up above) andre-baselined the same way, at the new merge-base
da2422728: 977 passed,17 skipped, 1267 subtests passed (609s). After this PR's changes on top
(including the
NestedPacketContextfix described above): 981 passed, 17skipped, 1267 subtests passed, 0 failed (638s) -- the same 4 new tests
join the passing count; the subtest total is unchanged from the new
baseline (main's own commits added tests of their own, which is why 1267
differs from the original 1264).
Also verified on a local Python 3.10 venv (this repo's CI runs 3.10-3.15;
only 3.10 carries #439's risk): 943 passed, 55 skipped, 1237 subtests
passed, 0 failed -- the higher skip count is only this venv missing some
optional runtime deps (dpkt/pyshark extras), not a difference in outcome.
Test plan
pcapng.py__packet__consumers verified unaffectedFieldValueError(ForwardMatchField's non-consuming bytes count toward Schema.__len__, so correct input fails a declared-length check #446), not theKeyError#437's threeExp_FFF*failures confirmed to be this same defecttests/protocols/test_option_roundtrip_unit.py) green,EXPECTED_FAILURESupdated for all 7 cases whose status changedtest_pcapng_unit.py, and a local full-suite run) confirmed clean after moving offChainMapmypy pcapkitconfirmed at 123 errors/40 files on this PR's head, against 124 onmain-- no new error, one pre-existing one incidentally cleaned up