[alphasafal]

Issue number: closes #8665

Summary

Changes

Confluent's Protobuf serializer writes the message index array as zigzag encoded varints (array length, then each index), with [0] as a single 0x00 byte. ProtobufDeserializer._remove_message_index read it with _DecodeSignedVarint, so it read twice the real length and skipped payload bytes as if they were indexes. Records for any message type other than the first in the schema failed with DecodeError, or decoded with fields silently missing.

This change:

• reads the array length and each index with _DecodeVarint plus zigzag decoding ((n >> 1) ^ -(n & 1)), in a small _decode_zigzag_varint helper
• raises ValueError for a negative array length (wrapped as KafkaConsumerDeserializationError by deserialize)
• fixes the default without a flag, as agreed in the issue

Tests in tests/functional/kafka_consumer/_protobuf/test_kafka_consumer_with_protobuf.py:

• test_confluent_schema_registry_complex_schema now also asserts userId and name. The fixture record with the 04 02 00 prefix was losing them, and the test only checked email and age.
• new test_confluent_schema_registry_message_index_is_zigzag_decoded, parametrised over indexes [0], [1], [2], [1, 0] and [70] (a multi-byte index), framed the same way confluent-kafka's serializer frames them

The tightened test and four of the five new cases fail on develop ([0] already worked) and pass with this change. All Kafka tests pass (60), and ruff format / ruff check are clean.

User experience

Before, for a Confluent-framed record whose message type is not the first in its schema:

message index [1, 0]: KafkaConsumerDeserializationError: ... DecodeError: ... Wire format was corrupt
message index [2]: KafkaConsumerDeserializationError: ... DecodeError: ... Wire format was corrupt

and the repo's own fixture record with index [1, 0] decoded without userId and name.

After, every index decodes into all fields:

message index [0]: {'name': 'order_id', 'number': 1, 'json_name': 'orderId'}
message index [1, 0]: {'name': 'order_id', 'number': 1, 'json_name': 'orderId'}
message index [2]: {'name': 'order_id', 'number': 1, 'json_name': 'orderId'}

(output of the snippet in #8665)

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[leandrodamascena]

Thanks @alphasafal, and good catch, the fixture with [1, 0] was silently dropping two fields and nobody noticed. I checked your reading against confluent-kafka 2.16.1 itself: framed [1], [2], [1, 0], [70], [300], [0, 3, 1] and a few more with their encoder, all of them fail or lose fields on develop and decode every field with your change. I pushed one small test for the negative length path so the new raise is covered.

I'm approving now. This goes out in 3.40.0, so it will sit here for a while before I merge it. No rebase needed in the meantime, and you'll be credited in the release notes.