[WIP][POC] Pfor encoding - #50088
Draft
prtkgaur wants to merge 78 commits into
Draft
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
Implements the PFOR (Patched Frame of Reference) integer compression algorithm as a standalone utility library in arrow/util/pfor/. Includes: - Cost model for optimal bit width selection (histogram-based) - Vector-level encode/decode with FOR + bit-packing + exceptions - Page-level wrapper with header, offset array, and multi-vector layout - Comprehensive unit tests covering edge cases and round-trips
Adds PFOR = 11 to the Encoding enum and wires it into the parquet read/write pipeline: - PforEncoder<DType> in encoder.cc (buffers values, calls PforWrapper::Encode) - PforDecoder<DType> in decoder.cc (decodes all values on first access) - PFOR case in column_reader.cc InitializeDataDecoder - Encoding string mapping in types.cc Supports INT32 and INT64 column types.
Benchmarks encode/decode throughput for int32/int64 across 10 data distributions inspired by Snowflake's NumericComprBenchmark: constant, sequential, small range, high-base-small-range (timestamps), with outliers (exception path), random, TPC-DS date/store/item/quantity keys. Each distribution runs at 1K/10K/100K/1M elements. Reports bytes/s, items/s, and compression ratio.
Load() now returns Result<PforVectorInfo> after the Status/Result refactoring. Use ASSERT_OK_AND_ASSIGN to properly unwrap the result in tests.
Make LoadHeader fallible: move the header-size check from Decode into LoadHeader, return Result<PforHeader>, and update Decode to use ARROW_ASSIGN_OR_RAISE. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Replace std::memcpy / raw byte writes in PforWrapper::StoreHeader, LoadHeader, and the offset-array read/write paths with util::SafeLoadAs and util::SafeStore. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Reject invalid packing_mode, value_byte_width mismatch, log_vector_size out of [kMin, kMax] range, and negative num_elements when loading the PFOR page header. Removes the redundant packing_mode and value_byte_width checks from Decode now that they live in LoadHeader. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…sites Replace size_t with int64_t for max_size/comp_size to match the PforWrapper API signature, and qualify pfor::PforWrapper as ::arrow::util::pfor::PforWrapper to avoid ADL ambiguity.
Aligns with Arrow buffer conventions (Buffer::data() returns uint8_t*). Removes the reinterpret_cast<char*> at the parquet encoder/decoder call sites and switches std::vector<char> compressed buffers to std::vector<uint8_t> in the unit test and benchmark. Also fixes a pre-existing size_t / int64_t* mismatch in pfor_benchmark.cc that surfaced once the buffer pointer type was tightened. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…th validation Per Google C++ style, replace the PforVectorInfo struct with a class that has private trailing-underscore members and getter/setter accessors. Replace std::memcpy calls in Store/Load and the exception patch loop in DecodeVector with util::SafeLoadAs / util::SafeStore. Add bit_width range validation inside Load() so callers don't have to repeat the check. Updates all access sites in pfor.cc and pfor_test.cc to go through the new accessors. Caches num_exceptions() in a local in DecodeVector so the #pragma GCC unroll can still see a constant loop bound. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Per Google C++ style, both types become classes with private trailing-underscore members and const getters, mutable getters, and setters. Updates all access sites in pfor.cc (EncodeVector, LoadView, SerializedVectorSize, SerializeVector) and pfor_test.cc to go through the new accessors. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…, use ctor in EncodeVector - Move the num_exceptions < 0 check from DecodeVector into PforVectorInfo::Load alongside the bit_width range check, so all loaded-data invariants are enforced at the same layer. - Use PforVectorInfo's parameterized constructor in EncodeVector instead of three separate setter calls on a default-constructed instance.
Commit 00b6318 introduced ARROW_DCHECK(bit_util::IsPowerOf2(vector_size)) in PforWrapper<T>::Encode, but vector_size is int32_t and bit_util has overloads only for int64_t and uint64_t -- the call is ambiguous and the file no longer compiles. Cast to int64_t to disambiguate. CeilDiv calls in the same file already promote to int64_t implicitly via its int64_t-only signature.
Portable C++ port of FastLanes (Afroozeh & Boncz, VLDB '23) for int32_t
columnar data. No SIMD intrinsics in the kernels — the inner lane loop
is structured (contiguous loads from packed[w*kLanes + lane], contiguous
stores to transposed[r*kLanes + lane]) so the compiler auto-vectorizes
to 4-wide NEON / 8-wide AVX2 / 16-wide AVX512 without source changes.
Layout: lane-interleaved 1024-bit format per the paper. 1024 values
pack as w u32 rows of 32 u32 lanes. FL_ORDER (8x16 -> 16x8 sub-block
transpose + 3-bit-reversal sub-block reorder) is applied OUTSIDE the
kernel: FastLanesForCodec::Encode gathers input[fromTransposed32(t)]
before packing; Decode produces output in transposed order (no scatter,
output[t] == input[fromTransposed32(t)] + min within each 1024-block).
FastLanesForCodec adds Frame-of-Reference on top:
- 2048-value chunks (2 FastLanes blocks per chunk)
- Per-chunk 5-byte header: [min(4B int32 LE)] [bit_width(1B)]
- Subtract min before packing; add back on decode
- bit_width=0 path stores no payload (constant chunk)
Files:
cpp/src/arrow/util/fastlanes/fastlanes_kernels.h
- PackBlock<W>(in, out) / UnpackBlock<W>(packed, out)
- W=32 fast path: std::memcpy
- fromTransposed32 helper
cpp/src/arrow/util/fastlanes/fastlanes_for.{h,cc}
- FastLanesForCodec::{Encode,Decode}
cpp/src/arrow/util/fastlanes/fastlanes_for_test.cc
- 5 round-trip tests (narrow range, single value, full int32 range,
multiple chunks, boundary values) — all passing
CMakeLists.txt wires the test as arrow-fastlanes-for-test.
Wires the new FastLanesForCodec into the existing pfor_comparison_benchmark harness alongside PFOR, DeltaBitPack, ZSTD, LZ4, RleBitPack, and Bss codecs. New BM_FastLanesEncode / BM_FastLanesDecode functions follow the same Gen32 + ::Apply(CustomArgs) shape; REGISTER_DATASET macro picks them up for every ClickBench dataset. Notes on the comparison: - FastLanes decoder produces output in TRANSPOSED order (output[chunk*2048 + block*1024 + t] == input[chunk*2048 + block*1024 + fromTransposed32(t)] + min). PFOR/DeltaBitPack produce flat output. The benchmark measures decoder throughput head-to-head; consumers of FastLanes output must be permutation-aware (which is the FastLanes paper's intended architecture). - num_values is rounded down to a multiple of 2048 (FastLanes chunk size) inside BM_FastLanesEncode / BM_FastLanesDecode for compatibility with the existing 102400-value test sizes. Also guards add_executable(parquet-pfor-comparison-benchmark) with if(ARROW_BUILD_BENCHMARKS) so non-benchmark configurations don't fail the cmake configure step. Bench numbers on aarch64 (102400 int32, 3-run median): EventDate decode: FastLanes 20us vs PFOR 36us vs Delta 122us EventTime decode: FastLanes 24us vs PFOR 56us vs Delta 140us GoodEvent decode: FastLanes 20us vs PFOR 34us vs Delta 119us Compression ratios match or slightly beat PFOR on every dataset tested.
`__restrict__` is a GNU spelling. MSVC accepts `__restrict` but not `__restrict__`, so the biased unpack_full would not compile on Windows -- and bpacking_dispatch_internal.h is pulled in by bpacking_scalar.cc, which every platform builds. ARROW_RESTRICT (arrow/util/macros.h) already spells this per compiler and is what the rest of the tree uses. The header already includes macros.h. The generated object is byte-identical on GCC, so the loop still vectorizes and the measurements stand.
C++20 implicit move makes the returned std::move redundant, and the PforWrapper::Decode Status was being dropped. Reflows the touched calls.
MakeEncoder and MakeDecoder already accept it, so the omission failed the SupportedEncodings consistency tests.
std::bit_width is 0 for a 0 input, so the explicit zero guards in BitsRequired go away too.
Also silences a cpplint false positive on the semicolon that closes a requires-expression initializer.
An element count over the output capacity, or an offset array or vector offset past the end of the page, no longer reads or writes out of bounds.
Every element of a maximum-size vector can be an exception, so the count reaches 32768, one past what the signed 16-bit field held.
Decode took its output first while Encode took its output last; both now read input, size, then output.
The preconditions were debug-only assertions, so a release build encoded with a vector size the page header cannot describe and said nothing.
Nullable INT32/INT64 columns raised NotImplemented; the encoder already writes non-null values densely, so spread them over the valid runs.
Both sides buffer a whole page today; say in each TODO that the streaming rework is a follow-up change rather than an open question.
They named bits 0..5 and reserved 6..7, contradicting both kBitWidthMask and the class doc four lines above.
Positions came off the wire and indexed the output buffer directly, so a corrupt page wrote past its end; num_exceptions was unchecked against it too.
A decoder is reused across the data pages of a column chunk, and a non-empty decoded_values_ marked the page as already decoded, so page 2 returned page 1.
LoadView now makes the same checks DecodeVector does, and GetMaxCompressedSize and SerializeVector return Result so their sizes are enforced, not assumed.
pfor.cc and pfor_wrapper.cc were in no library source list, so the test, the benchmark and libparquet each compiled their own copy.
It declared its own data_ and never called Base::SetData, so len_ stayed 0 and a full-page Decode now unpacks into the caller's buffer directly.
kHeaderSize and kVectorInfoSize were written out by hand; static_asserts now tie them to the members, and the exception count has a named wire type.
PforEncodedVectorView was reachable only from its own test and duplicated DecodeVector's validation. SerializeVector now rejects a vector whose info disagrees with the sections it carries.
An incompressible column encodes to its plain size plus the per-vector metadata, and only ColumnWriterImpl can relabel the page.
43 tests become 80. Ten cases that only ran at 32 bits now run at 64, which caught corrupt-page tests whose outlier the cost model declined to patch once an exception cost a 64-bit value.
A parquet page header counts nulls, and a PFOR page stores only the non-null values, so the level count the reader hands SetData is an upper bound on the values in the payload rather than the number of them. The decoder was using it as both: as the amount to decode and as the amount of decoded output to hand back, which for any nullable column decoded past the end of what the page actually holds. Read the count from the page header instead, where the encoder wrote it, and keep the level count only as the bound it is. Decoding a whole page in one call still writes straight into the caller's buffer; a partial read now decodes the page once into a pool-backed scratch buffer and serves the rest of the batch from it. DecodeArrow reserves once and then either advances over the values it just wrote or expands them leftward into their null positions, which drops the separate null-aware copy loop.
Two ways a malformed page got through. A header count short of the caller's capacity filled part of the output buffer and returned OK, leaving the rest of it holding whatever was there before; the count is now required to equal the expected value count, not merely fit inside it. And the offsets were each checked in isolation, only for being inside the buffer, so a page whose offsets overlap or run backwards decoded part-way and emitted values built out of another vector's bytes. Check the array as a chain up front instead: the first offset lands just past the array, each later one is strictly greater than the one before it, and all of them are inside the payload.
An all-null optional page buffers no values and is still written, and the empty payload the encoder emitted for it has no header for the reader to load, so reading such a column threw. Zero values now writes the header and nothing after it, and the wrapper accepts that count on both sides. The encoding test that covered this passed a level count of zero to SetData, which no real reader does -- the count it passes includes the nulls. It now passes the full count, and the writer tests round-trip an optional PFOR column with some nulls and with nothing but nulls.
Every multi-byte wire field -- the header's element count, the offset array, each vector's frame of reference and exception count, and the exception positions and values -- now converts explicitly, so the big-endian static_assert is gone. The bit-packed deltas already needed no conversion: BitWriter writes them little-endian and the unpacker reads them back the same way, so those bytes are copied verbatim.
The CMake build compiles them and runs pfor-test; meson knew about neither. The two translation units carry the only definitions of the PforWrapper and PforCompression instantiations parquet's encoder and decoder call, so a meson build has nothing to link them against.
A caller that reads a page in pieces decodes all of it up front, and the page layout can do better than that. Say so where the scratch buffer is, so the next reader of this code does not have to work it out.
Everything PFOR had so far drove the encoder and decoder classes directly, so nothing covered the path a real reader takes: writer properties, page headers, row group boundaries, null expansion. These 19 tests write real files and read them back, over the distributions the encoding is built for (a tight cluster far from zero, a constant vector, a cluster with outliers), the ones it has to survive (values at the bounds of each type, all-null and leading-null columns, row groups that end mid-vector, batches smaller than a page), and each supported compression codec. Every round trip also asserts the file's column chunks report PFOR. Without that a round trip passes just as happily when the writer picks some other encoding, and the test would be checking the default instead. arrow_reader_writer_test.cc is already long, so these go in arrow/arrow_encoding_test.cc, which later encodings can share.
The names now say what the install rules already do: arrow_install_all_headers globs a single directory and skips anything matching "internal", so nothing under arrow/util/pfor is installed and none of it carries a backward-compatibility obligation.
DecodeVector left UnpackOptions::max_read_bytes at its -1 default, which tells the bit-unpacker it may not read a byte past the vector's own packed payload. The vector kernels load a fixed-size window per step, wider than a step consumes at most bit widths, so under that bound the last step is refused and the tail of the vector falls to the scalar epilog -- 32 of 1024 values at bit width 3. The span DecodeVector is handed runs from this vector to the end of the page, so the true bound is already in hand; pass it. Whole-page int32 decode gains a median 1.13x over a 29-column corpus and 1.9x on the columns that pack to 3 bits, and nothing at widths 1, 2, 4, 8, 16 and 31, whose kernels strand no values to begin with. The gain is flat from a 78 KiB data page to a 4 MB destination, which is what a per-vector cost looks like. (cherry picked from commit d3490a55710e1c83fc0e2f8378e9c101e3e593f0)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doc : https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
Please remove this line and the above text before creating your pull request.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)