EnigmaConsultant/huntr-poc-arrow-ipc-list-offset-oob-heap-read

Out-of-bounds heap read / abort in Apache Arrow IPC (Feather V2): unvalidated nested-LIST child-buffer offset

Class: Out-of-bounds heap read (CWE-125) + reachable assertion abort (CWE-617) on parsing untrusted input Component: Apache Arrow C++ IPC (file & stream) RecordBatch reader — nested list<T> offset handling — exercised via PyArrow Affected version (tested, latest on PyPI): pyarrow==25.0.0 (bundles Apache Arrow C++ libarrow.so.2500) API surface: pyarrow.ipc.open_file(...).read_all() / RecordBatchFileReader (the default read path) followed by any element access (to_pylist / to_pandas / indexing / iteration) Impact: Reading an attacker-supplied .arrow / .feather file containing a list<...> column with a corrupted int32 offset and touching the data yields one of three outcomes depending on the offset value:

• a silent backwards out-of-bounds heap read that returns leaked adjacent heap memory as list elements (information disclosure) — no crash, and the cheap Table.validate() passes;
• an Arrow CHECK failure (arrow/array/data.cc:190) → std::abort (SIGABRT, denial of service);
• a SIGSEGV (backwards OOB read into an unmapped page).

---

Why this is distinct from the string/binary-offset OOB

A separately reported finding covers utf8/binary arrays where a corrupted last offset drives a forward OOB inside std::string(data+off, len) (arrow::Array::GetScalar), and where the cheap Table.validate() catches it.

This finding is a different type, code path, and — critically — a different validation behavior:

	string/binary offset finding	this: list offset finding
Array type	utf8 / binary	nested list<T>
Faulting code	std::string ctor in scalar materialization	arrow::ArrayData::Slice (array/data.cc:190)
Cheap Table.validate()	catches it (ArrowInvalid)	PASSES — real gap on the default path
Primary impact	forward OOB → SIGSEGV (DoS)	silent backwards OOB heap read (info leak), plus SIGABRT / SIGSEGV
Trigger offset	positive, too large	negative (leak/segv) or positive too large (abort)

The information-disclosure outcome (silent leak that survives both read_all() and the cheap validate()) is the most serious and has no analogue in the string case.

---

Root cause

For a list<T> array the int32 offset buffer must satisfy 0 == offsets[0] <= offsets[1] <= … <= offsets[n] <= len(child_values). The IPC reader trusts the flatbuffer buffer layout and never checks the offset values. On element access, Arrow materializes list slot i as

child_values->Slice(offsets[i], offsets[i + 1] - offsets[i]);

ArrayData::Slice(off, len) (arrow/array/data.cc) does:

ARROW_CHECK_LE(off, length);                 // line 190: aborts if off  > length
int64_t new_len = std::min(len, length - off);

• With a negative offsets[i] (e.g. -100), off = -100 passes off <= length, and the slice starts 100 * sizeof(int32) bytes before the child buffer and reads min(len, length-off) elements — a backwards OOB heap read. The bytes are handed back as list values → heap disclosure. A large negative offset walks off a mapped page → SIGSEGV.
• With a too-large positive offsets[i] (e.g. 0x7ffffff0), off > length fails ARROW_CHECK_LE → the CHECK prints Check failed: (off) <= (length) and calls std::abort() → SIGABRT.

The cheap Table.validate() does not walk offset contents, so it returns success in every case; only validate(full=True) catches it (Offset invariant failure: non-monotonic offset at slot 3: -100 < 5), and it is never called on the default read path.

---

Proof of Concept

Build a one-column list<int32> table [[1,2],[3,4,5],[],[6,7,8,9]] and write it as an Arrow IPC / Feather V2 file. The valid int32 offset buffer is [0,2,5,5,9]; overwrite offsets[3] (the value 5).

list_repro.py (public API only) builds the files and triggers each outcome:

$ python3 list_repro.py build
$ python3 list_repro.py leak ; echo "exit=$?"
read_all() OK, num_rows=4
cheap Table.validate(): PASS  <-- validation gap
leaked sublist length = 109 (should be 4)
leaked head (OOB heap ints) = [1330795073, 12631, -1, 168, 16, 655360, 393228, 524293]
exit=0

$ python3 list_repro.py abort >/dev/null ; echo "exit=$?"
/arrow/cpp/src/arrow/array/data.cc:190:  Check failed: (off) <= (length) Slice offset (2147483632) greater than array length (9)
exit=134            # 134 = 128 + SIGABRT(6)

$ python3 list_repro.py segv >/dev/null 2>&1 ; echo "exit=$?"
exit=139            # 139 = 128 + SIGSEGV(11)

Negative control (unmodified file reads and fully validates cleanly):

$ python3 list_repro.py control
read_all() OK, num_rows=4
cheap Table.validate(): PASS  <-- validation gap
control to_pylist: [[1, 2], [3, 4, 5], [], [6, 7, 8, 9]]
control validate(full=True) OK

The leak is deterministic across runs (returns the same 109-element sublist of adjacent heap ints), confirming a genuine OOB read of adjacent heap rather than an allocation quirk.

Included artifacts

• list_ok.arrow — negative control (valid, reads & fully-validates cleanly).
• list_leak_poc.arrow — offsets[3] = -100: silent backwards OOB heap read (info leak), exit 0.
• list_abort_poc.arrow — offsets[3] = 0x7ffffff0: CHECK failure → SIGABRT (exit 134).
• list_segv_poc.arrow — offsets[3] = -100000000: backwards OOB → SIGSEGV (exit 139).
• list_repro.py — self-contained build + trigger + negative control.

---

Captured evidence (verbatim, pyarrow 25.0.0)

### LEAK    read_all() OK, num_rows=4 ; cheap validate PASS ; leaked len 109 ; head [1330795073, 12631, -1, 168, ...] ; exit 0
### ABORT   /arrow/cpp/src/arrow/array/data.cc:190:  Check failed: (off) <= (length) Slice offset (2147483632) greater than array length (9) ; exit 134
### SEGV    exit 139
### FULL validate: ArrowInvalid  Offset invariant failure: non-monotonic offset at slot 3: -100 < 5   (never called on read path)
### CONTROL control to_pylist: [[1, 2], [3, 4, 5], [], [6, 7, 8, 9]] ; validate(full=True) OK

---

Impact & remediation

Any application that reads Arrow IPC / Feather V2 with nested (list/map) columns from an untrusted or semi-trusted source (uploaded datasets, model-artifact sidecars, message-bus payloads, dataframe interchange) and then touches the data is exposed to heap information disclosure and/or process crash. Because read_all() and the cheap Table.validate() both return success, a careful developer who validates cheaply still has no signal before the OOB read.

Mitigations:

• Callers must run RecordBatch.validate(full=True) / Array.validate(full=True) on every batch decoded from untrusted IPC before access; the cheap validate() is insufficient for the list-offset case.
• Upstream: the IPC reader should validate variable-length/list offset buffers (monotonic, 0 <= offsets[i] <= child_length) on load, and ArrayData::Slice should reject negative offsets rather than deferring an unchecked OOB into element access.

Dedup note

• Distinct format & code path from the reporter's Parquet findings (footer rowgroups alloc DoS, pushdecoder footer underflow, DELTA_LENGTH_BYTE_ARRAY, BYTE_STREAM_SPLIT) — those are the Parquet reader.
• Distinct from the sibling Arrow IPC string/binary offset OOB: different array type (list), different faulting frame (ArrayData::Slice vs std::string ctor), and a different validation gap (cheap validate() passes here), with an additional silent heap-disclosure outcome. Packaged under the closest available "parquet"/Arrow category; reviewer may recategorize to Arrow IPC/Feather.