Heap Buffer Overflow in msgpack_zbuffer_expand() via Integer Overflow #1199

Summary

msgpack_zbuffer_expand() in include/msgpack/zbuffer.h doubles its buffer size (csize * 2) with no overflow check before passing the result to realloc(). On any platform where size_t is 32 bits (or, in principle, whenever csize approaches SIZE_MAX/2), this computation wraps around, producing a realloc() call with a smaller size than the buffer's current usage, followed by an avail_out computation that underflows to a huge bogus value.

• File: include/msgpack/zbuffer.h
• Function: msgpack_zbuffer_expand
• Line: 107 (nsize = csize * 2 — unguarded doubling), 109 (the resulting realloc)
• Affected version: c_master branch, HEAD 82a6cc03
• Bug class: Integer overflow leading to heap buffer overflow (CWE-190 → CWE-787)

Root cause

// include/msgpack/zbuffer.h, msgpack_zbuffer_expand()
static inline bool msgpack_zbuffer_expand(msgpack_zbuffer* zbuf)
{
    size_t used = (size_t)((char *)(zbuf->stream.next_out) - zbuf->data);
    size_t csize = used + zbuf->stream.avail_out;

    size_t nsize = (csize == 0) ? zbuf->init_size : csize * 2;   // line 107 — NO overflow guard

    char* tmp = (char*)realloc(zbuf->data, nsize);                // line 109
    if(tmp == NULL) {
        return false;
    }

    zbuf->data = tmp;
    zbuf->stream.next_out  = (Bytef*)(tmp + used);
    zbuf->stream.avail_out = (uInt)(nsize - used);                // underflows if nsize < used

    return true;
}

At csize == 2^31 (reachable on a 32-bit build where size_t is 32 bits, once cumulative compressed output crosses 2 GiB), csize * 2 wraps to 0. realloc(zbuf->data, 0) then either frees the buffer and returns NULL (triggering the separate double-free issue tracked in Finding 3, i.e. issue #3) or returns a token allocation smaller than used, after which nsize - used underflows to a huge value assigned to avail_out (a uInt/unsigned int). The next write into the buffer, believing it has a huge amount of room (avail_out), writes past the actual (far smaller or freed) allocation — a heap buffer overflow.

The sibling buffer include/msgpack/sbuffer.h (msgpack_sbuffer_write, lines 67–84) has the correct, guarded pattern: an explicit (sbuf->size + len) < len wrap check before computing a new size, plus an independent doubling-loop fallback that itself checks for wraparound. zbuffer.h's msgpack_zbuffer_expand has neither.

Trigger / reproduction

Reached via ordinary public API usage: any sequence of msgpack_zbuffer_write() calls (directly, or transitively via msgpack_pack_* functions writing through a msgpack_zbuffer) that pushes the buffer's cumulative used + avail_out size past 2^31 bytes on a 32-bit build (size_t/uInt = 32 bits). This requires sustained, large output (~2 GiB of real compressed data pushed through one msgpack_zbuffer instance) — a realistic scenario for any long-running service that streams significant volumes of msgpack data through zlib compression using this buffer type.

Isolated-arithmetic proof (no 32-bit toolchain required)

The exact wraparound mechanism can be demonstrated directly, emulating 32-bit size_t via uint32_t, without needing a 32-bit build or 2 GiB of RAM:

#include <stdint.h>
#include <stdio.h>

int main(void) {
    uint32_t csize = 0x80000000u;              /* 2^31 — the exact overflow point */
    uint32_t used  = 0x80000000u;
    uint32_t nsize = csize * 2;                /* wraps to 0 */
    uint32_t avail_out = (uint32_t)(nsize - used); /* underflows to 0x80000000 */

    printf("csize = %u (0x%08x)\n", csize, csize);
    printf("nsize = csize*2 = %u (0x%08x)  <-- WRAPPED (nsize < csize)\n", nsize, nsize);
    printf("avail_out = (uint32_t)(nsize - used) = %u (0x%08x)  <-- UNDERFLOWED\n", avail_out, avail_out);
    return 0;
}

clang -fsanitize=undefined -g -O1 -o arith_poc arith_poc.c
./arith_poc

Output:

csize = 2147483648 (0x80000000)
nsize = csize*2 = 0 (0x00000000)  <-- WRAPPED (nsize < csize)
avail_out = (uint32_t)(nsize - used) = 2147483648 (0x80000000)  <-- UNDERFLOWED (huge bogus avail_out)

UBSan does not flag this: unsigned-integer wraparound is well-defined behavior in C, not UB — the bug is a logic/bounds error, not a sanitizer-detectable UB violation, which is exactly why it needs an explicit guard like the one sbuffer.h already has.

Negative control (ordinary sizes — 0, 64 KiB, 16 MiB) shows no wraparound; nsize/avail_out stay sane, isolating the exact boundary condition to csize crossing 2^31 under 32-bit size_t.

Full end-to-end reproduction (requires 32-bit toolchain)

sudo apt install -y gcc-multilib g++-multilib libc6-dev-i386
clang -m32 -fsanitize=address,undefined -g -O1 -I include \
  <harness calling msgpack_zbuffer_init/_write repeatedly with ~2GiB of low-compressibility data> \
  src/*.c -lz -o poc_bug2_real_32bit

Not run in this audit (requires ~2 GiB of data generation/compression and a 32-bit multilib toolchain not present in the sandbox); the isolated-arithmetic proof above conclusively demonstrates the wraparound mechanism that this real path would hit.

Impact

Integer overflow leading to a heap buffer overflow / corrupted buffer-size bookkeeping on 32-bit builds of msgpack-c once cumulative output through a single msgpack_zbuffer exceeds ~2 GiB. Impact ranges from crash (DoS) to heap corruption, depending on allocator behavior and what subsequently gets written into the under-sized/freed buffer. Any 32-bit deployment that compresses large or attacker-influenced volumes of msgpack output through this buffer type is affected.

Suggested fix

Mirror the guarded pattern already used in include/msgpack/sbuffer.h: add an explicit overflow check before the doubling multiply (and a fallback if even the doubling loop would wrap), e.g.:

size_t nsize;
if (csize == 0) {
    nsize = zbuf->init_size;
} else if (csize > SIZE_MAX / 2) {
    return false;   /* would overflow */
} else {
    nsize = csize * 2;
}