Skip to content

include/nuttx/ring.h: add a lock-free ring queue - #21945

Open
CV-Bowen wants to merge 45 commits into
apache:masterfrom
CV-Bowen:up/P91-ring
Open

CV-Bowen wants to merge 45 commits into
apache:masterfrom
CV-Bowen:up/P91-ring

Conversation

@CV-Bowen

@CV-Bowen CV-Bowen commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add include/nuttx/ring.h, a header-only lock-free byte ring queue modelled
on DPDK's rte_ring. Producer and consumer each keep a head/tail pair packed
with a thread counter into one 32-bit word ({cnt:8, idx:24}, DPDK's RTS
scheme), so multi-producer and multi-consumer sides commit with a single CAS
and never spin waiting for another thread's tail.

  • Four modes, chosen per call: SPSC (volatile + barriers, no atomic RMW, so it
    also works on shared memory between cores without coherent atomics), MPSC,
    SPMC, MPMC.
  • Byte queue with arbitrary (non-power-of-two) capacity up to 16 MiB.
  • Partial and all-or-nothing (_bulk) enqueue/dequeue, zero-copy
    _start/_finish, skip/skipto, peek/peek_at, and overwrite mode
    (SPMC/MPMC) that evicts the oldest data when full.
  • The ring is split into a caller-owned blob (indices, geometry and data;
    may be shared across cores or survive a reboot) and a per-user handle in
    ordinary RAM. ring_*_init() detects a blob left by a previous boot and
    recovers its contents (warm restart), with an optional rewind to replay
    already-consumed data, and resets a blob whose indices or geometry do not
    match. ring_spsc_init() has a slave mode so only one side of a
    cross-core ring ever resets it.

There is no in-tree user yet. Internally it backs the note/notesnap, ramlog
and uart_rpmsg drivers; those conversions will follow as separate PRs per
driver.

Why 45 commits: the history is kept as developed by several authors
(Bowen Wang, chenzihan1, wangjianyu3, Xiang Xiao, yezhonghui, ganjing) so each
change stays attributed and reviewable on its own. Every commit only touches
include/nuttx/ring.h; changes to other files in the original commits (the
in-tree users above) were dropped, and those commits carry an extra
Signed-off-by.

Depends on #21843 (atomic_ptr_t, atomic_cmpxchg_release_acquire) and
#21895 (irq_save/irq_restore), both merged.

Impact

Testing

Host: Ubuntu 22.04.5 LTS x86_64; sim: gcc 13.1.0; qemu-armv8a: aarch64-none-elf-gcc 12.2.1, QEMU 10.2.1

The cmocka tests from apache/nuttx-apps#4075 (80 cases: 36 SPSC, 11 MPSC,
11 SPMC, 22 MPMC, including pthread producer/consumer stress, wrap-around on
non-power-of-two capacities, overwrite, warm restart/rewind, geometry
mismatch and SPSC slave init), with CONFIG_TESTING_CMOCKA=y CONFIG_TESTING_RING=y (qemu-armv8a also needs CONFIG_ARCH_SETJMP_H=y
for cmocka):

configure result
sim:nsh 80/80 PASS, 0 warnings
qemu-armv8a:nsh_smp (CONFIG_SMP_NCPUS=2) 80/80 PASS, 0 warnings
qemu-system-aarch64 -cpu cortex-a53 -smp 4 -nographic \
  -machine virt,virtualization=on,gic-version=3 -net none \
  -chardev stdio,id=con,mux=on -serial chardev:con \
  -mon chardev=con,mode=readline -kernel nuttx
nsh> ring_test
sim:nsh
[==========] g_spsc_local_tests: 36 test(s) run.
[  PASSED  ] 36 test(s).
[==========] g_mpsc_local_tests: 11 test(s) run.
[  PASSED  ] 11 test(s).
[==========] g_spmc_local_tests: 11 test(s) run.
[  PASSED  ] 11 test(s).
[==========] g_mpmc_local_tests: 22 test(s) run.
[  PASSED  ] 22 test(s).
qemu-armv8a:nsh_smp
[CPU0] [==========] g_spsc_local_tests: 36 test(s) run.
[CPU0] [  PASSED  ] 36 test(s).
[CPU0] [==========] g_mpsc_local_tests: 11 test(s) run.
[CPU0] [  PASSED  ] 11 test(s).
[CPU1] [==========] g_spmc_local_tests: 11 test(s) run.
[CPU1] [  PASSED  ] 11 test(s).
[CPU0] [==========] g_mpmc_local_tests: 22 test(s) run.
[CPU0] [  PASSED  ] 22 test(s).

No regression in rpmsg: sim:rpserver/rpproxy and qemu-armv8a:rpserver/rpproxy boot and pass the fixed rpmsg cases (dump, ping, rpmsgfs; 15/15). qemu-armv8a was built with LIBC_ENVPATH and BINFMT_CHECKEXEC disabled, which current master needs for nxinit to start nsh there; unrelated to this change.

CV-Bowen and others added 30 commits November 17, 2026 16:59
Add nuttx/ring.h implementing a unified lock-free ring queue
supporting SPSC/MPSC/SPMC/MPMC modes using RTS (Relaxed Tail Sync)
algorithm with single CAS operations.

API uses explicit mode macros (ring_spsc_enqueue, ring_mpmc_dequeue,
etc.) with compile-time flags and sizeof for optimal codegen. RTS
tail CAS uses release/acquire ordering matching DPDK rte_ring.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Extract sp/sc/mp/mc start and finish as reusable building blocks,
refactor existing enqueue/dequeue to use them.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add dequeue_skip variants that advance ring indices without touching
data, built on the reserve + commit building blocks from the previous
patch.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add ring_buf_s descriptor and two-phase zero-copy operations:
- enqueue: start (get writable pointers) + copy + finish (publish)
- dequeue: start (get readable pointers) + copy + finish (release)

Unify enqueue/dequeue to use ring_buf_s internally via shared
ring_buf_fill, ring_copy_from, ring_copy_to helpers, eliminating
the old ring_copy_from/ring_copy_to that duplicated wrap-around
logic.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add an atomic_t magic field to ring_s for detecting whether a ring
has been previously initialized.  This supports rings placed in
memory that survives reboot (shared SRAM, battery-backed RAM).

ring_init() now uses a three-state CAS protocol:
  - Uninitialized (magic != READY/INIT): fresh init, zero indices
  - RING_MAGIC_READY: warm restart, validate and recover indices
  - RING_MAGIC_INIT: another core is initializing, spin-wait

Multi-core safe: concurrent ring_init() calls are serialized via
CAS on the magic field.  Loser cores spin until the winner sets
RING_MAGIC_READY, ensuring no core uses the ring before init
completes.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add ring_space(r) that returns the number of available slots
for enqueue, complementing the existing ring_count/ring_capacity.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
When count is zero, ring_buf_init returns early without
initializing ptr1/len1/ptr2/len2. GCC inlines the callers
(ring_mp_enqueue, ring_sc_dequeue, etc.) and cannot prove
these members are set, triggering -Werror=maybe-uninitialized.
Initialize all ring_buf_s members on the zero-count path.

Signed-off-by: yezhonghui <yezhonghui@xiaomi.com>
Add overwrite enqueue that discards oldest committed data when the ring
is full, guaranteeing writes always succeed. Typical use case: sensor
data streams and log buffers where latest data matters most.

The discard path acts as a virtual multi-consumer (CAS on cons_head +
mc_finish), so overwrite is only available for MC queues (SPMC/MPMC).
SC queues cannot support overwrite because the producer must CAS
cons_head, which conflicts with SC's plain-store protocol.

Provides both copy and zero-copy (start/finish) variants:
  ring_spmc_enqueue_overwrite / ring_mpmc_enqueue_overwrite
  ring_spmc_enqueue_overwrite_start / ring_spmc_enqueue_overwrite_finish
  ring_mpmc_enqueue_overwrite_start / ring_mpmc_enqueue_overwrite_finish

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
SPSC mode has no concurrent writers on the same index - producer
only writes prod_tail, consumer only writes cons_tail. Atomic
operations are unnecessary overhead for this case.

Replace atomic_read/atomic_set_release with volatile pointer access
and explicit UP_WMB/UP_RMB memory barriers:
 - Producer: write data -> UP_WMB -> volatile write prod_tail
 - Consumer: volatile read prod_tail -> UP_RMB -> read data

This avoids issues on platforms where atomic instructions are not
available or have high overhead (e.g. AMP without shared atomic
support), while maintaining correct memory ordering for SMP/BMP with
coherent cache.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add comprehensive usage documentation at the top of ring.h
covering declaration, initialization, mode selection (SPSC/MPSC/
SPMC/MPMC), overwrite mode, zero-copy pattern, and query APIs.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
When the ring is full (enqueue) or empty (dequeue), the start
functions return buf->cnt == 0. Previously ring_copy_from and
ring_copy_to would fall into the else branch and call memcpy
with len1=0 and ptr1=NULL, which is undefined behavior per the
C standard even when size is zero.

Change the else branch to else-if (buf->cnt != 0) so that
zero-count cases are safely skipped without adding an extra
early-return branch.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add a mask field to ring_s: if capacity is power-of-2, mask stores
capacity-1 for fast bitwise index; otherwise mask is 0 and ring_idx()
falls back to modulo. This allows ring queues to use any positive
capacity without wasting memory to round up.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add compile-time helpers for ring buffer size validation:
- RING_SIZE_MAX: maximum allowed capacity (2^23, limited by 24-bit index)
- RING_SIZE_VALID(sz): validates size is > 0, power-of-2, and <= RING_SIZE_MAX

This allows users of ring.h to validate buffer sizes consistently
without open-coding the constraints.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
The struct-level aligned_data(CACHELINE_SIZE) attribute on ring_s
makes some compilers (e.g. TASKING) propagate the alignment to every
member, inflating sizeof(struct ring_s) from 192 to 640 bytes and
breaking the cross-core shared memory layout. Move the attribute from
the type to each declaration/definition site (RING_DECLARE,
RING_DEFINE_BMP) so it binds to the variable, and use lib_memalign in
ring_alloc for the dynamic path. The cache-line isolation between
regions is preserved by the explicit pad0/pad1/pad2 arrays.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add two peek functions that read data from the ring using a caller-
managed index without advancing cons_tail:

- ring_sc_peek: single-consumer peek, no retry, suitable for SC or
  panic-path scenarios where overwrite race is not a concern.
- ring_mc_peek: multi-consumer peek with optimistic-read-then-verify
  loop, retries automatically if the read region was overwritten
  during copy.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
ring_*_enqueue_overwrite() guarantees the new element is admitted by
silently evicting older committed elements when the ring is full.
Until now that loss was invisible to callers: the function returned
the number of bytes admitted (always == count * elem_size) and gave
no way to learn how many bytes the eviction took with it.  For users
that care about end-to-end accounting (note backends in particular)
this turned overwrite enqueue into a black box.

Surface the eviction count by folding it into the return value:

 - ring_sp_enqueue_overwrite_start() / ring_mp_enqueue_overwrite_start()
   now return uint32_t, the number of elements evicted to make room
   for the request.  The MP variant accumulates evictions across
   multiple CAS retries when more than one tail block has to be
   reclaimed before head can advance.  Zero-copy callers can pair
   the start() return value with their own bookkeeping.

 - ring_sp_enqueue_overwrite() / ring_mp_enqueue_overwrite() capture
   the start() return value and now return buf.cnt - discard, i.e.
   the net change in ring occupancy rather than the gross number of
   elements written.  When the ring is not full this still equals
   count; when it is full the caller can compute eviction loss as
   count - return.  The macros ring_spmc_enqueue_overwrite /
   ring_mpmc_enqueue_overwrite are unchanged at the call site syntax
   level.

This is a generic ring extension; the first consumer (notebuf folding
the count into drop_bytes) is added in a separate patch.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
ring_reset() previously returned void and silently discarded all
elements still in the ring at the time of the call.  Callers that
care about loss accounting (note backends in particular) had no way
to learn how many elements were dropped, forcing them to either
sample ring_count() before the reset (racy) or stop tracking drops
entirely on NOTE_CLEAR-style operations.

Convert the macro to a GCC statement expression that snapshots the
producer/consumer index delta before zeroing the four indices and
yields it to the caller.  All existing void-context callers
(ring_init_magic warm-restart fallback, ring_init_magic first-time
init, ring_spsc_init, notebuf_ioctl NOTE_CLEAR) compile unchanged
because C permits discarding expression values.

Consumers that want to fold the discarded count into their own
"dropped" metric can now write:

    dropped = ring_reset(buf->ring);
    atomic_add(&stats.drop_bytes, dropped);

This is a generic ring extension; the first consumer (notebuf
folding the count into drop_bytes on NOTE_CLEAR) is added in a
separate patch.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
Add ring_sc_skipto()/ring_mc_skipto() that advances cons_tail to a target index
directly, computing the skip count internally. This avoids exposing
RTS_IDX encoding details to callers.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Shorten ring public macro names by removing redundant verbs:

- ring_xxxx_dequeue_skip -> ring_xxxx_skip
  Skip always operates on consumer side, "dequeue" is redundant.

- ring_xxxx_enqueue_overwrite -> ring_xxxx_overwrite
  Overwrite always implies enqueue, "enqueue" is redundant.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Insert UP_WFE() in busy-wait loops to hint the processor to reduce
power consumption during spinning. Add UP_SEV() at completion points
to signal waiting cores.

Signed-off-by: ganjing <ganjing@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Add DEBUGASSERT(r != NULL && cap > 0 && cap <= RING_IDX_MASK) at the
entry of ring_init_magic() to catch invalid parameters early.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Add RING_CAPACITY(bufsize) macro to extract data capacity from total
buffer size (bufsize - sizeof(struct ring_s)). All ring APIs now accept
total buffer size (header + payload) as the size parameter, and compute
the usable data capacity internally.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Remove char ring[] from struct ring_s to fix Clang warning
-Wgnu-variable-sized-type-not-at-end when ring_s is embedded as
a non-terminal member via RING_DECLARE.

Replace with ring_data(r) macro that computes the buffer address
as (struct ring_s *)(r) + 1. Memory layout is unchanged.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Add a static_assert(RING_CAPACITY(buflen) <= RING_SIZE_MAX) as the first
member of the ring struct in RING_DECLARE and RING_DEFINE_BMP, so an
over-sized buffer is rejected at compile time instead of silently
overflowing the 24-bit ring index. Centralizing the check in the macros
removes the need for callers to repeat it at each ring definition site.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
ring_is_valid_cons compared the raw 32-bit signed difference of two
indices, but ring indices are 24-bit free-running counters (RTS_IDX_MASK)
that wrap at 2^24.  Near the wrap boundary the comparison is wrong: e.g.
cons_index=0x000003 is actually ahead of cons_tail=0xfffffe, yet the plain
signed difference is large and negative, so the macro misjudged live data
as lapped.

Shift the 24-bit difference left by 8 so bit 23 lines up with the 32-bit
sign bit, turning the test into a correct modular "within half the 24-bit
range" comparison.  Since the live window is bounded by capacity <=
RING_SIZE_MAX (2^23), the half-range split exactly separates
ahead-of-cons from lapped-behind-cons.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
The overwrite helpers previously returned the net change in ring
occupancy (inserted minus evicted), which hides the true number of
elements discarded when the ring is full.  Change ring_sp_overwrite()
and ring_mp_overwrite() to return the eviction (discard) count directly
so callers can account for dropped data precisely; the count is zero
when the ring had enough free space.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
The external-index peek requires callers to handle absolute ring indices,
which forces scanners that inspect buffer contents to reach into ring
internals.  Add ring_*_peek_at(r, items, count, offset): it reads count
elements at a fixed offset from the oldest element (cons_tail) without
advancing, expressing the read position relative to the consumer tail so
the caller never sees an absolute index.  An out-of-range offset yields
zero.  Implemented by delegating to the existing single/multi-consumer
peek primitives.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
A reader that walks the ring by absolute index needs a way to start from
the oldest readable position without a valid index in hand.  Add the
RING_IDX_OLDEST sentinel: a real index never exceeds RTS_IDX_MASK (24
bits), so this out-of-range value is unambiguous.  When ring_mc_peek()
sees it in *index it skips ring_clamp_cons() (whose signed compare would
be meaningless on the sentinel) and starts from the live cons_tail.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
A reader that tracks an absolute position and wants to detect whether an
overwrite lapped it needs to compare its expected next position against
the index a peek reported.  The peek returns a 24-bit index, while the
expected position is a linear (unwrapped) value, so the two cannot be
compared directly without masking.  Add ring_wrap(pos) to wrap a linear
position into the same 24-bit index space using the public RING_IDX_MASK,
so callers avoid reaching into the internal packed layout.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
ring_mp_overwrite_start() bumps prod_head.cnt unconditionally once its
reserve CAS succeeds, even when count is 0.  Its commit counterpart
ring_mp_finish() returns early when buf->cnt == 0 and never bumps
prod_tail.cnt.  A zero-count overwrite therefore adds a prod_head.cnt
increment with no matching prod_tail.cnt increment, leaving a permanent
head.cnt != tail.cnt gap.

The RTS tail sync in ring_mp_finish() only advances prod_tail.idx when
tail.cnt catches up to head.cnt.  Once the gap is permanent that
condition can never hold again, so prod_tail.idx freezes.  With a frozen
tail, ring_mp_overwrite_start() computes committed == 0, every subsequent
overwrite finds discard > committed, and all producers spin forever in
the eviction retry loop.

Guard against count == 0 before the reserve loop and return an empty buf,
mirroring the n == 0 break already present in ring_mp_start().  The copy
wrappers stay correct because ring_buf_init() already zeroes buf on a
zero count.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
CV-Bowen and others added 15 commits November 17, 2026 16:59
User pass *init to judge whether this init is the first init
of the ring to avoid race condition when user want to
initialize their own data struct.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
…ings

ring_sc_start()/ring_mc_start() derive the available element count from a
non-atomic pair of index reads: prod_tail is loaded separately from the
consumer index. In OVERWRITE mode the producer is allowed to advance the
consumer-owned cons_head/cons_tail to evict the oldest entries, so a flush
consumer can observe a torn snapshot where prod_tail is stale while the
consumer index has already been pushed past it in 24-bit modular order.

ring_dist() then computes (prod_stale - cons_new) & RING_IDX_MASK, which
underflows to ~16M - far larger than the ring capacity. n = min(count,
available) stays huge and ring_buf_init() faithfully splits it into
len1 + len2 with no capacity clamp, producing an over-length segment. The
downstream memcpy in the note/netpoll path then walks off the mapped ring
and takes a bus trap.

A consistent ring can never hold more committed entries than its capacity,
so any available > capacity is by definition a torn read. Force available
to 0 in that case: the round yields no data and the periodic
watermark/timeout flush re-reads a coherent snapshot on the next pass with
no entries lost. Clamping to 0 rather than capacity avoids advancing the
consumer index past the real prod_tail, which would corrupt the ring.

The same producer/consumer divergence also underflows the producer side.
ring_mp_overwrite_start() computes free_space = capacity - ring_dist(
prod_head, cons_head); once the producer laps the consumer by more than
capacity in 24-bit modular order this subtraction wraps to ~4G. The
"free_space < count" test is then never true, so the eviction branch that
reclaims the oldest slots is never entered again: the producer advances
prod_head without bound and the evicted-byte accounting silently stops,
leaving megabytes of overwritten data uncounted. Guard the subtraction to
yield 0 when the ring is already full, forcing the eviction path to run
and restoring the invariant dist(prod_head, cons_head) <= capacity.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
…apacity

The RTS ring packs a 24-bit byte index that wraps at 2^24. The physical
mapping uses idx % capacity when capacity is not a power of two. Physical
continuity across the wrap requires 2^24 to be a multiple of capacity, which
only holds for power-of-two capacities. On non-power-of-two capacities the
physical position jumps by (2^24 % capacity) bytes at every 2^24 boundary,
tearing any record straddling the wrap and producing illegal bytes downstream.

Introduce a wrap field equal to (2^24 / capacity) * capacity, the largest
capacity-multiple not exceeding 2^24, and route all index arithmetic through
ring_advance/ring_dist so indices wrap at a capacity multiple. Power-of-two
capacities keep wrap == 2^24 (identical codegen, no regression); non-power-of-two
capacities now stay continuous across the wrap.

ring_is_valid_cons() takes the ring instead of the capacity, and the
ring_wrap macro is replaced by ring_advance/ring_dist.  sizeof(struct
ring_s) is preserved at 192 bytes.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
since the caller alredy add the barrier insruction correctly

Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
prepare for the head and data separation

Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Split the ring into a handle and a blob.  struct ring_blob_s is the
memory that may be shared across cores or survive a reboot: magic,
geometry, the producer/consumer indices and the data.  struct ring_s
is a per-user handle in ordinary RAM holding a blob pointer and a
boot-scoped magic.

The handle magic is cleared on every boot, so a second
ring_xxx_init() on the same handle in the same boot returns early
instead of resetting a ring already in use, while the blob magic
still drives warm-restart recovery.  The caller now provides the blob
storage and its total size; RING_DECLARE/RING_DEFINE/RING_INIT are
removed.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
ring_mpmc_init() takes a rewind count: on a warm restart it pulls
cons_tail back that many elements, so data the previous run had already
consumed becomes readable again.  Only meaningful for a blob that
survives reboot (a located .noinit section or shared SRAM); a cold start
has nothing to replay and a blob with corrupted indices is reset, so
neither path applies it.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
Reservation could push prod_head - cons_head past capacity, letting a
producer write over the oldest live element.

After evicting old data to make room, the loop refreshed old_prod before
the prod_head CAS.  That defeated the CAS: the space just evicted may
already have been taken by another producer, and with a refreshed
expected value the CAS succeeded anyway, reserving on top of the
eviction instead of retrying.

Two producers A and B, both writing count = 32 bytes into a full ring of
capacity = 1856, starting at prod_head = P, cons_head = C, P - C = 1856:

  #  who  action                                    prod_head  cons_head  used
  1  A    old_prod = P (read at top of loop)        P          C          1856
  2  A    free_space = 0, so discard = 32           P          C          1856
  3  A    CAS cons_head: C -> C+32, succeeds        P          C+32       1824
  4  A    ring_mc_finish(), cons_tail = C+32        P          C+32       1824
  5  B    sees free_space = 32 >= 32, skips the
          eviction branch entirely, and
          CAS prod_head: P -> P+32 succeeds         P+32       C+32       1856
  6  A    re-reads prod_head, gets P+32             P+32       C+32       1856
  7  A    CAS prod_head: P+32 -> P+64, expected
          value was just read, so it succeeds       P+64       C+32       1888

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
ring_mpmc_init / ring_spsc_init reused a persisted blob for a warm
restart whenever its magic already read READY, then immediately called
ring_dist to recover the live element count.  ring_dist divides by
wrappoint, so a blob left in memory by a previous firmware whose bytes
happen to decode as magic == READY but wrappoint == 0 divides by zero
on the very first ring_dist.  Even when wrappoint is non-zero, trusting
magic alone lets stale indices from an incompatible layout be reused as
if valid.

The ramlog syslog device hits this: it keeps its blob in a located
.noinit section with CONFIG_RAMLOG_RECOVER_SIZE, so the first log write
after a warm boot faults and the DIVBYZERO escalates to a HardFault.

Geometry (capacity and wrappoint) is a pure function of buflen and is
identical on every boot of a given build.  Compute it up front and
treat the blob as a valid warm restart only when magic == READY and the
stored capacity and wrappoint both match the freshly computed values;
any mismatch (stale residue, a resized buffer, or a zeroed wrappoint)
falls back to a full ring_reset.  The geometry is stamped before any
ring_dist, so the divisor is always valid and only a provably
self-consistent blob keeps its indices.

Signed-off-by: wangjianyu3 <wangjianyu3@xiaomi.com>
Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
…apped

ring_mp_overwrite_start() spun forever in an interrupt handler, wedging
the CPU.

When the ring is full the reservation has to evict old data first, bounded
by the committed region: dist(prod_tail, cons_head).  Slots between
prod_tail and prod_head belong to producers that reserved them but have
not called ring_mp_finish() yet, so evicting them would hand out memory
another producer is writing into.  The loop therefore retried until enough
data had been committed.

That retry assumes the in-flight producers can make progress.  The note
path is reentrant and does not disable interrupts, so they may not: a
producer preempted between its prod_head CAS and ring_mp_finish() cannot
run while an interrupt handler on the same CPU spins waiting for it.
prod_tail stays frozen, committed only shrinks as later producers evict,
and the retry never terminates.

Observed on TriCore during boot, with a note of 20 bytes on a ring of
capacity 1856:

  prod_head = 0x763cc545  cnt=118  idx=3982661
  prod_tail = 0x753cbded  cnt=117  idx=3980781
  cons_head = 0xc83cbde9  cnt=200  idx=3980777
  cons_tail = 0xc83cbde9  cnt=200  idx=3980777

  free_space = 0, committed = dist(prod_tail, cons_head) = 4,
  discard = 20 > 4

The RTS counters are consistent here - prod skew is 1, matching the single
in-flight producer - so nothing is corrupt; the loop simply has no exit.

Drop the request instead.  Trace records are discardable, and the whole
count is reported as discarded so CONFIG_DRIVERS_NOTE_STATS accounts for
it.  Returning an empty buf makes ring_mp_finish() return early, leaving
no prod_head reservation unmatched.  Once the stalled producer completes,
prod_tail catches up and the ring resumes accepting writes.

ring_sp_overwrite_start() is single-producer, so prod_head never runs
ahead of prod_tail there and it needs no equivalent change.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
None bluk is best-effort: it takes as much as fits and reports how much that
was.  That suits a byte stream, where a short write only shortens the data,
but not a framed one, where the reader locates each record by a length field
carried in the record before it.  A fragment left in the ring desynchronises
the reader from the frame boundaries, so every later record is misparsed -
the ring is not one record short, its whole remaining content is lost.

Callers can pre-check ring_space() and skip the enqueue themselves, but that
reads the consumer index twice for what is one decision, and every caller
with a framed payload has to repeat it.

Add a bulk variant that reserves count slots or none at all, so buf.cnt
comes back as count or 0 and never as a prefix.

Signed-off-by: chenzihan1 <chenzihan1@xiaomi.com>
ring_*_skipto computed the skip count as ring_dist(target, cons), a
modular forward distance. A target no longer ahead of the consumer
(already consumed, or lapped by the writer) wraps that distance into a
near-full-ring count, so the skip would evict almost all still-live
data instead of doing nothing. Check ring_is_valid_cons against the
same cons snapshot before taking the distance, and no-op when the
target is not ahead.

Signed-off-by: yezhonghui <yezhonghui@xiaomi.com>
The data area is a uint8_t array and capacity counts bytes, but every
index is still scaled by elem_size when an address is formed.  Any
elem_size above one walks past data[]: sixteen uint32_t enqueued into a
16-byte ring write 64 bytes.

Remove elem_size everywhere.  Counts, lengths and ring_buf_s fields are
bytes; the copy macros take a void pointer instead of sizing the
element from it, and the zero-copy start macros lose their elem_size
argument.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
ring_sc_start() and ring_sc_peek() read element data right after the
prod_tail load with no read barrier in between, so a weakly ordered core
can return bytes older than that prod_tail. ring_sc_finish() published
cons_tail after UP_WMB(), which orders stores only, so the producer
could reuse slots still being read.

Move the UP_RMB() in ring_sc_start() and ring_sc_peek() after the
prod_tail load, and use UP_DMB() before the cons_tail store.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
ring_spsc_init() must not run on both sides at once: two callers that
both find the blob uninitialized both reset it, so a side stalled
between its magic check and its reset wipes whatever the other side has
already initialized and sent.

Add a slave argument: a slave binds its handle only once the blob is
initialized and never resets it, so the master can own init and the
slave only bind, without atomic RMW on the shared memory. The return
type becomes int so a slave gets -EAGAIN/-EINVAL to retry.

Signed-off-by: Bowen Wang <wangbowen6@xiaomi.com>
@github-actions github-actions Bot added Area: OS Components OS Components issues Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: OS Components OS Components issues Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants