Skip to content

fix(s3, oc): ListAsync returns every page; document provider behaviour - #116

Merged
samerzughul merged 1 commit into
mainfrom
muhannad/list-pagination
Nov 19, 2026
Merged

samerzughul merged 1 commit into
mainfrom
muhannad/list-pagination

Conversation

@mmalkhatib

@mmalkhatib mmalkhatib commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Fix

S3 and Oracle ListAsync returned only the first page of results (1,000 keys); Azure and Google already returned everything, so the same call behaved differently per provider and callers listing large prefixes were silently truncated.

  • S3: follow NextContinuationToken until IsTruncated is false.
  • Oracle: follow ListObjects.NextStartWith until null.
  • New S3CompatibleStoreTests.List_returns_every_object_beyond_the_first_page (SeaweedFS via Testcontainers): writes 1,050 objects and expects all of them. Fails on main, passes here; all 11 S3 tests pass locally. The Oracle change has no test (the OC tests need live credentials).

Docs

  • docs/provider-behavior.md: method-by-method, provider-by-provider behaviour read from the code (write settings honoured, RemoteBlob fields, buffering, listing, metadata keys, delete results, URL shapes, signed-URL cost, lifetimes).
  • docs/s3-compatible-stores.md: client settings and why, the synchronous startup calls in AddS3CloudFiles, Amazon/Spaces/MinIO/SeaweedFS differences.
  • docs/evolving-the-contract.md: rules for changing the shared interface (new capabilities as separate interfaces, as ICloudFilesLifecycle is), ranked roadmap, out of scope.
  • CHANGELOG.md (8.1.2 → 10.0.4), README docs index and corrections (metadata keys are not the same everywhere), CLAUDE.md corrections (GetSignedUrl is implemented on every provider; OpenWrite is S3-only).

Findings recorded in the docs but not fixed here: Google OpenWrite signs a GET URL and PUTs to it; Google ignores Metadata/Public on writes and buffers reads in memory; S3 WriteTextAsync drops Metadata; DeleteAsync disagrees on missing keys.

S3 made one ListObjectsV2 call and Oracle one ListObjects call, so any prefix
holding more than a page of objects was silently truncated, while the Azure and
Google providers already returned the full listing. Follow the continuation
token (S3) and NextStartWith (Oracle) until the listing is complete.

Adds a SeaweedFS test that writes 1,050 objects and lists them all; it fails on
the previous code.
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML; Path: .coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: f356fa5f-6804-43ca-8fd9-c58be7766bd0
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@samerzughul
samerzughul merged commit da0388e into main Nov 19, 2026
5 checks passed
@mmalkhatib mmalkhatib changed the title fix(s3, oc): ListAsync returns every page, not just the first 1,000 keys fix(s3, oc): ListAsync returns every page; document provider behaviour Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants