Repository navigation
fix(storage): GCS download URLs for keys containing '%' return NoSuchKey - #5369
Merged
Merged
Conversation
Storage keys percent-encode their segments (key_segment), so a bank id
with a space, a dot or non-ASCII is stored under an object name that
literally contains '%' ("Express%20AI" is the name). obstore's GCS
signer puts the key into the URL path unescaped; GCS decodes the path,
looks up "Express AI", and the download fails with NoSuchKey even though
the export (or any other stored file) is there.
GCSFileStorage.get_download_url now escapes '%' as '%25' before signing,
so the path decodes back to the literal object name. The S3 signer
already escapes '%' -- pre-escaping there would double-encode -- so the
fix is GCS-only.
Tests:
- test_file_storage_gcs_signing.py signs for real, offline, with a
throwaway service-account key and checks that the URL path decodes to
the stored object name for plain, spaced, dotted, non-ASCII and '%'
bank ids. Without the fix the three encoded cases fail.
- test_file_storage_s3.py now downloads through the presigned URL
against SeaweedFS for the same bank ids (it previously only checked
the result was a URL string), pinning that S3 needs no escaping.
Code-review fixes on the presigned-URL fix: - Sibling parity: the offline signing test covered GCS only, leaving Azure with no signing coverage at all (its evidence for S3 was the SeaweedFS suite, which CI skips). Renamed to test_file_storage_signed_urls.py and parametrised over GCS, S3 and Azure — all three sign offline with throwaway credentials, so CI now pins the contract for every backend. Confirms Azure needs no escaping, which the PR had left untested. - Added test_every_obstore_backend_is_covered: enumerates the storage package and fails if a backend is not in the signed list, so the next backend cannot land unsigned. - gcs.py comment now names Azure alongside S3 and points at the one test. - Hoisted the key_segment import to module scope in the S3 test.
nicoloboschi
force-pushed
the
fix/signed-url-percent-keys
branch
from
October 7, 2026 07:36
f08bd26 to
259b5b6
Compare
This was referenced Oct 8, 2026
Merged
This was referenced Oct 16, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Exporting a bank whose id contains a space, a dot, or non-ASCII characters (e.g.
Express AI — Trial) completes, but downloading the archive from the returneddownload_urlfails on GCS:Why:
key_segment), such a bank's files are stored under an object name that literally contains%. For example,…/banks/Express%20AI%20%E2%80%94%20Trial/exports/<uuid>/transfer.zipis the actual object name.Express AI — Trial/…, a different object, and returns 404.getandheadescape correctly, which is why storing and reading through the backend work and only the presigned URL is broken.The same applies to any presigned download on GCS, not only exports.
Fix
GCSFileStorage.get_download_urlescapes%as%25before signing:%are signed exactly as before.GCS only: obstore's S3 and Azure signers already escape
%, so pre-escaping there would double-encode. Both are now pinned by the test below.Testing
tests/test_file_storage_signed_urls.py: signs for real, offline — obstore's signers with throwaway credentials, so no network, bucket or container is needed — and asserts the URL path decodes back to the stored object name forplain-bank,Express AI — Trial,team.notesand100% done.test_every_obstore_backend_is_coveredenumerates the storage package, so a fourth backend cannot land without signing coverage.tests/test_file_storage_s3.py: the presigned-URL test previously only checked that the result was a URL string; it now downloads through the presigned URL against SeaweedFS for the same bank ids and asserts the bytes — the real-server proof behind the offline contract.main→ 404NoSuchKey;test_bank_file_storageandtest_file_storage_bytes_contractpass locally, plusruffandty check.