Repository navigation
fix(dev-server): cap request bodies and retain at most 10k debug events - #846
Conversation
Co-Authored-By: wchieng@launchdarkly.com <williamchieng1991@gmail.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
| // maxDebugEvents bounds the debug_events table; older rows are deleted as new ones arrive. | ||
| const maxDebugEvents = 10000 | ||
|
|
There was a problem hiding this comment.
I'm surprised we have a 'debug events' table honestly
There was a problem hiding this comment.
The table is used by the dev server's event debugger. When someone opens the event view (/dev/events/tee), sdkEventObserver in api/events/events_stream.go creates a debug_session and writes every SDK event the dev server receives to debug_events. The UI can then page through events per session, and debug_session rows are cleaned up when they no longer have any events.
Before this PR, nothing ever deleted rows from it. While the debugger was open, a noisy SDK, or anything else posting to /bulk, could grow the SQLite file without limit. The 10k cap deletes the oldest rows on insert. It applies across all sessions combined, not per session.
If you think the feature isn't worth keeping, removing it would be a separate PR. This one only puts a bound on it.
|
@BugBot review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ff01b3a. Configure here.
|
got some complaints about this one: 'POST /dev/backup still fills the disk. RestoreToFile copies the body into a new temp file and does not remove it when the copy or the validation fails, so each request leaves another file of up to 256 MiB. The same request shape as /bulk (text/plain, no preflight) can do this in a loop. The debug-event cap only bounds the row count. Each row can be as large as the 10 MiB body limit, and the delete is global across sessions, so 10,000 rows is still on the order of 100 GiB.' |
…stores, require sqlite content type for restore Co-Authored-By: wchieng@launchdarkly.com <williamchieng1991@gmail.com>
|
Both complaints were valid. Fixed in 5afc430:
Debug event row cap doesn't bound size. Tests added: restore with an invalid database leaves |
Summary
This caps dev-server request bodies and caps how many debug events are kept on disk.
Before this change, any caller could send unlimited data:
/bulkand/mobile/events/bulkread the whole body into memory withio.ReadAll.POST /dev/backupcopied the whole body into a temp file./devhandlers decoded JSON without any size limit./events/teeis open, every event is written todebug_events, which never deleted old rows.A web page can POST
text/plainto/bulkwithout a CORS preflight. Together, that meant a page could fill the developer's disk.Changes:
limitRequestBodyrouter middleware applieshttp.MaxBytesReaderto every route: 10 MiB by default, 256 MiB forPOST /dev/backup.SdkEventsReceiveHandlerreturns 413 when the body is too large. Before, it returned an implicit 200.Sqlite.WriteEventdeletes rows older than the newest 10,000 after each insert:Where the limits come from:
Related: #845 changes the default bind address to
127.0.0.1.Testing
TestLimitRequestBody, which checks requests at and just over each limit, andTestWriteEventRetention.go test ./internal/dev_server/...passes.Link to Devin session: https://app.devin.ai/sessions/33518e0108ac4ea3aaec26dd594dbd62
Open in Devin Desktop: https://app.devin.ai/desktop/session/33518e0108ac4ea3aaec26dd594dbd62?variant=devin
Requested by: @wchieng
Note
Overview
Adds global request body limits on the dev server via
limitRequestBodymiddleware: 10 MiB by default and 256 MiB forPOST /dev/backup, with backup restore requiringapplication/vnd.sqlite3so simple cross-origin POSTs cannot hit restore without a preflight.SDK bulk event ingestion now responds with 413 when the capped body is exceeded instead of failing silently.
Debug event persistence trims
debug_eventsafter each insert: at most 10,000 rows and 64 MiB of stored event payload (whichever bound bites first), with tests for count- and byte-based retention.Failed SQLite restores delete their temp file in
RestoreToFile, with a test that TMPDIR stays empty after invalid input.Reviewed by Cursor Bugbot for commit 5afc430. Bugbot is set up for automated code reviews on this repo. Configure here.