Merged
Conversation
graydon
previously approved these changes
Nov 22, 2024
31d3d24 to
003ab36
Compare
003ab36 to
375cb37
Compare
graydon
previously approved these changes
Nov 22, 2024
5762fe4 to
51f7d0d
Compare
51f7d0d to
81aafab
Compare
graydon
previously approved these changes
Nov 22, 2024
SirTyson
reviewed
Nov 22, 2024
SirTyson
reviewed
Nov 22, 2024
81aafab to
5c63e7b
Compare
SirTyson
reviewed
Nov 22, 2024
SirTyson
previously approved these changes
Nov 22, 2024
Contributor
SirTyson
left a comment
There was a problem hiding this comment.
LGTM after the MODE_DISABLES_FSYNC change.
SirTyson
approved these changes
Nov 22, 2024
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the publishing mechanism by ensuring proper fsync of checkpoint files and gracefully handling situations where publishing is enabled mid-checkpoint. The key changes include:
- Switching HAS checkpoint files from JSON to binary format with proper fsync support
- Adding logic to wait for the next checkpoint boundary when publishing is enabled mid-checkpoint
- Adding debugging tools to inspect checkpoint files and the publish queue
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/util/XDRStream.h | Added documentation about durable write requirements and updated comments for clarity |
| src/util/BufferedAsioCerealOutputArchive.h | Added <cereal/types/string.hpp> include to support string serialization |
| src/main/dumpxdr.cpp | Updated regex to support .dirty suffix for checkpoint files |
| src/main/CommandLine.cpp | Added print-publish-queue command and necessary cereal vector include |
| src/history/HistoryManagerImpl.cpp | Changed checkpoint file format from JSON to binary with fsync, updated file extensions from .json to .checkpoint, and added loadCheckpointHAS helper |
| src/history/CheckpointBuilder.h | Added mPublishWasDisabled flag and changed ensureOpen return type to bool |
| src/history/CheckpointBuilder.cpp | Implemented logic to detect and handle mid-checkpoint publishing enablement, with proper checkpoint boundary detection |
| IMPORTANT: some areas of core require durable writes that | ||
| are resistant to application and system crashes. If you need durable writes: | ||
| 1. Use a stream implementation that supports fsync, e.g. OutputFileStream | ||
| 2. Write to a temp file first. If you don't intent to persist temp files across |
There was a problem hiding this comment.
Typo in comment: "intent" should be "intend".
Suggested change
| 2. Write to a temp file first. If you don't intent to persist temp files across | |
| 2. Write to a temp file first. If you don't intend to persist temp files across |
Comment on lines
121
to
122
| // Immediately produce a final checkpoint JSON (suitable for confirmed | ||
| // ledgers) |
There was a problem hiding this comment.
The comment still references "checkpoint JSON" but the file format has been changed to binary. The comment should be updated to reflect this change.
Copilot AI
added a commit
that referenced
this pull request
Jan 7, 2026
Co-authored-by: anupsdf <127880479+anupsdf@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains two changes:
Additionally, I added a few more thing to aid debugging, such as printing checkpoint files with dump-xdr and an offline command to dump all checkpoints scheduled to publish.