CIP-0165? | Canonical Ledger State

[qnikst]

This proposal defines the Simple Canonical Ledger State (SCLS), a stable, versioned, and verifiable file format for representing the Cardano ledger state. It specifies a segmented binary container with deterministic CBOR encodings, per-chunk commitments, and a manifest that enables identical snapshots across implementations, supports external tools (e.g., Mithril), and future-proofs distribution and verification of state.

---

(rendered latest document)

[rphair]

Tagging Triage for initial presentation at next CIP meeting: https://hackmd.io/@cip-editors/118

Note it's impossible this will be given a CIP number 015x as the directory is currently named (at this time the candidate list goes up to 0161), but you could leave the directory name the same since there wouldn't be any perceived naming conflicts.

[qnikst]

@rphair thanks for the fast feedback, comments, suggestions and explanation of the reason behind them.
I've updated the CIP to reflect all the comments and added some more headers where I expect linking is important!

[rphair]

@qnikst thanks but please you should avoid force-pushing again here. This is an extensive proposal and it would detract from the review process to have to keep re-reviewing the whole document each time we lose the change & suggestion history in this branch.

Not these least of the problems we'd keep having is that (as you can see at this time) all the review threads above remain unresolved. If you accept these changes on GitHub and pull the changes back to your local branch, the issues above will be marked resolved and then editors, co-authors, and Ledger reviewers won't have to go over them all again later.

I appreciate your first-time submission and look forward to an enthusiastic review of this one, but FYI we've already asked authors please not to force push here in CIP-0001: https://github.com/cardano-foundation/CIPs/blob/master/CIP-0001/README.md#1-early-stages

[rphair]

p.s. @qnikst thanks for confirming resolution of the presentation issues above... I think @WhatisRT @lehins now that the format is more canonicalised I think it could be ready for at least your initial review. I would personally be in favour of confirming this as a CIP candidate (i.e. assigning a number) but if you can post any initial reactions in the next 2½ hours we'll also go over these at the meeting.

[rphair]

@qnikst the recent changes look great (especially the deeper level headers); currently I think the only presentation issues are that the Markdown footnotes are not doing what I think you intended them to do...

see, for correct usage: https://docs.github.com/en/enterprise-cloud@latest/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#footnotes

[rphair]

@qnikst in the CIP meeting today we decided to leave this in Triage since other CIP editors had interested & critical reactions to it: but it came in shortly before the meeting so there hadn't been time to properly formulate reservations about it. This tag will keep this submission at the top of the editors' stack and help get those reservations documented ASAP so you can start working with them.

One bit that came from the meeting discussion is that CBOR is more confining than JSON would be: especially when canonical (cc @Crypto2099 to elaborate perhaps). With other languages having better support for JSON objects, perhaps this specification could be made in JSON, or with JSON in parallel with CBOR.

[qnikst]

@rphair thanks for updating the status!

I’ve added a CBOR vs JSON section under Implementation Alternatives, which explains the reasoning behind the current choice and the proposed direction.

We are not opposed to defining schemas using json-schema, especially given the ongoing work around CIP-139 where shared efforts could be valuable. However, our current focus is on CBOR, but we are looking at schema and plan to keep data compatible. At this point we believe that json-schema specification could be derived from the CBOR definition at a later stage.

At this point, we do not see strong reasons to prioritize a standalone json-schema description, but we would very much welcome feedback if others believe there are benefits to doing so earlier.

[qnikst]

We have checked the comments, and marked as resolved the ones that have an explicit reply or update. Now I believe that everything is resolved.

There are few that left, that I would like to mention explicitly, for this points I would prefer to have explicit agreement:

1. should we move the information about the current slot/era/block to it's own block (or to metadata block), I would personally prefer the latter and it would be pretty straightforward change

2. seems we need to clarify information about the multiple files.

3. the question raised during the implementation: currently for the pparams we use human readable field names, alternative approach is to use tags from the on-chain protocol, both approaches have it's own proc and cons, and at this point we have not enough experience to make a final decision. Until there will be a strong opinion on that matter we would stick to human readable names (to keep the status quo)

4. Add key sizes to the namespaces file (either blocks or header). Adding to block is simpler from the specification perspective.

[rphair]

thanks @qnikst & we'll hold this at Last Check until at least confirming that questions on your list in #1083 (comment) have been resolved.

[qnikst]

Weekly update :)

Key changes:

• After several discussions we have finally moved from names to tags to persist the on-chain structure. So CDDL were updated as well as proposal itself.
• We moved the slot_no from header to manifest block. This is a nice and definite place that simplifies implementation.
• We didn't add era and other info. If one find it useful we can introduce a namespace for that info.
• We have moved key size to the blocks. Specification and implementation ware updated accordingly.

Questions that still require clarification:

• agree on approach to multiple files (we are in the process of communication about that)

confirmed response to @perturbing's proofreading #1083 (review) is 100% complete

[rphair]

agree on approach to multiple files (we are in the process of communication about that)

@qnikst perhaps the issue below is related to the "multiple files" and is therefore still being worked out (?) ...

[rphair]

Questions that still require clarification:

• agree on approach to multiple files (we are in the process of communication about that)

@qnikst — for our CIP meeting beginning now — has this question been resolved to your satisfaction?

[qnikst]

Oh today I'm not that. I planned to address @sandtreader comment about multiple files. While the format allows that and there is an answer how to build merkle tree a top of the structure I plan to remove multiple files from the CIP because it introduces an additional complexity for no reason

[qnikst]

@rphair ok from me, this question was resolved.

UPD I've rechecked CIP, so now the question is resolved. All the details about multiple files left only in the alternative implementation section.

[rphair]

@qnikst we've not merged this as the CIP meeting today because of your more recently requested review from @sandtreader — once that's addressed the editors would be OK to merge this at the next opportunity.

(edit, p.s.) if you were only tagging him because of the "requested change" which is still showing on GitHub, FYI we would be able to merge this even if that old review is not dismissed.

[qnikst]

@rphair ah I would be happy to get this PR merged. Just didn't want to step on others toes and was not sure about the process. I don't see a good reasons do not merge.

[rphair]

@qnikst we can merge now if OK with @perturbing (i.e. if editors are unanimous - cc @Ryun1)

[rphair]

@qnikst it's just over a week before the next CIP meeting. I'm happy to see @Xophmeister coming into the authors list with 9262ffb although obviously (cc @Ryun1 @perturbing) the CIP editors would like to be sure the document stops changing on any given day, with no announcement or comment, while we are considering merging this.

So please notify anyone who might be working on this material that we can't (or at least shouldn't be) merging this unless the document appears to be stable. You could also post some definitive comment here like: "Please only commit urgent changes here, in this week of final review, and save any future changes for a PR after this baseline is merged" — currently targeted for this next CIP meeting: https://hackmd.io/@cip-editors/129

[qnikst]

@rphair my mistake! I didn't expect that that update will appear on this branch without additional communication and questions from my side here. But long story short we are working on the rust library and during implementation we have found out that there were some missing pieces about the implementation of the algorithms and so with CIP alone some details were underspecified.

That patch fixes the exactly this problem. I've explained the situation to the team and we are not expecting to have anything else to come in this revision.