Support standart canonical state instances#5486
Support standart canonical state instances#5486qnikst wants to merge 37 commits intoIntersectMBO:masterfrom
Conversation
In order to be able to export and import ledger data to CIP-165 [1]. The plan is to have `scls` public library for each era that supports it, starting with Conway. The scls library is structured using namespaces that CIP-165 proposes and has instances for all types. In order to not reimplemnet too many CBOR encodings for the types that already have instance they were implemented via LedgerCBOR. In order to check conformance to SCLS spec CDDL conformance and canonicity conformance tests were introduced, so it's possible to develop ledger types and state without looking at scls package and rely on typechecker and tests. [1] cardano-foundation/CIPs#1083
Prepare an example of using scls. Example provides
abilities to:
- read binary utxo and state files (generated by cardano-cli)
and reencode in in scls
- read and store the file as scls
This utility is useful as an example and for testing
scls canonicity.
lehins
left a comment
There was a problem hiding this comment.
Just a few quick comments after taking a peek at the PR.
I am not really confident about this whole structure. I need to give it some thought.
Maybe it would be safer and clearer if there was a concrete ledger state Haskell data type for what is being stored in the canonical ledger state.
Then there would be a step that converts to and from that data type, which is ultimately stored in a canonical CBOR form.
I know this work is still in an early stage, but I'd like to flash out the design before we put a whole lot of work in.
Could I please ask you to remove all redundant parenthesis in this PR. They are driving me nuts. Otherwise I'll end up posting hundreds of suggestions with their removal.
I'll try to spend some time later this week trying to dig deeper into understanding what exactly is being proposed here.
| import Data.Typeable (Typeable) | ||
| import Data.Word (Word8) | ||
|
|
||
| instance {-# OVERLAPPING #-} ToCanonicalCBOR v (CompactForm Coin) where |
There was a problem hiding this comment.
I would not let overlapping instances into this repo. They are trouble and there usually are ways to avoid them. In these examples, I think, those pragmas are totally unnecessary, since they aren't seem to overlap anything
|
|
||
| deriving via (LedgerCBOR v (EpochNo)) instance ToCanonicalCBOR v (EpochNo) | ||
|
|
||
| deriving via (LedgerCBOR v (EpochNo)) instance FromCanonicalCBOR v (EpochNo) |
There was a problem hiding this comment.
Redundant parenthesis her and everywhere else.
| deriving via (LedgerCBOR v (EpochNo)) instance FromCanonicalCBOR v (EpochNo) | |
| deriving via (LedgerCBOR v EpochNo) instance FromCanonicalCBOR v EpochNo |
cabal.project
Outdated
| source-repository-package | ||
| type: git | ||
| subdir: scls-cbor | ||
| location: https://github.com/tweag/cardano-canonical-ledger.git |
There was a problem hiding this comment.
I'd suggest renaming the repo, otherwise it sounds like there is Canonical and Non-Cononical Cardano Ledger 😅
| location: https://github.com/tweag/cardano-canonical-ledger.git | |
| location: https://github.com/tweag/cardano-canonical-ledger-state.git |
There was a problem hiding this comment.
Yes! Fixed that though renamed that to cardano-cls in order not to clash name with cardano-ledger.
cabal.project
Outdated
|
|
||
| source-repository-package | ||
| type: git | ||
| subdir: scls-cbor |
There was a problem hiding this comment.
Wouldn't be easier to just list them all?
| subdir: scls-cbor | |
| subdir: | |
| scls-cbor | |
| scls-format | |
| ... |
I mean, not that it matters, since we don't merge PRs with SRPs, but even for development purposes it feels like it woulb simpler
| testlib, | ||
|
|
||
| test-suite tests | ||
| test-suite testtestss |
There was a problem hiding this comment.
Accidental stuttering
| test-suite testtestss | |
| test-suite tests |
| instance ToCanonicalCBOR v (PParamsUpdate ConwayEra) where | ||
| toCanonicalCBOR v (PParamsUpdate ConwayPParams {..}) = | ||
| encodeAsMap | ||
| [ mkEncodablePair v ("a0" :: Text) (unTHKD cppA0) |
There was a problem hiding this comment.
Instead of using names for protocol parameters, we should be using Tags, same as in CBOR that is used on chain:
cardano-ledger/eras/conway/impl/cddl/data/conway.cddl
Lines 587 to 616 in 0f0168b
Names can change, while tags cannot.
| @@ -0,0 +1,138 @@ | |||
| {-# LANGUAGE AllowAmbiguousTypes #-} | |||
| {-# LANGUAGE DataKinds #-} | |||
| {-# LANGUAGE ImportQualifiedPost #-} | |||
There was a problem hiding this comment.
Consistency in the codebase is key. We don't want to mix different styles in the same repo. (FTR. I am all in favor of the extension itself in general)
| {-# LANGUAGE ImportQualifiedPost #-} |
|
Yes, definitely feel free to think about it as much as you need. We do not expect that quite a large PR that was done without prior discussions will be merged as-is immediately.
I'm not opposed to having such state especially if it makes sense for the However, there are reasons for having this set of instances defined. In SCLS CIP we planned the protocol to have two properties: I would take that for the ledger those requirements may make not much sense but implemented instances may help. Also such type would lead to more entanglement between As for the names vs tags, we did as in the first approach in the CIP (not yet accepted) and use names as meaningful entities, so they will not change unless the spec changes. And they were planned to be a sort of documentation that does depend on the protocol. But having those tags on the wire is definitely a better guarantees than CIP process as does not depend on the people. Thus switching to tags makes total sense (the only downside is that consumer will have to consult with the CDDL about what that field is and writing automated codes (if language supports them) can be harder). So regarding this PR, I'm going to do following:
|
We no longer define overlapping instances for the CompactForm X type, so OVERLAPPING instances can be dropped. Thanks to the Alexey Kuleshevich for noticing that.
Remove redundant parenthesis all over the project.
|
I think I've addressed all the comments except for names vs tags. After a short discussion with a team we decided that it would be right to put that into the alternatives section in CIP and get a final agreement in CIP. As CDDL interface for the SCLS data is defined by the CIP. Both tags and names have those pros and cons, but we decided to give a chance for the consumers to vote |
cardano-cls introduced additional namespace that is required to bootstrap a node. This commit introduces necessary support on the cardano-ledger side.
|
@lehins have you chances to think about the changes? From the last update there were few more changes from our side:
I would love to hear more feedback on this branch and if there are other updates we should proceed with. |
Because CoinPerByte is a newtype wrapper over the basic type we are not reusing that in internal proxy types, instead we convert between the types in the conversion functions.
4e5d68e to
fb21740
Compare
|
Finally we have managed to update the structure of the PR, it includes:
In addition we have updated the repo to the latest Our next step would be, as discussed to merge current master and update the work if needed. But it's possible to check the PR at this moment, because changes may be a bit opinionated and do not fit the general style of the project. CC @lehins |
lehins
left a comment
There was a problem hiding this comment.
I am really sorry, but this PR is way too big for a sensible review.
Let's take a step back and restart small and then I believe we can make progress. Could you please split out parts of this PR into a separate smaller PR that handles creation of canonical state that contains only UTxO and nothing else
Also there is no need to create canonical types for any of the types that live on-chain and stored in the state. For example TxOut (and everything it contains) is one of those type, they are guaranteed not to change Haskell representation (or serialization for that matter) for a specific era ever. This is very important to avoid unnecessary duplication. Same applies to PParams, if they are to use tags instead of names, hence my push for using tags, otherwise we will end up with this duplication for every single era.
Overall, it does look like good work and direction seems to be reasonable, but there is no way we are merging a huge chunk of code lime this, because there is no way that we will be able to maintain it without completely rewriting it later. So, if we want to design something maintainable by the Ledger team we need to do this work one step at a time, like all other work that we do.
|
@lehins thanks for starting the work on reviewing and helping us with finding a way of getting things merge! While I have to express that this is a bit afflicting (as it would require to reorganise our plans) but totally understandable decision, and I agree that it will be a reasonable way forward. Just wanted to express a hope that we will be able to have a review on the future PRs in a timely manner, in that case it would be possible to get things merged in a reasonable timeframe without much pressure on either team. To summarise the next actions we will do:
I would suggest to start with something more simple than |
It is much easier and quicker to review smaller, self contained PRs. That is precisely my motivation, behind the recommendation of splitting this work up, so we can actually review them in timely manner and get them merged, instead of be stuck in review process for months.
Yes, please.
In order to avoid redundant work, let's start with one PR, instead of a whole chain that will depend on each other. Using blocks namespace is also a fine choice for an initial PR. The reason why I was saying UTxO namespace would be a good contender is because:
Perfect. |
| @@ -0,0 +1,125 @@ | |||
| cabal-version: 3.0 | |||
| name: cardano-canonical | |||
There was a problem hiding this comment.
Let's rename this package in a way that follows the convention and is a bit more clear what it means. It's not a big deal that it is longer.
| name: cardano-canonical | |
| name: cardano-ledger-canonical-state |
| time, | ||
| tree-diff, | ||
|
|
||
| library scls |
There was a problem hiding this comment.
We try to avoid abbreviations as much as possible
| library scls | |
| library canonical-state |
|
|
||
| library scls | ||
| exposed-modules: | ||
| Cardano.Ledger.Conway.SCLS |
There was a problem hiding this comment.
I think this will be a better prefix. Again abbreviations are not really saving time.
| Cardano.Ledger.Conway.SCLS | |
| Cardano.Ledger.Conway.CanonicalState |
|
|
||
| library | ||
| exposed-modules: | ||
| Cardano.Ledger.SCLS.BaseTypes |
There was a problem hiding this comment.
Same here. Also wanted to point out that having Ledger and SL(edger)`S in the name is the duplication that bugs me a little 😀
| Cardano.Ledger.SCLS.BaseTypes | |
| Cardano.Ledger.CanonicalState.BaseTypes |
There was a problem hiding this comment.
In case you are worried of loosing Standard part, don't, we in ledger often have a different name that what the corresponding CIP defines the concept as.
| -- ^^^^^^^ | ||
| -- Namespace | ||
| -- @ | ||
| module Cardano.Ledger.SCLS.LedgerCBOR ( |
There was a problem hiding this comment.
It can just be CBOR, we are already in Ledger See my other comment for the reason about the name
| module Cardano.Ledger.SCLS.LedgerCBOR ( | |
| module Cardano.Ledger.CanonicalState.EraCBOR ( |
There was a problem hiding this comment.
Nevermind, I see why was called LedgerCBOR. This whole concept is actually incorrect and the name is not good either. I'll make a comment on the actual type
| -- ^^^^^^^ | ||
| -- Namespace | ||
| -- @ | ||
| module Cardano.Ledger.SCLS.LedgerCBOR ( |
There was a problem hiding this comment.
Nevermind, I see why was called LedgerCBOR. This whole concept is actually incorrect and the name is not good either. I'll make a comment on the actual type
| newtype LedgerCBOR (v :: Symbol) a = LedgerCBOR {unLedgerCBOR :: a} | ||
| deriving (Eq, Show) | ||
|
|
||
| instance EncCBOR a => ToCanonicalCBOR v (LedgerCBOR v a) where | ||
| toCanonicalCBOR _v (LedgerCBOR a) = assumeCanonicalEncoding $ toPlainEncoding (natVersion @9) (encCBOR a) |
There was a problem hiding this comment.
This is very wrong and dangerous, because any instance that will be derived using it will be forever stuck in Conway. I also changed the name to reflect it better. Here how it should look like:
| newtype LedgerCBOR (v :: Symbol) a = LedgerCBOR {unLedgerCBOR :: a} | |
| deriving (Eq, Show) | |
| instance EncCBOR a => ToCanonicalCBOR v (LedgerCBOR v a) where | |
| toCanonicalCBOR _v (LedgerCBOR a) = assumeCanonicalEncoding $ toPlainEncoding (natVersion @9) (encCBOR a) | |
| newtype EraCBOR era (v :: Symbol) a = EraCBOR {unEraCBOR :: a} | |
| deriving (Eq, Show) | |
| instance (Era era, EncCBOR a) => ToCanonicalCBOR v (EraCBOR era v a) where | |
| toCanonicalCBOR _v (LedgerCBOR a) = assumeCanonicalEncoding $ toEraCBOR @era |
|
Thanks for the deep review! I'm converting this PR to draft as it will be superseded by the smaller ones. |
Description
This MR adds support for canonical ledger state instances for the types that are using in canonical ledger state according to the CIP-0165.
To support SCLS instances for canonical types are introduced together with a test-suite.
This PR is done for the early review as for example it still uses scls libraries that are not in CHaP, we are going to add them once discussion questions about this PR will be resolved (if there will be questions).
Checklist
CHANGELOG.mdfiles updated for packages with externally visible changes.NOTE: New section is never added with the code changes. (See RELEASING.md).
.cabalandCHANGELOG.mdfiles when necessary, according to theversioning process.
.cabalfiles updated when necessary.NOTE: If bounds change in a cabal file, that package itself must have a version increase. (See RELEASING.md).
scripts/fourmolize.sh).scripts/cabal-format.sh).scripts/gen-cddl.sh)hie.yamlupdated (usescripts/gen-hie.sh).