add support for leanchain genesis (devnet0)

[pk910]

This PR adds support for building leanchain genesis states for pq-devnet-0.

./bin/eth-genesis-state-generator leanchain --eth1-config ./genesis.json --config ./config.yaml --additional-validators ./validators.txt --json-output ./state.json --state-output ./state.ssz

most of these parameters are currently not used, but will probably be needed to build proper states in future.

Genesis state currently looks as simple as this:

{
  "config": {
    "num_validators": 3
  },
  "latest_justified": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "latest_finalized": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "historical_block_hashes": [],
  "justified_slots": [],
  "justifications_roots": [],
  "justifications_validators": []
}

[unnawut]

Thanks for lightning fast work!

To follow the original 3SF-mini impl strictly it would be this I think (initializing historical_block_hashes and justified_slots with slot0 block):

{
  "config": {
    "num_validators": 3
  },
  "latest_justified": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "latest_finalized": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "historical_block_hashes": [0x0000000000000000000000000000000000000000000000000000000000000000],
  "justified_slots": [true],
  "justifications_roots": [],
  "justifications_validators": []
}

cc @g11tech @syjn99 for confirmation

[syjn99]

Thanks @pk910 for your work!

@unnawut I agree with your comment. Here's some additional information we need to understand the genesis generation of 3SF-mini (and also I expect there's few inconsistency with current beacon tooling):

1. Genesis block starts with slot number 1. This is why historical_block_hashes and justified_slots have one element in the start.

• I didn't investigate too much whether we can just start with empty list of historical_block_hashes and justified_slots. Gonna take a look for next days.

2. get_current_slot() adds 2 after division with SLOT_DURATION, which is quite confusing in my sense.

At this moment, Ream strictly follows both assumption (Genesis generation / Slot calculation), but I'm keen to fix it if it is needed.

[pk910]

Heya @unnawut & @syjn99,

thanks a lot for your reviews :)
I've added the initial zero-hash & justified flag, so the genesis looks like this now:

{
  "config": {
    "num_validators": 3
  },
  "latest_justified": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "latest_finalized": {
    "root": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "slot": "0"
  },
  "historical_block_hashes": [
    "0x0000000000000000000000000000000000000000000000000000000000000000"
  ],
  "justified_slots": [
    true
  ],
  "justifications_roots": [],
  "justifications_validators": ""
}

[KolbyML]

Would it be fine to put a network field like the el/cl configs?, so we could be "devnet0" or just "dev" if we want to run a local instance without P2P? I need to ask @syjn99 if we are keeping the local thing

[pk910]

Something like this?
I think the config format is pretty much undefined so far. I just assumed there will be a similar architecture with a config.yaml and a preset.
It also looks like there is no peer discovery at the current stage anyway. All nodes are listed in a nodes.yaml with their ENR. local devnets would just need to add these local ENRs to the list..

[KolbyML]

@pk910 would it be possible to export the lean consensis config in YAML? as it would be nice to stay consistent with the beacon chain in that regard

[pk910]

the lean config is an input of this tool. there is currently an empty default config used, but that's just a placeholder for the unspecified config format :D
but yea, that's how it works, you supply a consensus and execution config and get the genesis state.
the consensus & execution configs are usually generated by the ethereum-genesis-generator on the layer above. It's basically just a template with some dynamic values.
However, given that the config format isn't really specified yet, I don't think it makes sense to add leanchain to the ethereum-genesis-generator yet.
It doesn't drive any execution client yet, so the el genesis is not needed, and the cl config is actually empty at this point :D

[KolbyML]

Something like this?

That would be nice

I think the config format is pretty much undefined so far. I just assumed there will be a similar architecture with a config.yaml and a preset. It also looks like there is no peer discovery at the current stage anyway. All nodes are listed in a nodes.yaml with their ENR. local devnets would just need to add these local ENRs to the list..

I need to check with @syjn99 and @unnawut but they added a 1 node devnet, I am not sure if they are planning to keep it or not, but if they are it would be nice to just distinguish it via a config_name

[KolbyML]

the lean config is an input of this tool. there is currently an empty default config used, but that's just a placeholder for the unspecified config format :D but yea, that's how it works, you supply a consensus and execution config and get the genesis state. the consensus & execution configs are usually generated by the ethereum-genesis-generator on the layer above. It's basically just a template with some dynamic values. However, given that the config format isn't really specified yet, I don't think it makes sense to add leanchain to the ethereum-genesis-generator yet. It doesn't drive any execution client yet, so the el genesis is not needed, and the cl config is actually empty at this point :D

Oh, yeah it is fine if it isn't add lean stuff isn't added toethereum-genesis-generator yet . I just assumed that we are defining the config format here myb.

[KolbyML]

Looks like the ephemery mode @syjn99 added was just for early testing and we will be removing it, so nvm

[g11tech]

there is no eth1 config yet

[g11tech]

there is no validators in state yet in devent0 as its without real signatures, but will come back on it if we want to add validators array still in state

[pk910]

I've extended the validators input format and added a enr generator.
the input validator-config.yaml looks like this:

shuffle: roundrobin
validators:
  - name: "ream_0"
    enr: "enr:-IS4QHCYrYZbAKWCBRlAy5zzaDZXJBGkcnh4MHcBFZntXNFrdvJjX04jRzjzCBOonrkTfj499SZuOh8R33Ls8RRcy5wBgmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQPKY0yuDUmstAHYpMa2_oxVtw0RW_QAdpzBQA8yWM0xOIN1ZHCCdl8"
    count: 3 # number of indices for this node

  - name: "zeam_0"
    privkey: "a94f5374fce5edbc8e2a8697c15331677e6ebf0b906c5aa9515b9c5cb11b1e0f"
    enrFields:
      ip: "172.20.0.100"
      tcp: 9000
      quic: 9001
      seq: 1
      whatever: "0x01000000"
    count: 3

  - name: "quadrivium_0"
    privkey: "c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9"
    enrFields:
      ip6: "2001:db8:85a3::8a2e:370:7334"
      tcp: 30303
      udp: 30303
      quic: 8080
      seq: 1
    count: 3

the tool also takes a config.yaml similar to the beaconchain config.yaml format:

# Genesis Settings
GENESIS_TIME: 1704085200

# Validator Settings  
VALIDATOR_COUNT: 0  # This will be updated

the tool can be run with this command:

mkdir genesis
./bin/eth-genesis-state-generator leanchain --config config.yaml --mass-validators validator-config.yaml --json-output genesis/state.json --state-output genesis/state.ssz --config-output genesis/config.yaml --nodes-output genesis/nodes.yaml --validators-output genesis/validators.yaml

this will generate these output files:

# config.yaml

# Genesis Settings
GENESIS_TIME: 1704085200

# Validator Settings  
VALIDATOR_COUNT: 9

# nodes.yaml

- enr:-IS4QHCYrYZbAKWCBRlAy5zzaDZXJBGkcnh4MHcBFZntXNFrdvJjX04jRzjzCBOonrkTfj499SZuOh8R33Ls8RRcy5wBgmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQPKY0yuDUmstAHYpMa2_oxVtw0RW_QAdpzBQA8yWM0xOIN1ZHCCdl8
- enr:-Jq4QLr_Fk_QGFra0TebsQCGJ8cq1c4ohik7TC0huKTiMSvQTSBSQLYKUoAQzwLHxbf71JQU8KnmgkHvFJAFn6myxMEBgmlkgnY0gmlwhKwUAGSEcXVpY4IjKYlzZWNwMjU2azGhA2hqUIfSG58w4lGPMiPp9llh1pjFuoSRUuoHmwNdHELwg3RjcIIjKIh3aGF0ZXZlcoQBAAAA
- enr:-KC4QD-aISxtgctnjeVxgrF5Qp-k2zvyMl5wI6bfzT1rGj3xd3y-AT9F6CmI3267KfBv2JpMPnR8J71At0hK8P2t_z8BgmlkgnY0g2lwNpAgAQ24haMAAAAAii4DcHM0hHF1aWOCH5CJc2VjcDI1NmsxoQOzU8YHzphhNhEGuB4bF8RTnh3Uy2CHNjBAXM8ppvSFDYN0Y3CCdl-DdWRwgnZf

# validators.yaml

quadrivium_0:
    - 2
    - 5
    - 8
ream_0:
    - 0
    - 3
    - 6
zeam_0:
    - 1
    - 4
    - 7

[g11tech]

this looks fairly complete to me, will give a deeper look

[syjn99]

@pk910 This seems nice, Ream will also prepare for those outputs.

[pk910]

There is now a docker image for this branch too: ethpandaops/eth-beacon-genesis:pk910-leanchain

you can run it like this:

mkdir -p genesis
docker run --rm -v .:/data -it ethpandaops/eth-beacon-genesis:pk910-leanchain leanchain \
  --config /data/config.yaml \
  --mass-validators /data/validator-config.yaml \
  --state-output /data/genesis/genesis.ssz \
  --json-output /data/genesis/genesis.json \
  --nodes-output /data/genesis/nodes.yaml \
  --validators-output /data/genesis/validators.yaml \
  --config-output /data/genesis/config.yaml

this needs a config.yaml & validator-config.yaml as described in the comment above and outputs the same files as described above.
just a bit easier for docker nerds as it avoids building or downloading a binary ;)

[GrapeBaBa]

I want to know what is the purpose of this file and how to consume it in the client.

# validators.yaml

quadrivium_0:
    - 2
    - 5
    - 8
ream_0:
    - 0
    - 3
    - 6
zeam_0:
    - 1
    - 4
    - 7

[g11tech]

this looks fairly complete to me, will give a deeper look

I want to know what is the purpose of this file and how to consume it in the client.

# validators.yaml

quadrivium_0:
    - 2
    - 5
    - 8
ream_0:
    - 0
    - 3
    - 6
zeam_0:
    - 1
    - 4
    - 7

explained in the team conversation

[GrapeBaBa]

I've extended the validators input format and added a enr generator. the input validator-config.yaml looks like this:

shuffle: roundrobin
validators:
  - name: "ream_0"
    enr: "enr:-IS4QHCYrYZbAKWCBRlAy5zzaDZXJBGkcnh4MHcBFZntXNFrdvJjX04jRzjzCBOonrkTfj499SZuOh8R33Ls8RRcy5wBgmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQPKY0yuDUmstAHYpMa2_oxVtw0RW_QAdpzBQA8yWM0xOIN1ZHCCdl8"
    count: 3 # number of indices for this node

  - name: "zeam_0"
    privkey: "a94f5374fce5edbc8e2a8697c15331677e6ebf0b906c5aa9515b9c5cb11b1e0f"
    enrFields:
      ip: "172.20.0.100"
      tcp: 9000
      quic: 9001
      seq: 1
      whatever: "0x01000000"
    count: 3

  - name: "quadrivium_0"
    privkey: "c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9"
    enrFields:
      ip6: "2001:db8:85a3::8a2e:370:7334"
      tcp: 30303
      udp: 30303
      quic: 8080
      seq: 1
    count: 3

the tool also takes a config.yaml similar to the beaconchain config.yaml format:

# Genesis Settings
GENESIS_TIME: 1704085200

# Validator Settings  
VALIDATOR_COUNT: 0  # This will be updated

the tool can be run with this command:

mkdir genesis
./bin/eth-genesis-state-generator leanchain --config config.yaml --mass-validators validator-config.yaml --json-output genesis/state.json --state-output genesis/state.ssz --config-output genesis/config.yaml --nodes-output genesis/nodes.yaml --validators-output genesis/validators.yaml

this will generate these output files:

# config.yaml

# Genesis Settings
GENESIS_TIME: 1704085200

# Validator Settings  
VALIDATOR_COUNT: 9

# nodes.yaml

- enr:-IS4QHCYrYZbAKWCBRlAy5zzaDZXJBGkcnh4MHcBFZntXNFrdvJjX04jRzjzCBOonrkTfj499SZuOh8R33Ls8RRcy5wBgmlkgnY0gmlwhH8AAAGJc2VjcDI1NmsxoQPKY0yuDUmstAHYpMa2_oxVtw0RW_QAdpzBQA8yWM0xOIN1ZHCCdl8
- enr:-Jq4QLr_Fk_QGFra0TebsQCGJ8cq1c4ohik7TC0huKTiMSvQTSBSQLYKUoAQzwLHxbf71JQU8KnmgkHvFJAFn6myxMEBgmlkgnY0gmlwhKwUAGSEcXVpY4IjKYlzZWNwMjU2azGhA2hqUIfSG58w4lGPMiPp9llh1pjFuoSRUuoHmwNdHELwg3RjcIIjKIh3aGF0ZXZlcoQBAAAA
- enr:-KC4QD-aISxtgctnjeVxgrF5Qp-k2zvyMl5wI6bfzT1rGj3xd3y-AT9F6CmI3267KfBv2JpMPnR8J71At0hK8P2t_z8BgmlkgnY0g2lwNpAgAQ24haMAAAAAii4DcHM0hHF1aWOCH5CJc2VjcDI1NmsxoQOzU8YHzphhNhEGuB4bF8RTnh3Uy2CHNjBAXM8ppvSFDYN0Y3CCdl-DdWRwgnZf

# validators.yaml

quadrivium_0:
    - 2
    - 5
    - 8
ream_0:
    - 0
    - 3
    - 6
zeam_0:
    - 1
    - 4
    - 7

@pk910 @unnawut @syjn99 @KolbyML @g11tech I saw the ENR in the example files for Ream and Quadrivium are not consistent with spec, since Zeam currently only support the format in the spec so that I got issues when testing. Is that just example error, Ream and Quadrivium will use the same format in the spec, right?

[pk910]

Heya @GrapeBaBa,
Your tests are complaining about too many or unknown fields in the ENRs, right?
If so, then yea, these were just example ENRs with various fields to showcase the generic approach of ENR generation.
These should match the current lean spec:

# validator-config.yaml

shuffle: roundrobin
validators:
  - name: "ream_0"
    privkey: "a94f5374fce5edbc8e2a8697c15331677e6ebf0b906c5aa9515b9c5cb11b1e12"
    enrFields:
      ip: "172.20.0.1"
      quic: 8080
      seq: 1
    count: 3

  - name: "zeam_0"
    privkey: "a94f5374fce5edbc8e2a8697c15331677e6ebf0b906c5aa9515b9c5cb11b1e0f"
    enrFields:
      ip: "172.20.0.2"
      quic: 8080
      seq: 1
    count: 3

  - name: "quadrivium_0"
    privkey: "c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9d4b8b6b8c2bbdac5e876b3e9"
    enrFields:
      ip: "172.20.0.3"
      quic: 8080
      seq: 1
    count: 3

ENRs:

- enr:-IW4QA0pljjdLfxS_EyUxNAxJSoGCwmOVNJauYWsTiYHyWG5Bky-7yCEktSvu_w-PWUrmzbc8vYL_Mx5pgsAix2OfOMBgmlkgnY0gmlwhKwUAAGEcXVpY4IfkIlzZWNwMjU2azGhA6mw8mfwe-3TpjMMSk7GHe3cURhOn9-ufyAqy40wEyui
- enr:-IW4QNx7F6OKXCmx9igmSwOAOdUEiQ9Et73HNygWV1BbuFgkXZLMslJVgpLYmKAzBF-AO0qJYq40TtqvtFkfeh2jzqYBgmlkgnY0gmlwhKwUAAKEcXVpY4IfkIlzZWNwMjU2azGhA2hqUIfSG58w4lGPMiPp9llh1pjFuoSRUuoHmwNdHELw
- enr:-IW4QOh370UNQipE8qYlVRK3MpT7I0hcOmrTgLO9agIxuPS2B485Se8LTQZ4Rhgo6eUuEXgMAa66Wt7lRYNHQo9zk8QBgmlkgnY0gmlwhKwUAAOEcXVpY4IfkIlzZWNwMjU2azGhA7NTxgfOmGE2EQa4HhsXxFOeHdTLYIc2MEBczymm9IUN

[GrapeBaBa]

@pk910 I found the genesis_time is always 1704085200, the tool will not change it at all. Is that expect?

[pk910]

@GrapeBaBa
Yea, this is expected. This tool doesn't generate a config.yaml on its own. It 's a bit pointless doing so, as it'd still need all the values as cli parameters. Instead, it takes a existing config.yaml and updates the validator count inside. All other values are just passed through as is (incl. comments).
So the genesis time needs to be updated in the config before calling the tool with it. It actually also doesn't matter if the timestamp is specified in seconds or milliseconds, as the generator does not use the value for anything.

Creating this config.yaml manually upfront shouldn't be a big hassle. We can even automate the genesis time value with a simple bash script that sed-replaces the timestamp before calling this tool.
In a later stage, the config.yaml is generated by the ethereum-genesis-generator, which contains appropriate templates. But yea, that seems a bit unnecessary for a config with 2 properties :D

[GrapeBaBa]

@GrapeBaBa Yea, this is expected. This tool doesn't generate a config.yaml on its own. It 's a bit pointless doing so, as it'd still need all the values as cli parameters. Instead, it takes a existing config.yaml and updates the validator count inside. All other values are just passed through as is (incl. comments). So the genesis time needs to be updated in the config before calling the tool with it. It actually also doesn't matter if the timestamp is specified in seconds or milliseconds, as the generator does not use the value for anything.

Creating this config.yaml manually upfront shouldn't be a big hassle. We can even automate the genesis time value with a simple bash script that sed-replaces the timestamp before calling this tool. In a later stage, the config.yaml is generated by the ethereum-genesis-generator, which contains appropriate templates. But yea, that seems a bit unnecessary for a config with 2 properties :D

Got it.