v1 Label Healing Strategy

[lightwalker-eth]

Background

1. The ENS Subgraph uses https://github.com/graphprotocol/ens-rainbow for healing unknown labels.
2. Placeholder: Verified backwards compatibility with ENS Subgraph #51 requires us to implement an equivalent label healing strategy as the ENS Subgraph.
3. Plans are under development / discussion with @djstrong for how we can propose a new ENS protocol standard for dynamically growing the set of healed labels.
   1. To help stay focused on shipping the "v1" release of ENSNode, this goal should be processed separately and is generally outside the scope of this issue.

Key Implementation Constraints

1. Steps for someone to deploy their own instance of ENSNode should be kept as simple as possible.
2. Ideally, when ENSNode starts up it would be able to detect if it has successfully "synced" the rainbow tables.
   1. This operation should be implemented carefully, such that ENSNode can recover from any possible crash / inconsistent state while a sync is in progress but not 100% completed.
   2. Assuming that the process of initializing all rainbow table state into a new ENSNode deployment could take some time. We should provide operational visibility into this process at both the command-line level, log level, and ideally also at an API level to support goals that @tk-o will be advancing for a "Realtime Dashboard" of ENSNode state.
3. The rainbow tables need to be fully synced before any indexing can correctly begin.
4. Very open to applying some simplifications where we can in our V1. For example: very open to assuming for now that the set of rainbow table records is immutable. Advice appreciated.
5. Operations that exceed the limits of a non-Enterprise plan on Railway need to be avoided. For example, not sure if unzipping a multi gb file would cause out of memory errors. Hopefully not.
6. When we begin our internal development its fine to make use of the cloud hosted files referenced at https://github.com/graphprotocol/ens-rainbow. Suggest we provide our own hosted copy of these files for the use of others who run ENSNode. Please ask @BanaSeba to support this goal. For example, he can put a copy of these files into a public S3 bucket for us, or whatever strategy might be proposed.
7. While we advance this task, please look for opportunities to refine our README. For example:
   1. Identify the value proposition of how we automatically sync rainbow tables.
   2. Identify any additional "Getting Started" steps.
   3. Identify any additional environment variables (if any).
   4. Refine any comments about initial sync / indexing times.
   5. Identify any special considerations for someone running their own ENSNode instance. For example:
      1. Minimum required db size
      2. Minimum required ram
      3. Etc..
   6. Share a note of appreciation / credit with the teams whose previous efforts were building on top of. Ex: The Graph (including their work on the ENS Rainbow Tables).

[lightwalker-eth]

Hey @shrugs, also very open to creative ideas here that would enable us to pragmatically accelerate the launch of our v1. I don’t mean to suggest we need to build something with a bunch of scope here initially. Very open to more lightweight solutions.

[djstrong]

Would it helpful to create a TS script that downloads the rainbow tables file (or better reads it as stream) and insert label and labelhash pairs into the Postgres?
Then during indexing the ENSNode can use this table to resolve labelhashes.

[shrugs]

The solution here will pretty much just look like loading the rainbow tables into postgres. Since they're required for a successful sync, and i imagine 9/10 people using ensnode will want healed names, i think it's ok to make this a hard requirement for startup.

I originally considered embedding them into the project as a pglite database, but their size makes that difficult/not-ideal.

They're >5GB compressed which means they can't be stored in git or lfs, so S3 or other cloud storage is likely necessary. IIRC cloudflare R2 doesn't have egress fees, so perhaps that's a nice place to store it. haven't used it myself to confirm this use-case.

I agree that a script that downloads/loads the rainbow tables into the provided postgres database is likely the best solution. it should be as simple as wget | unzip | psql. that script will place the table in a separate postgres schema called ensnode_healed_labels or something, and we should be able to access it from ponder.

[lightwalker-eth]

Would it helpful to create a TS script that downloads the rainbow tables file (or better reads it as stream) and insert label and labelhash pairs into the Postgres? Then during indexing the ENSNode can use this table to resolve labelhashes.

@djstrong That sounds good to me. For example into a totally separate Postgres schema such as ensnode_healed_labels as @shrugs was suggesting.

We already have a DATABASE_URL environment variable for the Ponder app. It would be nice to use that in these scripts. https://github.com/namehash/ensnode/blob/main/.env.local.example#L35

A few questions for you guys:

1. What approach are you guys thinking for how this script is implemented? Ex: Would it be written in TypeScript, perhaps as part of the Ponder app itself? A separate bash script? Something else?
2. What initial strategy in our v1 are you thinking for how the Ponder app verifies that the rainbow tables are fully / correctly initialized? Perhaps something basic such as just verifying the count of rows in a predefined table matches the expected count?

Also, however it is that we define this script, highly suggest that we make the execution of this script idempotent such that if it fails for some reason or is interrupted, everything gracefully recovers itself just by executing the script again.

@shrugs I like your idea for using R2 to avoid egress fees.

@BanaSeba What do you think about us using R2 to host our own copy of the big files inside https://github.com/graphprotocol/ens-rainbow. If this seems like a good solution to you please let me know how I can support. For example: want me to setup a Cloudflare account for us?

[shrugs]

i would place this in the ensnode repo and make it either a nodejs (ideally nodejs, not ts, to avoid needing ts-node dependency) script or a bash script and then make it easier to call with pnpm run import-rainbowtables or something. yeah we can use count of rows for idempotency of the script itself. if it fails halfway resumption might be difficult but imo it's ok to just kick it off again, it shouldn't be too long-running. it could also live as a separate project in the proposed monorepo, like 'rainbowtables-importer' and called from there — it just wants access to the DATABASE_URL env var

[djstrong]

The file size can be reduced by about 40% by saving hashes as bytes if necessary (the same in database). So it may fit GitHub LFS - or we can split the file into more.

Ad.2. I thought about the same simple verification mechanism as number of rows.

BTW ens rainbow tables seems to be a list of Internet domains.

https://colorlib.com/wp/domain-name-statistics/

The number of domains increased significantly since ens rainbow tables creation.

[shrugs]

If we can compress them much further, then we might be able to store them in lfs and just load them into memory on startup — could be much more efficient as well and avoid the user-cost of someone needing to run the import script beforehand. Do you have the ability to see what the size of the pglite database with those tables would be on-disk?

[djstrong]

Maybe it is better not to load it into RAM (and save GBs of RAM).
I found some other embedded db: https://www.npmjs.com/package/level I think we need just a key-value store for label and labelhash pairs.

I would like to see a chart of resolved names by these rainbow tables in time.

[djstrong]

In rocksdb it takes like 11GB on disk, using bytes as keys 7GB.
If we want to load it into memory then we can just load it into Map but it will take like 15GB...