Tool: Count Unknown Names & Unknown Labels

[lightwalker-eth]

Goals

1. Help us calculate important metrics related to Unknown Names / Unknown Labels
2. Help us communicate the scale of the problem Unknown Names / Unknown Labels create, and therefore also to help us communicate the value proposition we're providing by reducing that problem.
3. Help us (or anyone else) reproduce the same calculation whenever desired by running a simple command.
   1. NOTE: These metrics grow across time. They are not fixed data points.

For example, we're currently wireframing some rough design concepts for how to tell this story on the ENSRainbow landing page:

• NOTE: Please see the placeholder "fine print" text in the wireframe above where we write "* In the ENS Registry on Ethereum Mainnet as of 13 Feb, 2025". This placeholder is planned since we will NOT be making dynamic API calls for the sum total count of Unknown Names / Unknown Labels at this time. Instead, we need to produce these counts offline, then hardcode the result of the count into constants in the landing page. If anything about this isn't 100% clear please ask @lightwalker-eth.
  • An exception to this is the ability to make a dynamic API call to ENSRainbow to return the sum of labels it is able to heal. But please note how this metric isn't actually used at all in the "benchmarks" chart shown above. The label count returned by the ENSRainbow API returns the set of healable labels. Please note how this is very different than the count of labels in the ENS Subgraph that would otherwise be unknown. Not all labels that can be healed have actually been registered into the ENS Registry.
  • We still have plans to make use of the dynamic API call to ENSRainbow to return the sum of labels it is able to heal, but this data will be presented in a different format on the ENSRainbow landing page. Please see the rough design concept we are currently wireframing for this:

Definitions

The following definitions should be strictly applied.

1. "Unknown Label"
   1. A label returned by the ENS Subgraph in the format of "[{labelhash}]" where {labelhash} is a valid normalized labelhash without a 0x prefix and with the correct number of fully lowercase hex digits.
2. "Unknown Name"
   1. A name returned by the ENS Subgraph containing 1 or more Unknown Labels.

NOTE: For now, we should anchor these definitions to the ENS Subgraph, and not to ENSNode.

Background Info

For the future we will be able to make enhancements to ENSNode so that a calculation for the sum of Unknown Labels / Unknown Names will be dynamically calculated during indexing, such that precomputed counts can be efficiently returned in a simple O(1) operation.

For now though, we're postponing such enhancements to optimize for shipping ENS Subgraph Backwards Compatibility.

Therefore, the only way to calculate aggregated metrics for Unknown Labels / Unknown Names by paginating through lists of Domains from the ENS Subgraph.

To ensure a deterministic calculation while paginating through lists of results, the Subgraph's "Time Travel Queries" feature is important: https://thegraph.com/docs/en/subgraphs/querying/graphql-api/#time-travel-queries This enables each pagination query to the ENS Subgraph to ask for a consistent state snapshot across pages for the same block number.

@shrugs might already have built most of the solution for this inside https://github.com/namehash/ens-subgraph-transition-tools. Please carefully review how the "Snapshot Equivalency Tool" works. This tool already builds a complete snapshot of all domains returned by the ENS Subgraph at a specified block number / block height.

Therefore, one idea is to explore how we can add a capability to command line app Matt has already built that would take as input a reference to a snapshot that's already been built, and then iterate through that snapshot to calculate the desired metrics about Unknown Labels / Unknown Names. This calculation then wouldn't require any further requests to the ENS Subgraph. It could just read from the snapshot that's already been built!

Another very important detail: For the metrics we are producing, it will also be extremely helpful to count the total number of Unknown Labels / Unknown Names in the ENS Subgraph if no label healing were performed at all. Here's a plan for how we will be able to calculate this:

• Take advantage of how ENSNode offers backwards compatibility with the ENS Subgraph.
• Take advantage of how ENSNode defers to ENSRainbow for label healing data.
• Take advantage of how we can create a special deployment of ENSNode and ENSRainbow that is completely equivalent to the ENS Subgraph, except that the ENS Rainbow instance will contain 0 healable labels. In this way, the final snapshot we produce for that ENSNode instance can tell us how many Unknown Labels / Unknown Names would exist if the ENS Subgraph performed zero label healing at all.

Subtasks

For any given Snapshot produced by Matt's "snapshot" tool, return:

• The block number / block height associated with the snapshot / results.
• Count the total number of Unknown Names.
  • These should already be distinct within Domain snapshots.
• Count the total number of non-distinct Unknown Labels.
• Count the total number of distinct Unknown Labels.
• Return a list of the 100 Unknown Labels that occur most frequently in the snapshot in descending order (from most frequently occurring, to least, then sorted by labelhash as needed for a deterministic sorting tiebreaker). For each, format the result as:
  • "{unknown label} {count of times found in snapshot}"

[lightwalker-eth]

@shrugs Hey, before we begin any development on this goal, could you please review and critique the strategy proposed above for building a solution for this on top of the existing snapshot CLI tools you've been building in https://github.com/namehash/ens-subgraph-transition-tools ?

Appreciate your advice on how we might proceed. Cheers 👍

[djstrong]

How to treat addr.reverse names? Just like any other name or they need a special category?

[djstrong]

I am not sure about definition of "Unknown Label", because ENS Subgraph is not returning labels but domains (names).
global.[b50225e66585b05f85fa55923224991b0517e07903ac43c0e5258b4d48950a97].eth here we have one unknown label?
[b50225e66585b05f85fa55923224991b0517e07903ac43c0e5258b4d48950a97].[b50225e66585b05f85fa55923224991b0517e07903ac43c0e5258b4d48950a97].eth here are two unknown labels?

These are results from my previous research:

Count the total number of Unknown Names: 1029350
Count the total number of non-distinct Unknown Labels: 1032611
Count the total number of distinct Unknown Labels: 1021509

Top 100 most frequent unknown labels:
[090fdc7e2b2c9f51e9f228395059bf4d88379c2cea8082bec00613ed967407cc] 2122
[b50225e66585b05f85fa55923224991b0517e07903ac43c0e5258b4d48950a97] 1202
[17b7d2bcf3a9a21b8c13ea2f80455c2e5725593527788ea42b4deac1b327b620] 539
[03eda0b49ca0a5517e86e4e6fff0353db67e81f8d1f79adb96ce8fd7ffaff898] 462
[636afe9927340430a1dac2bc08e3e8188d0dd23a856988e7c08de745ad0f59fa] 373
[4330f1adfa526ce75372dd5d6b910ba965c9116b3374ef7ce8db489d18a08875] 264
[2dc1318376e3f5c1df02850294bbc90e16a9c311ab446771d182088df644d885] 204
[2e64049e898164f5026a863629813a1cc19816fd55c5427d05dbc7f16098403b] 113
[781f95d9216f00e18ae371662ae33f6e64b9f5edf8e1cb5ef0a181fe51d0227f] 95
[e4b1702d9298fee62dfeccc57d322a463ad55ca201256d01f62b45b2e1c21c10] 74
[40b7fa9b4f846614c090bceb0ea657efea7010271dce51c5ab4c8323538a8c04] 73
[c48a050317206447ef1b37e99077afbf2453c51ee233c3e802cce5cb9abeab68] 72
[52f1a9b320cab38e5da8a8f97989383aab0a49165fc91c737310e4f7e9821021] 68
[be4e9092d84b60a13a21eed79011cf9ab145101230863931b7f4cb7a8d4f4f50] 57
[3d52e5a7feaf7163883125a48a29ab801bd0a0166201c9f5c2e5424be11f5b78] 54
[c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470] 49
[e3917102134e5ed42a1109579a617975277e2a449d17ad8a6769e8268b6ecb5a] 40
[072c27708f29137e22967e650fe73376163f0599224878d8e2e29c9c54bd5c4f] 39
[97310c951da9cdc5814a31672ebafa7722d3ac6f64944f2b0766576bdf54a5b2] 38
[31c4738a4cbf75f0b0b19c1a11830e47a4086361084c66f4d6f1655e3d608ab6] 38
[f06a8f5a858a9920622c18b3f0378c9a153503a6ef2cd5c7f4c5ed5896207697] 37
[ddbf3779fbcddd6139d12a598629c9073b1f99d968b7a967b4c781d0f29188f1] 29
[29c44035c639021b255adbfc65f937ec4b6e96b8abd3fd89bb5d190cb5a40636] 29
[56ca023500263c04a5a099ba1f0ed91e41d97443540b5f4d4604dfef55be023c] 27
[09f569decb59b9f1f1127bd2d6f402fccaebe9218e7848646c616a64d58eb5ff] 26
[522c6dfcf77f48d9d767e9b48c79cd6dfc5746154503a75f1f37faec6874a81f] 25
[4e00e25b084b6c97bac32e27caa0dd85f19b7649ae19a838da60eabe4ef6ce8b] 25
[9e72d801ba2525de32711e5073887893f702cd6236076f5bad0af47e4c56ea76] 24
[b118e704014b8004edebdad1eb2997fb3f670f0fd819a62a3baa4c5b799c191d] 23
[82c2546f68a52eeccabc0c6a58f17861e5013b31bb5b4679547187c54776d052] 21
[55b3d52f4d4d7d05df4311c2ecaa0a527397653aca45e8be878ccf2bfed5fb91] 21
[ca42c85e1bb0b67a92f17e5888d5ec18df13f5bd5d998891aa2e4f1e36aa125a] 19
[a16790fd3fb6f22bb98d1f3724ec081b6b0a6765b1556e18251a5c871f9afa70] 17
[d16fd9316fa1b40c8a86df2d6de3f614265eb4ac33ac2413778ede45fc577568] 16
[731180061c11ca577a4397d17005512011784a5bd41d7fa04e4ffacb518a4971] 16
[c10bae37bc902f024a78afc961efe51e790b6d5f695c3854644376fdeef88dfb] 15
[81b91214a17c2b4fd719179b33988fecc9c795d9b6ded754437c057d32519edf] 15
[466c19921c7e5937399d81f16b5021ed051a96ce1e170cbf9e0e7e070db0e76e] 14
[c466eb0422d03704306231e0f6fd5b79e5889171172d94846069b9dd5e1fa6a4] 14
[17eea091d728c1639f8f10bfe1ef1e789a53da442524a61d2aa905d8e78c9fad] 13
[ab7bc20d72016e83889aec75d6fcbac1ac8e5bb577ba7b96c635101455b971be] 13
[41afb7993e3cb4bde6c84eacecd2a713286acc81c0cb950c8908d5c7f36e544e] 12
[851577984373f401afae5507b06dac74c2dd05ef8f0dcb46d4ee8738507078d4] 12
[743f34511e8124a4bbc9e24c023277940e81531b9a5586c35e7e313309117e2c] 12
[a2085d6c1646d9520e1ee54fe4e0f9eec95e6ff8892bb2e51bede10aae79a821] 11
[277b6add00e7a3464e7c9d245379710a606693baa82cfae9be277da403f3025a] 11
[15774a7a3bc6c8c8e908fca5275038c821b4b3aef17d364f9a00976f9a548386] 11
[1226a50111d3ddb0ac66065a7ba777807edf06a6f8ad4742580e80f6ec042688] 11
[2eb278530aeb392e30450cf8054f46d8a501832c8fbbb06765e2ec8f55009673] 11
[61b58029e47cfcc30dc3778f5a2d226cf5f9c93cefe3df4bd4ce3bad00b7db79] 11
[052f5f7197e34e3a4da12a763ecf1c9f3a3cfc8a4df012032e6b8549ec344591] 10
[16964e7225dd4922926c1b00514a964ec04f3ef26b7ee6900af3be2e2171b560] 10
[08e28b3c8e9e77e8d11816585b0e66d114d752bcb817eb8570f0f5a0fde7546f] 10
[942e81b4281d5ac742276ebaf587612dbc8107c14909c1fedb8f9fd027274423] 10
[462a1d6391f7ea5916874504f3b5fc8cd43626f6bbabc8a22fe4312dc1585362] 9
[9ab6fe9fd3c7bf898f1df1d8a5ce521a1174446e5ce6edeef343956a62b7c37c] 9
[c0b0a88f098e2bff96f761dd0a5e7541b17060962604a9ec4a00797efeebcddb] 9
[62b0f594a6cf552939b37df8f5702dc92b6c1d4f0ae1d5f30bb245bfa5cec8ba] 9
[4e83a5d8d9374e5e10fd9e64ee8e34248df2aa5b8fd4122e6cca31a772654ae8] 9
[149a76805a959de96890738ce9dc587159953d1f59f4336d0a9220be956c5c5e] 9
[67a1cb9cca63df52547f23f86db828febd205fee9544c9e7b5fcc8a15b3c1f28] 8
[9e8c4364cef6bf7f1fc1b02f34e210ba543fa92554789018e72f13e48e8ac55b] 8
[83ac0e78c2b6ccba6b1147caae1afd2c5bbfb9f9c6435d0621dd28b4444433b0] 8
[a069e0ce05e9036a49d4968ecf60a65468cfe096b389bde7fa379c94487baa45] 8
[00ed32c5bca8c546f9b1bf72e4b76dd9bf1f0bc14bd2b44f80d122f1ee169302] 8
[5b27bed6d8333e3d9ba5dd7d7376822c897a711c0a03ea7d4bf41022a2d39826] 8
[c8d60d27635b0a4fdafe579aafdc0d18b452a1be3546bf6ac6379065be6020e6] 8
[a62a7d7f6ee898826f7af1a53f9ba25e5a81c5e1ef9a94dbfa9fa533b0c0d259] 8
[ed2cc33b0587afe42a04579d79c23b72b25c5416efcb50c674f3f59e2db9cce6] 8
[dbe385fd6c1cd68e1e4e3e89b31a7ce1f7e23c51226abf1175a623a21c47df22] 8
[f09e939dd7d59be2dee24a0fa03110c96910351c12f63abb35506d3778b1809b] 7
[82121c5c6de2dde887d9f5d90f72e8572eef6806790e2142dd7d251232e995c0] 7
[f536c949036255008aa815516718cab4d8db7fcc4fcb27fdb18caff06d9eecc1] 7
[3f197493fc5ee4ec30f443071d7cf208ac2c93d10dd6ade13d97961f6fc47776] 7
[661ba4042fead372f402a868385c8b5e8f2fa322888a9b3439ea64e093a1aefa] 7
[68931a6595315888efc66f68fd604625c02c49d861b4848d3ecb577c67686a04] 7
[63f95616e29257e0c0f335effef265265a790e38444bd19355493ff7dd5e155a] 7
[9f9a23999c595792d380d534e1bd2e64e0d6d221cc8cb30ffe384768d7fc8073] 7
[a46786b5c0ceaca6df3ce286dcf889e739ef09e85a115d90619d049a57afb91e] 7
[54b3f319dfde882a321cc1e533231bef427671d1eff0756022db540585f7eca4] 7
[fd9fa5d43b78b95004b630b5a85c783c456f80fc4a44cda348e5d976bf2806f4] 7
[79c546b96f2b55705e90a6f03ea2398d641083a58532d722620057c62d3d9a13] 7
[a0fc0f7b9714afb29f672bd38a96650a12ff32b9c3fe7884a3416b47961d334d] 7
[e874e9a12ffd2ca66e68d58c0199370d9b676c6fb847a5167bc672eb82d8786c] 7
[1ab0969ed46f3829af25c2defb67343540b6664c1c00894ac9ba2a4424bf48c2] 7
[6a20e0d9319875fc220fd51193a11e8f9ea16d938443aa52a8048c1a483ead1f] 6
[c1c0cb349e24fcad590cf2f4e9ac782f129a565369edd5e6c3733e4f5eb0845b] 6
[24ea06d18ad191b35b9717104bd483ec9f388d61b51aa803933ef4b46bb5e964] 6
[d7719c5cdc6da405495b93dfba3094ff47495a51986115deb61ce88e4e679fc8] 6
[f4498a5832ab64031573e198a39450413521d1e38a575bdce196e2d15139693b] 6
[050c5c82c5fdb7f3ceaf3050b0b293e46c4ea997f8483fa613063a11254b0dc7] 6
[4bbb263b52a676704f4e2890be6f3e51113b22917cfe4058b93ec52a9c9360dd] 6
[c31b709692e248d00c389182f33f348b94a1abd617ebd0a4b941bccca1b3378d] 6
[948639554fe7952fec063717ecb0eea6b2ecba30b3a861fb53ff1f64f875cae1] 6
[ef9bdb9d1d3e82b4628694f7cc2be81c1f3816faa4b2a8491bc4ef83f668e719] 6
[dbd5f4acab5574ed6ce54020f8bce6a1b0d51ae24934ba9e5a5b9c29b3a52d80] 6
[ef1e72db2598cd2a3c045139f87550dff21cffdea9a2f65a920e2492c8a811b2] 6
[f480e1ac3b4bfdb89026b61d0b643d6555ac3f5339f9973cf8403096008836df] 6
[6a8541b3cacc0e89126404a9ee179859eaadb431fb042b8b70685f19183c87d6] 6
[7c1cf24dc5c27c6ffa17be362024a188e5ef96920069c02b580dea2123da5f80] 6

[shrugs]

terminology-wise, I've been using "un-healed" over "unknown" (as in #277), which do we want to standardize on? unknown labels is more correct

---

the strategy for determining "number of labels actually healed by ensrainbow" where we index mainnet without ENSRainbow is certainly the most straightforward. as long as we're running this somewhat infrequently that seems fine.

i wonder if we could us the subgraph's logic to arrive at a simpler calculation, though it gets less clear because subnames obviously inherit their parent's healed labels and it may not be trivial to determine if a name was healed through the rainbow tables or the namewrapper.

ah, perhaps we do the reverse: for every healed name in the subgraph, determine which label values are in the rainbow tables (a reverse lookup), and subtract from the set of all healed labels in the index — this will give us

A) the set of labels registered on mainnet that the rainbow tables include and
B) the set of labels registered on mainnet via the namewrapper that the rainbow tables do not include

---

implementing another option for this tool to do those calculations seems reasonable as well. likely it depends on namehash/ens-subgraph-transition-tools#16 because the snapshots don't current contain enough data to differentiate them (they're currently just dumps of the output itself, no info about what those entities are or how they were queried). but anyway those suggested unknown label counting operations can definitely be done against the snapshots.

that said these calculations may just as easily be done against the postgres database, so perhaps a tool that executes some sql against postgres is simpler? @djstrong how are you producing those results from your research?

[shrugs]

i've implemented those changes to the snapshot tool, so these count commands can be grafted onto that tool when needed. can take the lead on that if we're confident about this approach and the originally proposed spec