feat: introduce ENSAnalytics API into ENSApi#1239
Conversation
🦋 Changeset detectedLatest commit: 2cb7b7f The changes in this PR will be included in the next version bump. This PR includes changesets to release 14 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
lightwalker-eth
left a comment
lightwalker-eth
left a comment
There was a problem hiding this comment.
@Goader Hey great to see this. Nice work. Reviewed and shared some suggestions with feedback 👍
| await this.refreshData(); | ||
|
|
||
| // Set up automatic refresh every N minutes | ||
| this.refreshTask = cron.schedule(`*/${CACHE_REFRESH_INTERVAL_MINUTES} * * * *`, async () => { |
There was a problem hiding this comment.
Appreciate if you can investigate how we might align the way we're solving similar problems in ENSApi.
If we define the problem to be solved as:
- At startup, build cache X into memory
- Asynchronously refresh the cache of X every Y duration, such that if a refresh fails for some reason, we retain the previously cached stale value of X.
We have a current implementation of this goal where X is the "Indexing Status" which can be found here: https://github.com/namehash/ensnode/blob/main/apps/ensapi/src/middleware/indexing-status.middleware.ts
If I'm missing something please let me know, but ideally we could align the strategy used to solve what seems like an identical problem / need.
There was a problem hiding this comment.
I looked at that, and it depends. As far as I understand, the indexing status is checked every time a request arrives, and if it is still in cache, then we use it. But if the cache entry is too old, then we ask the ENS Indexer its status. This is nice, since if we do not have any requests coming, we are not generating redundant traffic.
However, it will work differently than what #1185 describes (it will be dependent on the requests). And another concern I have is that if the cache cannot be renewed for some reason (SQL queries fail), then we will not retain the previously cached referrers, since they will expire anyway with a TTL cache. But at the same time, if we were to run into such a problem, that ENSApi could not execute the SQL query to ESNDb, then I think the cache status of ENSAnalytics would be the least of our concerns.
So if switching our strategy from refreshing cache independently from the incoming requests to the way it is done with indexing status, then I think it's ok, and I will implement it that way (should it be a middleware, though?). Correct me if I misunderstood something.
…zation, client method for ensanalytics api
lightwalker-eth
left a comment
lightwalker-eth
left a comment
There was a problem hiding this comment.
@Goader Hey super work 🚀 Very happy with this feature! I shared a few small suggestions. Please feel welcome to take the lead to merge this PR when ready 🚀
| /** | ||
| * Timestamp (from `Date.now()`) indicating when the value was last successfully updated | ||
| */ | ||
| updatedAt: number; |
There was a problem hiding this comment.
Suggest making this a UnixTimestamp
Goal: prefer that as much as possible all timestamp related math in our work uses our UnixTimestamp type alias to reduce cognitive load of different time formats in different places.
| * Optional promise tracking an in-progress revalidation request. | ||
| * Used to deduplicate concurrent revalidation attempts. |
There was a problem hiding this comment.
| * Optional promise tracking an in-progress revalidation request. | |
| * Used to deduplicate concurrent revalidation attempts. | |
| * Optional promise of the in-progress revalidation attempt. | |
| * If undefined, no revalidation attempt is in-progress. | |
| * If defined, a revalidation attempt is already in-progress. | |
| * Used to enforce no concurrent revalidation attempts. |
| * @link https://datatracker.ietf.org/doc/html/rfc5861 | ||
| */ | ||
| export function staleWhileRevalidate<T>(fn: () => Promise<T>, ttl: number): () => Promise<T> { | ||
| if (!Number.isInteger(ttl) || ttl <= 0) { |
There was a problem hiding this comment.
Continue to suggest making this a Duration value for the goal that I put a lot of value on standardizing the types we use for timestamp math as much as possible. I'm perfectly ok if this removes the ability to set ttl values in millisecond resolutions.
| let cache: SWRCache<T> | null = null; | ||
|
|
||
| return async (): Promise<T> => { | ||
| const now = Date.now(); |
There was a problem hiding this comment.
Continue to suggest this 👍
| // Start initial build | ||
| initialBuild = fn() | ||
| .then((value) => { | ||
| cache = { value, updatedAt: now }; |
There was a problem hiding this comment.
Suggest calculating a fresh value for now here.
| import { factory } from "@/lib/hono-factory"; | ||
| import logger from "@/lib/logger"; | ||
|
|
||
| const TTL_MS = 5 * 60 * 1000; // 5 minutes |
There was a problem hiding this comment.
Continue to think this would be a nice step 👍
apps/ensapi/src/index.ts
Outdated
| if (cache) { | ||
| logger.info(`ENSAnalytics cache warmed up with ${cache.referrers.size} referrers`); | ||
| } else { | ||
| logger.warn("ENSAnalytics cache returned null - no cached data available yet"); |
There was a problem hiding this comment.
Could you have a look at referrersCacheFetcher to see if it is still possible that it might throw an error?
If it can't throw an error anymore, we can:
- remove the try / catch logic here
- Update this log statement so it's an error not a warning
However, if my assumption here is wrong then please feel welcome to ignore. Appreciate your advice 👍
| export const ENS_HOLIDAY_AWARDS_START_DATE = getUnixTime(new Date("2025-12-01T00:00:00.000Z")); | ||
|
|
||
| /** | ||
| * End date for the ENS Holiday Awards referral program. | ||
| * December 31, 2025 at 23:59:59 UTC | ||
| */ | ||
| export const ENS_HOLIDAY_AWARDS_END_DATE = getUnixTime(new Date("2025-12-31T23:59:59.999Z")); |
There was a problem hiding this comment.
We might not need to include date-fns package in ens-referrals package in case this is converting a date into unix timestamp is the sole need for the date-fns package.
Instead, let's set these const directly to unix timestamp values
| export const ENS_HOLIDAY_AWARDS_START_DATE = getUnixTime(new Date("2025-12-01T00:00:00.000Z")); | |
| /** | |
| * End date for the ENS Holiday Awards referral program. | |
| * December 31, 2025 at 23:59:59 UTC | |
| */ | |
| export const ENS_HOLIDAY_AWARDS_END_DATE = getUnixTime(new Date("2025-12-31T23:59:59.999Z")); | |
| export const ENS_HOLIDAY_AWARDS_START_DATE: UnixTimestamp = 1764547200; | |
| /** | |
| * End date for the ENS Holiday Awards referral program. | |
| * December 31, 2025 at 23:59:59 UTC | |
| */ | |
| export const ENS_HOLIDAY_AWARDS_END_DATE: UnixTimestamp = 1767225599; |
| * "BaseRegistrar" contract for direct subnames of .eth) for the provided namespace. | ||
| * | ||
| * @param namespace The ENS namespace to get the Ethnames Subregistry ID for | ||
| * @returns The AccountId for the Ethnames Subregistry contract for the provided namespace. | ||
| * @throws Error if the contract is not found for the given namespace. | ||
| */ | ||
| export function getEthnamesSubregistryId(namespace: ENSNamespaceId): AccountId { |
There was a problem hiding this comment.
Nice! I can see how we can later add getEthnamesSubregistryNode function here 🚀
tk-o
left a comment
tk-o
left a comment
There was a problem hiding this comment.
The PR looks awesome! Let's fix the circular dependency problem, and we'll be good to go 🚀
packages/ens-referrals/package.json
Outdated
| }, | ||
| "dependencies": { | ||
| "date-fns": "catalog:" | ||
| "@ensnode/ensnode-sdk": "workspace:*" |
There was a problem hiding this comment.
This is a circular dependency. @ensnode/ensnode-sdk has had @namehash/ens-referrals package on the list of dependencies already.
| @@ -1,13 +1,13 @@ | |||
| import { getUnixTime } from "date-fns"; | |||
| import type { UnixTimestamp } from "@ensnode/ensnode-sdk"; | |||
There was a problem hiding this comment.
We need to not use ensnode-sdk package inside the ens-referrals one. Doing so creates circular dependency.
Since UnixTimestamp is just an alias for the number type, I suggest we re-create the UnixTimestamp type here.
| import type { UnixTimestamp } from "@ensnode/ensnode-sdk"; | |
| /** | |
| * Unix timestamp value | |
| * | |
| * Represents the number of seconds that have elapsed | |
| * since January 1, 1970 (midnight UTC/GMT). | |
| * | |
| * Guaranteed to be an integer. May be zero or negative to represent a time at or | |
| * before Jan 1, 1970. | |
| */ | |
| export type UnixTimestamp = number; |
lightwalker-eth
left a comment
lightwalker-eth
left a comment
There was a problem hiding this comment.
@Goader Nice updates 😄 Great work 🚀 Just 1 small question. You're welcome to take the lead to merge this!
| return result; | ||
| } catch (error) { | ||
| logger.error({ error }, "Failed to build aggregated referrer snapshot"); | ||
| throw error; |
There was a problem hiding this comment.
Should this return null instead? Appreciate your advice. Please ignore if not relevant 👍
There was a problem hiding this comment.
Sorry, missed it. The responsibility for handling the error is inside the SWR logic, so this function can throw errors without disrupting the flow. And also this way SWR will know that revalidation failed and will return stale data, instead of treating null as the updated value.
tk-o
left a comment
tk-o
left a comment
There was a problem hiding this comment.
Awesome updates 🚀 Merge when ready.
3 participants
advances: #1185
This PR introduces ENSAnalytics API as part of ENSApi. ENSAnalytics implements a Stale-While-Revalidate (SWR) caching strategy via middleware to provide sub-millisecond response times while keeping data fresh with automatic background updates every 5 minutes. All required ENV variables are already available in ENSApi, so nothing was added there.
Architecture:
aggregatedReferrerSnapshotCacheMiddlewareuses SWR pattern to serve cached data immediately (even if stale) while asynchronously revalidating in the backgroundENS_HOLIDAY_AWARDS_START_DATEandENS_HOLIDAY_AWARDS_END_DATEfrom@namehash/ens-referralsSDK additions:
AggregatedReferrerMetrics,AggregatedReferrerMetricsContribution, etc.)ITEMS_PER_PAGE_DEFAULT(25),ITEMS_PER_PAGE_MAX(100)Endpoint implemented:
/ensanalytics/aggregated-referrers: Main endpoint for paginated aggregated referrer leaderboardpage(optional): Page number, must be >= 1 (default: 1)itemsPerPage(optional): Items per page, must be 1-100 (default: 25)page >= 1(error: 400)1 <= itemsPerPage <= 100(error: 400)pagemust not exceed total pages (error: 400)