How to handle removals from a list

[twitch-jordanpotter]

At Twitch, we provide an API endpoint to get the list of a user's followers. For some of our popular users, this list might have 100,000+ entries.

A subscriber can subscribe to the topic https://api.twitch.tv/helix/users/follows?first=1&to_id=1234, which will send a content distribution request whenever user 1234 receives a new follower. Since the topic includes the query parameter first=1, our subscribers know that the content distribution request will include only the most recent entry, not the entire massive list.

However, it's unclear to us how we should handle someone unfollowing. If someone from the middle of the list unfollows, what payload should we send in the content distribution request? How do we send a payload that says "the list is now missing an element"? 🤔

[aaronpk]

That's a great question. WebSub was originally created to distribute append-only feeds like Atom/RSS feeds. Most feeds will show the latest N entries, and once an entry disappears from the feed it does not mean it was deleted. In order to explicitly tell a subscriber that something has been deleted you'd need to WebSub and the WebSub payload as a transport layer for that additional information, by creating some sort of "this item was deleted" payload that can go in the feed.

Not that it's exactly the same, but we've been doing something similar with h-feed, where a deleted entry will show up again in the feed with its unique URL and a deleted property that indicates it's been deleted. This means an old post might have to pop up in the feed again, so the feed is actually more sorted by updated date than published date.

[twitch-jordanpotter]

Interesting! We were considering a similar approach, where our endpoint returns a list of follow relationship events. The most recent event (follow or unfollow) would be at the beginning of the list. Sounds like this would be the best approach to match pre-existing industry practice.

If there are any other options we should consider, please let us know 😄