feat(endpoint-microsub): add Microsub endpoint for feed reading #823
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Display user profile, commits, starred repos, PRs/issues, repositories - Support featured repos with recent commits display - Include private repo activity when authenticated - Add caching for API responses - JSON API endpoints for external widgets Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add featured controller for /github/featured page and API - Add /github/api/featured JSON endpoint for external widgets Co-Authored-By: Claude Opus 4.5 <[email protected]>
…details GitHub's Events API no longer includes commit details in PushEvent payloads for many requests. This adds a fallback that fetches commits directly from the user's recently pushed repositories when the events API returns empty. - Try events API first (existing behavior) - If no commits found, fetch from user's top 5 recently pushed repos - Parallelize repo commit fetching for performance - Sort combined results by date This mirrors the approach used by the featured repos controller which fetches commits directly from repos. Co-Authored-By: Claude Opus 4.5 <[email protected]>
This package is maintained separately at rmdes/indiekit-endpoint-github and published to npm as @rmdes/indiekit-endpoint-github. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…ucture - Package setup with proper dependencies (feedparser, microformats-parser, ioredis, sanitize-html) - MongoDB storage layer for channels, feeds, items, notifications, muted, blocked - Core controllers implementing Microsub API actions - Endpoint discovery with microsub link in homepage template - Locale strings for all supported languages This is the foundation for the full Microsub server plugin. Subsequent phases will add feed parsing, real-time updates, and the built-in reader UI. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add feed parsing, fetching, and polling infrastructure: - lib/feeds/parser.js: Feed type detection and parser dispatch - lib/feeds/rss.js: RSS 1.0/2.0 parser using feedparser - lib/feeds/atom.js: Atom parser using feedparser - lib/feeds/jsonfeed.js: JSON Feed parser - lib/feeds/hfeed.js: Microformats h-feed parser with feed discovery - lib/feeds/normalizer.js: Converts all feed formats to jf2 - lib/feeds/fetcher.js: HTTP fetcher with ETag/Last-Modified caching - lib/cache/redis.js: Redis utilities for caching and pub/sub - lib/polling/tier.js: Adaptive tier-based polling algorithm (Ekster pattern) - lib/polling/processor.js: Feed processing pipeline with filtering - lib/polling/scheduler.js: Scheduler with interval-based polling - lib/controllers/follow.js: Trigger immediate fetch on subscribe - lib/storage/feeds.js: Extended with tier update methods Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add feed discovery, preview, and full-text search: - lib/controllers/search.js: Feed discovery from URLs via autodiscovery links, h-feed parsing, and channel item search - lib/controllers/preview.js: Fetch and preview feeds without storing, returns sample items - lib/search/indexer.js: MongoDB text index creation for full-text search - lib/search/query.js: Full-text search with fallback to regex matching, includes search suggestions Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add mute, block, and content filtering: - lib/storage/filters.js: Comprehensive filter operations including muted/blocked URL checks, type filtering, regex filtering, and channel filter settings - lib/storage/items.js: Add deleteItemsByAuthorUrl for blocking - lib/controllers/block.js: Remove past items when blocking a URL - lib/polling/processor.js: Use filters module for channel filtering Supports: - Muting URLs per-channel or globally - Blocking author URLs with retroactive item removal - ExcludeTypes: filter out likes, reposts, bookmarks, replies, etc. - ExcludeRegex: pattern matching for content filtering Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add SSE broker and WebSub integration: - lib/realtime/broker.js: SSE connection management with ping keepalive, channel subscriptions, Redis pub/sub integration - lib/controllers/events.js: Updated to use broker for real-time event streaming - lib/websub/discovery.js: WebSub hub discovery from Link headers and content (Atom, RSS, HTML) - lib/websub/subscriber.js: Subscribe/unsubscribe to hubs with HMAC signature verification, subscription renewal detection - lib/websub/handler.js: WebSub callback handler with signature verification, subscription confirmation, and content processing Supports: - GET ?action=events for SSE stream with channel subscriptions - 10-second ping keepalive - Redis pub/sub for multi-instance broadcasting - WebSub subscription with automatic signature verification - WebSub content push handling with background processing Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add webmention verification and processing for notifications channel: - verifier.js: Fetch source URL, verify link to target, parse microformats to extract author, content, and mention type (like, reply, repost, etc.) - processor.js: Store notifications in MongoDB, publish real-time events via Redis pub/sub, support read/unread tracking per user - receiver.js: Accept webmentions async (202 Accepted), process in background Implements IndieWeb webmention receiving spec for social reader notifications. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add full social reading experience in Indiekit admin: - views/reader.njk: Channel list with unread counts - views/channel.njk: Timeline view with pagination - views/channel-new.njk: Create new channel form - views/item.njk: Single item detail view with actions - views/settings.njk: Channel filtering settings (ExcludeType/Regex) - views/compose.njk: Compose reply/like/repost form - views/partials/: Reusable item-card, timeline, author, actions Updated index.js to mount reader routes as sub-router for correct baseUrl handling. Navigation links to /microsub/reader. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Change package name to @rmdes/indiekit-endpoint-microsub - Update repository URLs to rmdes/indiekit fork - Ready for npm publish Co-Authored-By: Claude Opus 4.5 <[email protected]>
…B cursors .toSorted() is a JavaScript array method (ES2023), but MongoDB cursors require .sort() method. This was causing 'toSorted is not a function' errors when accessing /microsub routes. Added eslint-disable comments since the unicorn/no-array-sort rule doesn't distinguish between Array#sort and MongoDB cursor's sort(). Co-Authored-By: Claude Opus 4.5 <[email protected]>
The templates were using `request.baseUrl` directly in Nunjucks,
but the request object isn't available in the template context.
Updated all controllers to pass `baseUrl` as a render variable
and updated all templates to use `{{ baseUrl }}` instead.
Fixes 404 errors when navigating within the reader UI.
Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Replace `field` macro with `input` (field isn't exported globally) - Replace `fieldset` + checkboxes filter with `checkboxes` macro - Fix textarea label to use string not object - Add attributes for autofocus support - Add missing locale key for channel name Co-Authored-By: Claude Opus 4.5 <[email protected]>
…e macros - Replace non-existent icons with valid Indiekit icons: - arrow-left/arrow-right → previous/next - heart → like - settings → updatePost - bell → mention - plus → createPost - external → public - dot → Unicode bullet ● - Fix textarea macro to use string label instead of object - Fix checkboxes macro usage - Bump version to beta.6 Co-Authored-By: Claude Opus 4.5 <[email protected]>
…polation
- Fix locale interpolation: change %{channel} to {{channel}} syntax
- Add /channels/:uid/feeds route for viewing and managing feeds
- Add form to subscribe to new feeds with immediate background fetch
- Add button to unfollow feeds
- Add "Feeds" button to channel view header
- Add locale strings for feed URL label and placeholder
- Add JSDoc @returns annotations
- Bump version to beta.7
Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add /reader/search page to discover feeds from any URL - Add feed discovery using auto-detect for RSS, Atom, JSON Feed, h-feed - Add subscribe form with channel selection dropdown - Add "Follow" button to main reader view for easy access - Bump version to beta.8 Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add console.info/warn messages during init to track startup - Log whether database is available when scheduler starts - Helps diagnose why scheduler might not be running - Bump version to beta.9 Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Initialize Redis connection from application.redisUrl config - Use ioredis with lazy connect and error handling - Log connection status for debugging Co-Authored-By: Claude Opus 4.5 <[email protected]>
When a feed is unfollowed/deleted, also delete all items that were fetched from that feed. Previously items remained orphaned in the database. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Prevents "name.replace is not a function" error when code passed to IndiekitError is not a string (e.g., when response.statusText is undefined in some edge cases) Co-Authored-By: Claude Opus 4.5 <[email protected]>
…iance Features: - Micropub compose integration (reply, like, repost, bookmark) - WebSub auto-subscription when hub discovered in feeds - Media proxy with Redis caching (4h TTL, 2MB max) - Enhanced full-text search with weighted indexes - Basic test suite (auth, validation, media-proxy, jf2, feeds, pagination, tier, filters) Fixes: - Entry matching now works with URLs (Microsub spec compliance) - getUserId helper for proper auth in API token requests - All controllers use consistent user identification Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fixes two feed processing issues: 1. Invalid time value for non-standard date formats: - Added parseDate() helper that handles formats like "2026-01-28 08:40" - Gracefully falls back to undefined instead of throwing 2. Unable to detect feed type for ActivityPub sites: - WordPress sites with ActivityPub plugin return JSON instead of HTML - Now detects ActivityPub JSON and tries common feed paths (/feed/, etc.) - Added tryCommonFeedPaths() as fallback for feed discovery Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fixes multiple issues with the reader UI: 1. Author photos showing [object Object]: - Added extractPhotoUrl() to handle photo objects in h-card normalization - Added normalizeAuthor() in transformToJf2 for robust photo extraction 2. Empty photo galleries: - Added normalizeMediaArray() to ensure all media URLs are strings - Handles legacy data that may have stored photo objects 3. Empty source spans: - Added url field to _source alongside feedUrl - Processor now adds feed name to item._source.name 4. Item links returning 404: - Fixed item-card.njk to use item._id instead of item.uid - Updated getItemById to try both _id and uid lookups - Added 404.njk template for graceful error handling 5. Item context links (replies, likes, reposts): - Fixed property names to use hyphenated jf2 format (in-reply-to, etc.) Bumps version to 1.0.2 Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Wrap cache.put() in try-catch to handle network error responses - Return valid Response objects in all error paths - Add fallback responses for offline/network failure scenarios - Better error logging with request URLs Prevents "Cache.put() encountered a network error" and "Failed to convert value to Response" errors Co-Authored-By: Claude Opus 4.5 <[email protected]>
…terns - Add dedicated CSS file with comprehensive reader styling - Add yellow glow effect for unread items - Add multi-photo grid layouts (2/3/4 photos) - Add context bars for interaction types (like, repost, reply, bookmark) - Add inline action buttons on item cards - Add keyboard navigation (j/k for items, o/Enter to open) - Add character counter to compose view - Create custom reader layout for CSS inclusion - Update all views to use new reader layout - Bump version to 1.0.3 Co-Authored-By: Claude Opus 4.5 <[email protected]>
Nunjucks doesn't have a built-in 'min' filter. Use conditional expression instead for photo count. Co-Authored-By: Claude Opus 4.5 <[email protected]>
The item-card action links use short params (reply, like, repost, bookmark) but the controller only extracted long-form params (replyTo, likeOf, etc). Now supports both forms. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Temporary debug logging to diagnose why interactions are being created as empty notes instead of likes/replies/reposts. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Enable the profile scope in IndieAuth to return user profile information (name, url, photo) parsed from the h-card on the user's site. - Add profile.js module to fetch and parse h-card microformats - Enable profile scope in scope.js - Include profile data in authorization and token responses - Add unit test for profile module Co-Authored-By: Claude Opus 4.5 <[email protected]>
rmdes
force-pushed
the
feat/endpoint-microsub
branch
from
95d04c9 to
c2f476b
Compare
- Normalize interaction URLs (like-of, repost-of, in-reply-to, bookmark-of) to strings in the normalizer - Update item-card template to extract URL from object values - Add ensureString helper in compose controller - Fix template crashes when interaction properties contain objects Co-Authored-By: Claude Opus 4.5 <[email protected]>
…ssion - Use 'content' instead of 'error.message' for error template - Ensure statusText is a string with fallback - Parse JSON error response for better error messages - Fixes "Input data should be a String" error on repost/compose Co-Authored-By: Claude Opus 4.5 <[email protected]>
Author
|
Closing this PR as discussed. Will resubmit as a series of smaller, focused PRs for easier review. Thanks for the feedback! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a new
@indiekit/endpoint-microsubpackage that implements a full-featured Microsub server, enabling feed subscription and reading using the Microsub protocol.Features
Core Microsub API
Feed Support
Adaptive Polling
WebSub Integration
Real-time Events
Content Filtering
Media Proxy
Built-in Reader UI (Aperture-inspired)
Compose Integration
Additional Features
Compatibility
Tested with these Microsub clients:
Test plan
Notes
This implementation draws inspiration from Ekster and Aperture, other Microsub server implementations.
🤖 Generated with Claude Code