feat: hive_posts_cache_temp for hot queries (90-day window) (#364)

* feat: hive_posts_cache_temp for hot queries (trending/hot/payout)

Author: ety001

docs/DESCRIPTION-hive_posts_cache_temp-table.md

@@ -0,0 +1,51 @@
+# PR: hive_posts_cache_temp for hot queries (90-day window)
+
+## Summary
+
+This branch introduces a second cache table **`hive_posts_cache_temp`** that holds only the last ~90 days of post cache data. Hot list APIs (trending, hot, created, promoted, payout, payout_comments) are routed to this table to reduce query load on the main `hive_posts_cache` table.
+
+## What’s in this branch
+
+### 1. Temp table and schema
+- **`hive/db/schema.py`**: New table `hive_posts_cache_temp` with the same columns as `hive_posts_cache` (plus optional `_synced_at`), and indexes tuned for hot sorts.
+- **`hive/db/db_state.py`**: Migration (v24→25) creates the temp table if missing and runs a **one-time cold-start backfill** (INSERT from main table, 90-day window). Uses `SET LOCAL statement_timeout = '0'` for the long-running insert; adapter whitelist updated to allow `SET LOCAL`.
+
+### 2. Query routing
+- **`hive/db/cache_router.py`**: `CacheRouter.get_table(sort)` returns `hive_posts_cache_temp` for hot sorts and `hive_posts_cache` otherwise.
+- **Bridge / condenser / hive_api**: List and cursor code use `CacheRouter.get_table(sort)` so hot sorts hit the temp table; `comments_by_id` queries temp first and falls back to main for missing IDs; `posts_by_id` remains main-only.
+
+### 3. Writes (dual-write)
+- **`hive/indexer/cached_post.py`**: Every INSERT/UPDATE to `hive_posts_cache` is applied to **both** the main and temp table in the **same batch/transaction** (dual-write). Undelete path updated to write to both tables.
+- **Logging**: `[DUAL-WRITE] batch: N posts written to main+temp` per batch; `[PREP] posts cache process (main+temp): ...` for large flushes, so operators can confirm dual-write in logs.
+
+### 4. Deletes
+- **`hive/indexer/cache_sync.py`**: Every **60s** (triggered from listen when `num % 20 == 0`), a background thread runs `DELETE FROM hive_posts_cache_temp WHERE created_at < :cutoff` (90-day cutoff). No INSERT/sync from main—temp is fed only by dual-write and cold-start backfill.
+- **`hive/indexer/cached_post.py`**: `CachedPost.delete()` deletes from both main and temp when a post is removed (e.g. delete_comment).
+- **`hive/indexer/blocks.py`**: Fork rollback (`_pop_blocks()`) deletes affected post_ids from both main and temp.
+
+### 5. Documentation
+- **`docs/hive_posts_cache_temp-90day-boundary.md`**:
+  - Describes routing and 90-day boundary behavior (list APIs do *not* auto-fallback to main at 90 days; only comment-by-ID falls back).
+  - **§6** documents all write/delete locations for the temp table (cold-start, dual-write, 60s prune, delete at delete time, fork rollback).
+
+## Design notes
+
+- **Dual-write** was chosen instead of a periodic sync (e.g. chunked INSERT from main every 60s) so that temp and main are updated in the same transaction and write timing is consistent; the previous 60s chunked-sync approach was dropped for performance and simplicity.
+- Hot lists are intentionally 90-day only; “load more” does not automatically switch to the main table beyond that window.
+
+## Testing
+
+- New tests: `tests/db/test_cache_router.py`, `tests/db/test_cache_sync.py`.
+- Cache sync tests updated (e.g. no INSERT batch size assertions after sync became delete-only).
+
+## Related code (quick ref)
+
+| Area            | Files |
+|-----------------|--------|
+| Schema / migration | `hive/db/schema.py`, `hive/db/db_state.py` |
+| Routing         | `hive/db/cache_router.py` |
+| Dual-write      | `hive/indexer/cached_post.py` |
+| 90-day prune    | `hive/indexer/cache_sync.py`, `hive/indexer/sync.py` |
+| Delete at delete / fork | `hive/indexer/cached_post.py`, `hive/indexer/blocks.py` |
+| API usage       | `hive/server/bridge_api/cursor.py`, `hive/server/condenser_api/cursor.py`, `hive/server/hive_api/objects.py`, `hive/server/hive_api/thread.py` |
+| Docs            | `docs/hive_posts_cache_temp-90day-boundary.md` |

docs/hive_posts_cache_temp-90day-boundary.md

@@ -0,0 +1,110 @@
+# hive_posts_cache_temp and 90-Day Boundary Evaluation
+
+This document records the evaluation of the temp table (`hive_posts_cache_temp`) routing logic and whether list APIs automatically fall back to `hive_posts_cache` when "load more" crosses the ~90-day boundary.
+
+## Summary
+
+- **List APIs (including "load more")**: They do **not** automatically read from `hive_posts_cache` at the 90-day boundary. Each request uses a single table (temp or main) chosen by `CacheRouter.get_table(sort)`; there is no cursor-based fallback to the main table.
+- **By-ID object APIs**: `comments_by_id` queries temp first, then fills missing IDs from main, so it **does** automatically read `hive_posts_cache`. `posts_by_id` uses only the main table.
+
+So: **"Load more" does not automatically read `hive_posts_cache`**; only comment-by-ID resolution falls back to the main table when data is missing from temp.
+
+---
+
+## 1. Temp Table and 90-Day Boundary
+
+- **Temp table**: `hive_posts_cache_temp` holds only the last ~90 days of hot data. Rows older than 90 days are pruned by `CacheSync` (see `hive/indexer/cache_sync.py`).
+- **Routing**: In `hive/db/cache_router.py`, `CacheRouter.get_table(sort)` returns the temp table for hot sorts (`trending`, `hot`, `created`, `promoted`, `payout`, `payout_comments`) and the main table otherwise. Routing is based only on `sort`; there is no logic that considers cursor position or proximity to the 90-day cutoff.
+
+---
+
+## 2. List + Pagination ("Load More") Implementation
+
+List and pagination are implemented in `hive/server/bridge_api/cursor.py` and `hive/server/condenser_api/cursor.py`:
+
+- `table = CacheRouter.get_table(sort)` is used once per request; the same table is used for the entire list query and for the cursor/seek condition.
+- Pagination uses `seek_id` / `last_id` on that single table with `ORDER BY ... LIMIT`; there is no "query temp then fall back to main" logic.
+
+Example from `pids_by_category` (bridge):
+
+```python
+table = CacheRouter.get_table(sort)
+# ...
+sql = ("""SELECT post_id FROM %s WHERE %s
+          ORDER BY %s DESC, post_id LIMIT :limit
+          """ % (table, ' AND '.join(where), field))
+return await db.query_col(sql, tag=tag, last_id=last_id, limit=limit)
+```
+
+So for sorts that use the temp table (e.g. trending, hot, created), the entire list (first page and all "load more" pages) is read only from the temp table. When the user scrolls near the 90-day boundary, the temp table simply has no older rows (they were pruned), so the next page returns fewer or no rows; the code does **not** switch to `hive_posts_cache` for older data.
+
+- If the product expectation is "hot lists only show the last 90 days", the current behavior is correct and does not wrongly read the main table.
+- If the product expectation is "load more should continue to show posts older than 90 days", then the current implementation does **not** satisfy that; you would need to add logic to use the main table (or a combined strategy) when the cursor is near or past the 90-day boundary.
+
+---
+
+## 3. APIs That Do Automatically Read hive_posts_cache
+
+Only the **comment-by-ID** path does a temp-then-main fallback. In `hive/server/hive_api/objects.py`, `comments_by_id`:
+
+- Queries the temp table first with the requested IDs.
+- Collects which IDs were not found.
+- If there are missing IDs, runs the same query against `hive_posts_cache` and appends those rows.
+
+So comments older than 90 days are still returned by reading from the main table when they are missing from temp. List APIs do not have this fallback.
+
+---
+
+## 4. Whether the 90-Day Boundary Is a "Bug"
+
+- **Correctness**: When only the temp table is used for a list, the results are consistent and do not mix in unintended rows from the main table. At the boundary, "load more" simply runs out of rows in temp; there is no automatic read from `hive_posts_cache`.
+- **Product expectation**: If the design is "hot lists are 90-day only", there is no bug. If the design is "load more should include data beyond 90 days", then the current list implementation does not do that and would need to be extended (e.g. use main table or union when the cursor is near the 90-day cutoff).
+
+---
+
+## 5. Quick Reference
+
+| Scenario | Automatically reads hive_posts_cache? | Notes |
+|----------|--------------------------------------|--------|
+| List first page + load more (trending / hot / created, etc.) | **No** | Only temp is used; no fallback at boundary |
+| Comment by ID (`comments_by_id`) | **Yes** | Temp first, then main for missing IDs |
+| Post by ID (`posts_by_id`) | N/A (main only) | Uses main table only, not temp |
+
+**Direct answer**: "Load more" does **not** automatically read `hive_posts_cache`. Only the comment-by-ID resolution falls back to the main table when IDs are missing from temp. If you want list APIs to continue beyond the 90-day boundary using the main table, that logic would need to be added explicitly (e.g. cursor-based table selection or a combined temp+main query).
+
+---
+
+## 6. Temp Table: Write and Delete Locations
+
+Where `hive_posts_cache_temp` is written to and deleted from (for verification and operations).
+
+### Writes (INSERT / UPDATE)
+
+| Location | Description |
+|----------|-------------|
+| **`hive/db/db_state.py`** | **Create table + cold-start backfill.** On migration, if the temp table does not exist: create it, then run a one-time `INSERT INTO hive_posts_cache_temp (...) SELECT ... FROM hive_posts_cache WHERE created_at >= :cutoff` (90-day window) to backfill. |
+| **`hive/indexer/cached_post.py`** | **Dual-write with main table.** Every write to `hive_posts_cache` also writes to the temp table in the same batch: `_insert()` produces both main and temp INSERTs; `_update()` produces both main and temp UPDATEs. Used in listen (per-block), `from_steemd` (batch sync), and undelete placeholder writes. |
+
+### Deletes
+
+| Location | Description |
+|----------|-------------|
+| **`hive/indexer/cache_sync.py`** | **Every 60s:** `CacheSync._sync()` runs in a background thread and executes `DELETE FROM hive_posts_cache_temp WHERE created_at < :cutoff` (cutoff = now − 90 days). Triggered from `hive/indexer/sync.py` when `num % 20 == 0` (every ~60s in listen). |
+| **`hive/indexer/cached_post.py`** | **On post delete:** `CachedPost.delete()` runs `DELETE FROM hive_posts_cache_temp WHERE post_id = :id` when a post is removed (e.g. delete_comment op), in sync with the main table delete. |
+| **`hive/indexer/blocks.py`** | **Fork rollback:** `_pop_blocks()` deletes from the main cache for affected blocks and also runs `DELETE FROM hive_posts_cache_temp WHERE post_id IN :ids` so temp stays in sync. |
+
+### Summary
+
+- **Writes to temp:** (1) One-time cold-start backfill in `db_state.py`; (2) Dual-write in `cached_post.py` (same transaction as `hive_posts_cache`).
+- **Deletes from temp:** (1) 90-day prune every 60s in `cache_sync.py`; (2) Single-post delete in `cached_post.delete()`; (3) Bulk delete on fork rollback in `blocks._pop_blocks()`.
+
+---
+
+## Related Code
+
+- `hive/db/cache_router.py` – table selection by `sort`
+- `hive/indexer/cache_sync.py` – 90-day pruning of `hive_posts_cache_temp`
+- `hive/db/db_state.py` – migration that creates and backfills temp (v24)
+- `hive/server/bridge_api/cursor.py` – list + pagination (e.g. `pids_by_category`, `pids_by_community`)
+- `hive/server/condenser_api/cursor.py` – `pids_by_query`
+- `hive/server/hive_api/objects.py` – `comments_by_id` (temp + main), `posts_by_id` (main only)

hive/db/adapter.py

@@ -189,6 +189,6 @@ def _is_write_query(sql):
             return False
         if action in ['DELETE', 'UPDATE', 'INSERT', 'COMMIT', 'START',
                       'ALTER', 'TRUNCA', 'CREATE', 'DROP I', 'DROP T',
-                      'ANALYZ']:  # ANALYZE command
+                      'ANALYZ', 'SET LO']:  # ANALYZE command; SET LOCAL for migration
             return True
         raise Exception("unknown action: {}".format(sql))

hive/db/cache_router.py

@@ -0,0 +1,24 @@
+"""Query routing for hive_posts_cache vs hive_posts_cache_temp."""
+
+
+class CacheRouter:
+    """Route hot-data queries to temp table, others to main table."""
+
+    TEMP_TABLE = 'hive_posts_cache_temp'
+    MAIN_TABLE = 'hive_posts_cache'
+
+    HOT_QUERIES = {
+        'trending', 'hot', 'payout', 'payout_comments', 'created', 'promoted'
+    }
+
+    @classmethod
+    def get_table(cls, query_type=None):
+        """Return table name for the given query type."""
+        if query_type and query_type in cls.HOT_QUERIES:
+            return cls.TEMP_TABLE
+        return cls.MAIN_TABLE
+
+    @classmethod
+    def get_temp_sql(cls, base_sql):
+        """Replace main table name with temp table in SQL."""
+        return base_sql.replace(cls.MAIN_TABLE, cls.TEMP_TABLE)

hive/db/db_state.py

@@ -4,10 +4,12 @@
 
 import time
 import logging
+from datetime import datetime, timedelta
 
 from hive.db.schema import (setup, reset_autovac, build_metadata,
                             build_metadata_community, teardown, DB_VERSION,
-                            build_metadata_blacklist, build_trxid_block_num)
+                            build_metadata_blacklist, build_trxid_block_num,
+                            build_temp_cache_metadata)
 from hive.db.adapter import Db
 
 log = logging.getLogger(__name__)
@@ -383,6 +385,38 @@ def _check_migrations(cls):
 
             log.info("[HIVE] idx_posts_id_author_deleted_depth_community index created")
             cls._set_ver(24)
+        if cls._ver == 24:
+            # Hot-data temp table for hive_posts_cache (90-day window, synced every 30s)
+            if not cls.db().query_col(
+                    "SELECT EXISTS(SELECT 1 FROM information_schema.tables WHERE table_name='hive_posts_cache_temp')"
+            )[0]:
+                log.info("[HIVE] Creating hive_posts_cache_temp table...")
+                build_temp_cache_metadata().create_all(cls.db().engine())
+                log.info("[HIVE] hive_posts_cache_temp created")
+                # One-time cold-start backfill (90-day window); disable statement_timeout for long run
+                cls.db().query("SET LOCAL statement_timeout = '0'")
+                cutoff = datetime.now() - timedelta(days=90)
+                backfill_sql = """
+                    INSERT INTO hive_posts_cache_temp (
+                        post_id, author, permlink, category, community_id, depth, children,
+                        author_rep, flag_weight, total_votes, up_votes, title, preview, img_url,
+                        payout, promoted, created_at, payout_at, updated_at, is_paidout,
+                        is_nsfw, is_declined, is_full_power, is_hidden, is_grayed,
+                        rshares, sc_trend, sc_hot, body, votes, json, raw_json, _synced_at
+                    )
+                    SELECT post_id, author, permlink, category, community_id, depth, children,
+                           author_rep, flag_weight, total_votes, up_votes, title, preview, img_url,
+                           payout, promoted, created_at, payout_at, updated_at, is_paidout,
+                           is_nsfw, is_declined, is_full_power, is_hidden, is_grayed,
+                           rshares, sc_trend, sc_hot, body, votes, json, raw_json,
+                           NOW() as _synced_at
+                    FROM hive_posts_cache
+                    WHERE created_at >= :cutoff
+                """
+                result = cls.db().query(backfill_sql, cutoff=cutoff)
+                n = result.rowcount if hasattr(result, 'rowcount') else 0
+                log.info("[HIVE] hive_posts_cache_temp cold-start backfill done, rows=%s", n)
+            cls._set_ver(25)
 
         reset_autovac(cls.db())
 

hive/db/schema.py

@@ -10,7 +10,7 @@
 
 #pylint: disable=line-too-long, too-many-lines, bad-whitespace
 
-DB_VERSION = 24
+DB_VERSION = 25
 
 def build_metadata():
     """Build schema def with SqlAlchemy"""
@@ -247,8 +247,64 @@ def build_metadata():
 
     metadata = build_trxid_block_num(metadata)
 
+    metadata = build_temp_cache_metadata(metadata)
+
+    return metadata
+
+
+def build_temp_cache_metadata(metadata=None):
+    """Build hive_posts_cache_temp table for hot-data queries (90-day window)."""
+    if not metadata:
+        metadata = sa.MetaData()
+
+    sa.Table(
+        'hive_posts_cache_temp', metadata,
+        sa.Column('post_id', sa.Integer, primary_key=True, autoincrement=False),
+        sa.Column('author', VARCHAR(16), nullable=False),
+        sa.Column('permlink', VARCHAR(255), nullable=False),
+        sa.Column('category', VARCHAR(255), nullable=False, server_default=''),
+        sa.Column('community_id', sa.Integer, nullable=True),
+        sa.Column('depth', SMALLINT, nullable=False, server_default='0'),
+        sa.Column('children', SMALLINT, nullable=False, server_default='0'),
+        sa.Column('author_rep', sa.Float(precision=6), nullable=False, server_default='0'),
+        sa.Column('flag_weight', sa.Float(precision=6), nullable=False, server_default='0'),
+        sa.Column('total_votes', sa.Integer, nullable=False, server_default='0'),
+        sa.Column('up_votes', sa.Integer, nullable=False, server_default='0'),
+        sa.Column('title', sa.String(255), nullable=False, server_default=''),
+        sa.Column('preview', sa.String(1024), nullable=False, server_default=''),
+        sa.Column('img_url', sa.String(1024), nullable=False, server_default=''),
+        sa.Column('payout', sa.types.DECIMAL(10, 3), nullable=False, server_default='0'),
+        sa.Column('promoted', sa.types.DECIMAL(10, 3), nullable=False, server_default='0'),
+        sa.Column('created_at', sa.DateTime, nullable=False, server_default='1990-01-01'),
+        sa.Column('payout_at', sa.DateTime, nullable=False, server_default='1990-01-01'),
+        sa.Column('updated_at', sa.DateTime, nullable=False, server_default='1990-01-01'),
+        sa.Column('is_paidout', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('is_nsfw', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('is_declined', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('is_full_power', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('is_hidden', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('is_grayed', BOOLEAN, nullable=False, server_default='0'),
+        sa.Column('rshares', sa.BigInteger, nullable=False, server_default='0'),
+        sa.Column('sc_trend', sa.Float(precision=6), nullable=False, server_default='0'),
+        sa.Column('sc_hot', sa.Float(precision=6), nullable=False, server_default='0'),
+        sa.Column('body', TEXT),
+        sa.Column('votes', TEXT),
+        sa.Column('json', sa.Text),
+        sa.Column('raw_json', sa.Text),
+        sa.Column('_synced_at', sa.DateTime, nullable=True),
+        sa.Index('hive_posts_cache_temp_ix6a', 'sc_trend', 'post_id',
+                 postgresql_where=sql_text("is_paidout = '0'")),
+        sa.Index('hive_posts_cache_temp_ix7a', 'sc_hot', 'post_id',
+                 postgresql_where=sql_text("is_paidout = '0'")),
+        sa.Index('hive_posts_cache_temp_ix9a', 'depth', 'payout', 'post_id',
+                 postgresql_where=sql_text("is_paidout = '0'")),
+        sa.Index('hive_posts_cache_temp_ix30', 'community_id', 'sc_trend', 'post_id',
+                 postgresql_where=sql_text("community_id IS NOT NULL AND is_grayed = '0' AND depth = 0")),
+        sa.Index('hive_posts_cache_temp_created', 'created_at'),
+    )
     return metadata
 
+
 def build_metadata_community(metadata=None):
     """Build community schema defs"""
     if not metadata:

hive/indexer/blocks.py

@@ -223,6 +223,7 @@ def _pop(cls, blocks):
             # remove posts: core, tags, cache entries
             if post_ids:
                 DB.query("DELETE FROM hive_posts_cache WHERE post_id IN :ids", ids=post_ids)
+                DB.query("DELETE FROM hive_posts_cache_temp WHERE post_id IN :ids", ids=post_ids)
                 DB.query("DELETE FROM hive_post_tags   WHERE post_id IN :ids", ids=post_ids)
                 DB.query("DELETE FROM hive_posts       WHERE id      IN :ids", ids=post_ids)