Hybrid retrieval system combining vector search (pgvector) with a knowledge graph (Neo4j) over real SEC 10-K filings. Built to answer: does graph-augmented retrieval actually beat vector-only on multi-hop questions?
Corpus: FY2024 10-Ks for Apple (AAPL), Microsoft (MSFT), NVIDIA (NVDA) — 828 chunks, 982 graph entities, 68 relations.
36 hand-verified questions (16× 1-hop, 14× 2-hop, 6× 3-hop) over the same ingested corpus. Full results: bench/results/benchmark_stub.json.
| System | Token-F1 | Recall@5 | Recall@10 | MRR | p50 latency | p95 latency |
|---|---|---|---|---|---|---|
| Vector-only | 0.051 | 0.514 | 0.639 | 0.492 | 38ms | 89ms |
| Hybrid | 0.053 | 0.514 | 0.625 | 0.488 | 37ms | 70ms |
By hop (Recall@5):
| Hop | n | Vector-only | Hybrid |
|---|---|---|---|
| 1 | 16 | 0.625 | 0.625 |
| 2 | 14 | 0.464 | 0.464 |
| 3 | 6 | 0.333 | 0.333 |
Finding: With the deterministic stub provider, hybrid retrieval achieves parity with vector-only, not improvement. The rule-based router correctly identifies metric-seeking questions for vector search, but the graph adds little on this question set. Token-F1 is low because the stub generates placeholder text — retrieval metrics (recall/MRR) are the meaningful comparison.
Cost: $0.00 (stub). 66k/73k tokens in, ~4k out.
Status: Blocked by Groq free-tier daily token limit (200k TPD). The 72-question benchmark (36 × 2 systems) exceeds the daily budget. Pending limit reset.
The pipeline is verified working with Groq openai/gpt-oss-20b (see demo evidence). Real-provider benchmark will be added when tokens are available.
Question → Router → Vector Search (pgvector) ←→ Graph Search (Neo4j)
↓
Hybrid Retrieval → Grounded Answer (Groq/Stub)
↓
Citation Validation → Answer + Sources
Key design decisions:
-
No raw Cypher generation. The LLM never writes Cypher. Code selects from parameterized, validated templates (
entity_neighborhood, 1/2/3-hop fixed patterns). -
Controlled ontology. 12 relation types:
HAS_SEGMENT,HAS_SUBSIDIARY,ACQUIRED,SELLS_PRODUCT,LED_BY,REPORTS_METRIC,OPERATES_IN,COMPETES_WITH,DEPENDS_ON,PARTNERS_WITH,FACES_RISK,USES_TECHNOLOGY. All triples validated against the ontology. -
Deterministic ingestion. Chunk IDs are deterministic positional IDs (
chunk_id_for(doc_id, index), e.g.{doc}#c{NNNNN}) shared by Neo4j and pgvector. Document-level SHA256 content hashes make re-ingestion idempotent — unchanged documents are skipped. -
Entity resolution. spaCy NER + rule-based extractors, with alias merging ("Apple Inc." → "Apple").
- Python 3.12+
- Neo4j (bolt://localhost:7687)
- PostgreSQL 16 + pgvector (localhost:5432)
python -m venv .venv && source .venv/bin/activate
pip install -e .export NEO4J_URI="bolt://localhost:7687"
export NEO4J_USER="neo4j"
export NEO4J_PASSWORD="..."
export DATABASE_URL="postgresql://.../..."
export EMBEDDING_MODEL_PATH="/path/to/all-MiniLM-L6-v2"# Download SEC filings (writes data/raw/ + SHA256 manifest)
python scripts/download_sec.py --out data/raw
# Ingest into Neo4j + pgvector
python -m krag.services.ingest --data-dir data --extractor spacy \
--graph-backend real --vector-backend real# CLI demo
python scripts/demo_queries.py --llm stub --out demo/demo_run.json
# API server
uvicorn krag.api.main:app --host 127.0.0.1 --port 8000
curl -X POST http://127.0.0.1:8000/ask \
-H "Content-Type: application/json" \
-d '{"question": "What are Apple'"'"'s reportable segments?"}'# Deterministic stub (no API costs)
python -m eval.run_benchmark --llm stub --out bench/results/benchmark_stub.json
# Real provider (requires Groq credential, respects rate limits)
python -m eval.run_benchmark --llm groq --groq-model openai/gpt-oss-20b \
--out bench/results/benchmark_groq20b.jsonsrc/krag/
domain/ # Pydantic schemas, ontology, controlled triples
storage/ # Neo4j + pgvector adapters (real + in-memory)
services/ # Extractor, resolver, router, retriever, QA, LLM
api/ # FastAPI app
eval/
gold_qa.json # 36 hand-verified questions
run_benchmark.py # Vector vs hybrid comparison
metrics.py # token-F1, recall@k, MRR
bench/results/ # Committed benchmark JSON
demo/ # Demo transcripts
scripts/ # SEC downloader, demo runner
GET /health— service status, corpus countsPOST /ask—{question}→ grounded answer with citationsGET /graph/entity/{id}— entity details + relationsGET /graph/search?name=...— entity search
# Unit + integration (requires Neo4j + PostgreSQL)
pytest tests/ -v
# Lint + typecheck
ruff check src/ && ruff format --check src/
mypy src/Current status: 24 tests pass against real Neo4j + pgvector. Integration tests use isolated KRAGTEST_* prefixes and never touch production data.
-
Hybrid ≈ vector-only on the current 36-question set with stub provider. The graph helps for entity-relationship questions but the corpus is dominated by metric questions best served by vector search.
-
Groq benchmark pending due to free-tier daily token limits.
-
Extraction is rule-based (spaCy + regex). LLM extraction (Groq 20b/120b) is implemented but not yet benchmarked against the rule-based baseline.
-
No Docker daemon in this environment —
docker-compose.ymlis provided butdocker compose upwas not executed here.
MIT