Semiosis is an open-source framework for measuring the semantic quality of static documentation and context systems. Think "unit testing for your knowledge base" - Semiosis reveals how much information is redundant, what's critical, and where your documentation breaks down.
The Problem: You've built extensive documentation (DBT projects, API docs, knowledge bases) but don't know if it's actually good. Is there redundancy? What happens if parts go missing? Is it token-efficient?
The Solution: Semiosis measures context system quality using standardized LLM probes to evaluate:
- Completeness: Does your documentation cover all necessary concepts?
- Redundancy: How much can you remove while maintaining performance?
- Semantic Density: How much information per documentation unit?
- Robustness: How gracefully does performance degrade as context is removed?
- Critical Boundaries: What's the minimum viable documentation set?
When complete, Semiosis will provide comprehensive documentation quality analysis:
# Analyze your DBT project documentation quality
semiosis evaluate \
--context dbt \
--context-args project_path=./my_dbt_project \
--environment text-to-sql \
--interventions progressive_removal,schema_corruption
# Expected results: context quality report
# ๐ Baseline Performance: 94% (excellent documentation)
# ๐ฏ Semantic threshold: ฮท_c = 0.35 (robust to 65% removal)
# ๐ Critical components: schema.yml files (high impact)
# ๐ Redundancy: Column descriptions (medium overlap)
# ๐ Benchmark: 75th percentile vs industry averageSemiosis provides a modular framework for context quality measurement:
- ๐ Environments: Define evaluation scenarios (text-to-SQL, code generation, custom domains)
- ๐ค Standardized Probes: Built-in LLM agents as measurement instruments
- ๐ Context Systems: Integration with documentation sources (DBT, API docs, knowledge bases)
- โก Interventions: Systematic context modifications (removal, corruption, reordering)
- ๐ Quality Engine: Mathematical framework for measuring semantic information density
# Find minimal documentation set for reliable performance
semiosis evaluate --context dbt --interventions progressive_removal
# Expected: Need only 40% of semantic models for 90% accuracy# Test documentation robustness before agent deployment
semiosis evaluate --interventions corruption,missing_schemas,outdated_docs
# Expected: Performance drops to 60% with 30% schema corruption- ๐ Quality Curves: How performance degrades with documentation removal
- ๐ฏ Semantic Thresholds: Critical information boundaries (ฮท_c values)
- ๐ Component Analysis: Which documentation sections are most valuable
- ๐ Redundancy Maps: What information overlaps and can be consolidated
- ๐ Benchmarking: How your context compares to industry standards
- โก Intervention Impact: Quantified effects of specific documentation changes
- Frontier Models: All your favourites
- Open Source Models: SQLCoder, Kimi K2, Mistral, etc.
- Cloud Platforms: AWS Bedrock, Google Vertex AI for enterprise deployment
- Text-to-SQL: Spider 2.0, BIRD-SQL datasets for database query generation
- Code Generation: Programming task evaluation with execution validation
- Custom Domains: YAML-configurable environments for any documentation type
- DBT Projects: Schema definitions, model docs, semantic layer analysis
- API Documentation: OpenAPI specs, endpoint descriptions, parameter definitions
- Knowledge Bases: Markdown files, wikis, technical documentation
- Custom Sources: Any structured documentation via plugins
Semiosis will implement a rigorous mathematical framework based on semantic information theory:
Agent state: ๐ = (q, y, โ, c, b, ฮธ)
Environment state: ๐ = (D, Q, T)
Context system: ๐ฎ_ฮท = [sโ, โฆ, sโ]
Intervention: ๐ฎ_ฮท' = ๐ฎ_ฮท + s_{n+1}
Agent output: p_ฮธ(y | q, D, ๐ฎ_ฮท)
Token probability: p_ฮธ(tแตข | t_{<i}, q, D, ๐ฎ_ฮท)
Log-likelihood: LL_ฮท(t) = ฮฃแตข log p_ฮธ(tแตข | t_{<i}, q, D, ๐ฎ_ฮท)
Cross-entropy: H_ฮท = ๐ผ[โLL_ฮท(t(q))]
Trust update: โ' = โ + f(LL(t))
Budget update: b' = b โ c + g(โ')
Viability: V(ฮท) = Pr(โ > โ_min โง b > 0)
Semantic threshold: ฮท_c = inf{ฮท | V(ฮท) โค ยฝV(1)}
Where agents maintain trust (โ) through performance and budget (b) through resource management, with viability measuring sustainable operation probability.
We welcome contributions! Key areas for community involvement:
- ๐ Environments: Create evaluation scenarios for specific domains
- ๐ Context Systems: Integrate new semantic layer/knowledgebase/documentation technologies
See our Contributing Guide for detailed instructions.
git clone https://github.com/AnswerLayer/semiosis.git
cd semiosis
pip install -e ".[dev]"
# Note: Core framework still in development - tests coming soonIf you use Semiosis in your research, please cite:
@software{semiosis2025,
title={Semiosis: Evaluate Semantic Layers for AI Agent Performance},
author={AnswerLayer Team},
year={2025},
url={https://github.com/AnswerLayer/semiosis}
}This framework builds on foundational work in semantic information theory:
[1] Kolchinsky, A. and Wolpert, D.H. Semantic information, autonomous agency, and nonequilibrium statistical physics. New Journal of Physics, 20(9):093024, 2018. arXiv:1806.08053
[2] Sowinski, D.R., Balasubramanian, V., and Kolchinsky, A. Semantic information in a model of resource gathering agents. Physical Review E, 107(4):044404, 2023. arXiv:2304.03286
[3] Balasubramanian, V. and Kolchinsky, A. Exo-Daisy World: Revisiting Gaia Theory through an Informational Architecture Perspective. Planetary Science Journal, 4(12):236, 2023. PSJ
[4] Sowinski, D.R., Frank, A., and Ghoshal, G. Information-theoretic description of a feedback-control Kuramoto model. Physical Review Research 6, 043188, 2024. arXiv:2505.20315
MIT License - see LICENSE file for details.
- GitHub Issues - Bug reports and feature requests
- GitHub Discussions - Community discussion
- AnswerLayer - Parent organization
Status: Alpha - Active development. APIs may change.
Roadmap: See GitHub Issues for current development plan.
