feat: transform RAG bot into data model harmonizer for ACORD mapping

[Copilot]

Extends the RAG chatbot with a 6-phase semantic data mapping pipeline (Extract → Index → Generate → Review → CodeGen → Run) for harmonizing MSSQL schemas to ACORD standards using LLM-based reasoning.

New Modules

Mapping & Search

• mapping/MappingEngine.py - Orchestrates mapping lifecycle with confidence scoring, approval workflow, and audit trail
• retrieval/MetadataLoader.py - Converts SQL schemas, ACORD definitions, and business dictionaries into RAG-indexable documents
• retrieval/AcordSearcher.py - Cross-metadata semantic search with ranking

Data Extraction & Code Generation

• source_inspectors/MSSQLScanner.py - SQLAlchemy-based schema extraction with extended properties
• output_gen/SparkCodeGen.py - Generates PySpark ETL with data quality checks from approved mappings
• mapping_prompts/HarmonizationPrompts.py - Domain-specific prompts for chain-of-thought mapping decisions

API Extensions

Added 14 REST endpoints to FlaskApp.py:

# Mapping lifecycle
POST /api/v1/mappings/generate          # Single column mapping
POST /api/v1/mappings/batch             # Batch table mapping
POST /api/v1/mappings/{t}/{c}/approve   # Approve with audit
POST /api/v1/mappings/{t}/{c}/reject    # Reject with reason
GET  /api/v1/mappings/summary           # Statistics

# Code generation
POST /api/v1/codegen/transform          # Column transformation
POST /api/v1/codegen/table-etl          # Full table ETL

# Search & health
GET  /api/v1/acord/search               # ACORD entity/field lookup
GET  /api/v1/health                     # Status check

Enhanced Modules

DocUploader.py - Added metadata indexing functions:

index_harmonizer_metadata(
    acord_json_path="acord_standard.json",
    schema_json_path="sql_schema.json", 
    dictionary_json_path="business_dict.json"
)

Dependencies & Compatibility

• Added sqlalchemy for schema inspection, pyodbc for MSSQL connectivity
• Fixed langchain imports: langchain.schema → langchain_core.documents, langchain.prompts → langchain_core.prompts
• All original routes (/, /admin, /query) preserved - fully backward compatible

Documentation

• HARMONIZER_ARCHITECTURE.md - Complete system design
• QUICKSTART.md - Setup and workflow guide
• MODULES_REFERENCE.md - API reference
• examples_integration_workflow.py - End-to-end demo

Data Model

@dataclass
class ColumnMapping:
    source_table: str
    source_column: str
    target_acord_entity: str
    target_acord_field: str
    confidence_score: float
    reasoning: str
    alternative_mappings: List[Tuple[str, float]]
    status: str  # PENDING, APPROVED, REJECTED
    approved_by: str
    notes: str

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• us.i.posthog.com
  • Triggering command: /usr/bin/python3 python3 -c from source_inspectors.MSSQLScanner import MSSQLScanner from mapping.MappingEngine import MappingEngine from retrieval.MetadataLoader import MetadataLoader from retrieval.AcordSearcher import AcordSearcher from output_gen.SparkCodeGen import SparkCodeGen (dns block)
  • Triggering command: /usr/bin/python3 python3 -c import sys sys.path.insert(0, '.') print('Starting Flask app import test...') try: from FlaskApp import app print('��� Flask app imported successfully') print('Available routes:') for rule in app.url_map.iter_rules(): print(f' {r (dns block)
  • Triggering command: /usr/bin/python3 python3 (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

feat: transform rag-bot into data model harmonizer for insurance data mapping

This comprehensive transformation converts the generic RAG chatbot into a specialized semantic data mapping tool for harmonizing database schemas to industry standards (ACORD).

New Modules (6)

• mapping_prompts/HarmonizationPrompts.py: Insurance domain-specific prompts for semantic mapping and code generation
• retrieval/MetadataLoader.py: Index SQL schemas, ACORD definitions, and business dictionaries
• retrieval/AcordSearcher.py: Semantic search with confidence ranking optimized for ACORD
• mapping/MappingEngine.py: Complete mapping lifecycle management (generate, approve, reject, export)
• source_inspectors/MSSQLScanner.py: Database schema extraction and inspection via SQLAlchemy
• output_gen/SparkCodeGen.py: PySpark ETL code generation with data quality checks

Enhanced Modules (2)

• DocUploader.py: Added metadata loading functions for harmonizer (ACORD, SQL schema, business dictionary)
• FlaskApp.py: Added 18+ REST endpoints for mapping operations and code generation

Documentation (5)

• HARMONIZER_ARCHITECTURE.md: Complete architectural reference (~3000 lines)
• QUICKSTART.md: Step-by-step getting started guide with examples
• MODULES_REFERENCE.md: API reference and data models
• IMPLEMENTATION_CHECKLIST.md: Feature checklist and status verification
• INDEX.md: Documentation navigation hub

Examples

• examples_integration_workflow.py: Complete 6-phase workflow demonstration

Key Features

✓ 6-phase harmonization pipeline (Extract → Index → Generate → Review → CodeGen → Run)
✓ Vector-based semantic search for accurate database-to-standard mapping
✓ LLM chain-of-thought reasoning for mapping decisions
✓ Confidence scoring with alternative suggestions
✓ MSSQL database schema extraction with extended properties
✓ PySpark transformation code generation with data quality checks
✓ Databricks job orchestration support
✓ Full REST API with 18+ endpoints
✓ Complete audit trail and approval workflow
✓ 100% backward compatible with original RAG functionality

Architecture: 6-Phase Pipeline

1. EXTRACT - Pull SQL schema from MSSQL databases (MSSQLScanner)
2. INDEX - Index ACORD standards, SQL schema, business dictionaries (MetadataLoader)
3. GENERATE - Generate semantic mappings with confidence scores (MappingEngine + LLM)
4. REVIEW - Approve/reject mappings in UI or via API (MappingEngine approval)
5. CODEGEN - Generate PySpark transformation code (SparkCodeGen)
6. RUN - Execute orchestrated ETL in Databricks/Spark (orchestration scripts)

REST API Endpoints (18+)

• GET /api/v1/mappings - List mappings with filters
• GET /api/v1/mappings/{table}/{column} - Get specific mapping
• POST /api/v1/mappings/generate - Generate mapping for column
• POST /api/v1/mappings/batch - Batch generate mappings for table
• POST /api/v1/mappings/{table}/{column}/approve - Approve mapping
• POST /api/v1/mappings/{table}/{column}/reject - Reject mapping
• GET /api/v1/mappings/summary - Get mapping statistics
• POST /api/v1/codegen/transform - Generate column transformation
• POST /api/v1/codegen/table-etl - Generate table ETL script
• GET /api/v1/acord/search - Search ACORD entities/fields
• GET /health, /api/v1/health - Health checks
• Plus original /query, /admin endpoints

Data Models

• ColumnMapping: Complete mapping representation with source/target details, confidence, reasoning, status, approval history
• TableInfo/SchemaInfo: MSSQL schema representation
• SparkTransformation: Generated code container with validation

Backward Compatibility

✓ All original RAG functionality preserved
✓ Original /query endpoint unchanged
✓ Original /admin endpoint unchanged
✓ ChromaDB integration maintained
✓ LLM connectors (OpenAI, Ollama) still supported
✓ No breaking changes to existing API

Next Steps

1. Install dependencies: pip install -r requirements.txt (sqlalchemy, pyodbc if needed)
2. Prepare metadata: See QUICKSTART.md for JSON format templates
3. Configure .env with database connection strings
4. Start mapping workflow: See examples_integration_workflow.py
5. Deploy: Follow deployment checklist in IMPLEMENTATION_CHECKLIST.md

Created from VS Code via the GitHub Pull Request extension.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[inbravo]

good