[FEATURE] Add MUVERA ingest and search processors for multi-vector ANN prefetch

[praveenMprasad]

Is your feature request related to a problem?

Multi-vector retrieval models like ColBERT and ColPali produce per-token embeddings that require MaxSim scoring across all token pairs. This is expensive at scale because there's no way to do ANN prefetch on variable-length multi-vector representations — you're forced to either brute-force score every document or rely on external tooling to pre-encode vectors client-side.

Currently, the k-NN plugin supports lateInteractionScore for MaxSim reranking, but the inner query is typically match_all or a text filter, meaning every matching document gets scored. OpenSearch supports mean pooling approach where the query and doc token vectors are averaged to form single vectors which can then be used for regular ANN search. However this is a lossy approach where averaging the token vectors loses lot of information from the token level vectors and spoils the purpose of multi-vector late interaction models. There's no native way to narrow candidates without losing information using the multi-vector embeddings themselves.

What solution would you like?

Add two new processors implementing the MUVERA algorithm (Multi-Vector Retrieval via Fixed Dimensional Encodings, [paper](https://arxiv.org/abs/2405.19504)):

1. muvera ingest processor — Converts variable-length multi-vector embeddings into a single fixed-dimensional encoding (FDE) vector using SimHash clustering and random projections. The FDE is stored in a knn_vector field for ANN indexing. The original multi-vectors remain in _source for reranking.

2. muvera_query search request processor — Intercepts script_score queries containing query_vectors in script params, MUVERA-encodes them into an FDE, and replaces the inner match_all with a knn query on the FDE field. The lateInteractionScore script wrapper stays intact for MaxSim reranking on the prefetched candidates.

User flow

Step 1: Create ingest pipeline

PUT _ingest/pipeline/muvera-ingest
{
  "description": "MUVERA FDE encoding for ColBERT vectors",
  "processors": [
    {
      "muvera": {
        "source_field": "colbert_vectors",
        "target_field": "muvera_fde",
        "dim": 128,
        "fde_dimension": 2560
      }
    }
  ]
}

Defaults: k_sim=4, dim_proj=8, r_reps=20, seed=42. FDE dimension = r_reps * 2^k_sim * dim_proj = 2560. The fde_dimension parameter validates the computed value so the user explicitly acknowledges the output size.

Step 2: Create index

PUT muvera-index
{
  "settings": {
    "index.knn": true,
    "default_pipeline": "muvera-ingest"
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "muvera_fde": {
        "type": "knn_vector",
        "dimension": 2560,
        "method": {
          "name": "hnsw",
          "space_type": "innerproduct",
          "engine": "faiss"
        }
      },
      "title": { "type": "text" }
    }
  }
}

Note: colbert_vectors is intentionally left unmapped, it stays in _source for reranking but doesn't need its own field mapping.

Step 3: Index documents

POST muvera-index/_doc/1
{
  "title": "example document",
  "colbert_vectors": [
    [0.1, 0.2, ...],
    [0.3, 0.4, ...],
    [0.5, 0.6, ...]
  ]
}

The ingest processor reads colbert_vectors, produces the FDE, and writes it to muvera_fde. Both fields end up in the stored document.

Step 4: Create search pipeline

PUT _search/pipeline/muvera-search
{
  "request_processors": [
    {
      "muvera_query": {
        "target_field": "muvera_fde",
        "dim": 128,
        "fde_dimension": 2560,
        "oversample_factor": 4
      }
    }
  ]
}

Same MUVERA hyperparams as ingest (must match). oversample_factor controls how many candidates the knn prefetch retrieves relative to the requested result size.

Step 5: Search

POST muvera-index/_search?search_pipeline=muvera-search
{
  "size": 10,
  "query": {
    "script_score": {
      "query": { "match_all": {} },
      "script": {
        "source": "lateInteractionScore(params.query_vectors, 'colbert_vectors', params._source, params.space_type)",
        "params": {
          "query_vectors": [[0.1, 0.2, ...], [0.3, 0.4, ...]],
          "space_type": "innerproduct"
        }
      }
    }
  }
}

What happens:

1. Search processor extracts query_vectors from script params
2. MUVERA-encodes them into a query FDE
3. Replaces match_all with knn on muvera_fde (k = size × oversample_factor = 40)
4. lateInteractionScore reranks the 40 candidates using exact MaxSim on original multi-vectors
5. Top 10 returned to user

What alternatives have you considered?

• Client-side MUVERA encoding (works but requires users to maintain encoding logic outside OpenSearch)
• Binary quantization of multi-vectors (lossy, doesn't preserve MaxSim structure)
• Text-based prefetch with BM25 (misses semantic signal from embeddings)

Do you have any additional context?

• MUVERA is already implemented in [fastembed](https://github.com/qdrant/fastembed) (Python) and used in production with Qdrant
• We have a working implementation with unit tests, stable across multiple iterations with random seeds
• The implementation uses only public APIs — no reflection or core OpenSearch modifications required
• Tested end-to-end on a live cluster: ingest pipeline creates FDE vectors, search pipeline rewrites queries, lateInteractionScore reranking produces correct MaxSim scores

[praveenMprasad]

OpenSearch MUVERA Benchmark Results

MUVERA params: k_sim=5, dim_proj=16, r_reps=20, FDE dim=10,240
Index: faiss HNSW, m=16, ef_construction=512, ef_search=512, 1 shard, 1 segment

nfcorpus — ColBERTv2 (3,633 docs, 323 queries)

Approach	NDCG@1	NDCG@5	NDCG@10	% of Exact
Exact MaxSim (brute-force)	0.483	0.386	0.344	100%
MUVERA + MaxSim rerank (4x)	0.449	0.349	0.311	90.3%
Mean pool + MaxSim rerank	0.251	0.184	0.144	42.0%

SciFact — ColBERTv2 (5,183 docs, 300 queries)

Approach	NDCG@1	NDCG@5	NDCG@10	% of Exact
Exact MaxSim (brute-force)	0.597	0.674	0.692	100%
MUVERA + MaxSim rerank (4x)	0.590	0.655	0.671	97.1%
Mean pool + MaxSim rerank	0.350	0.359	0.363	52.4%

IRPAPERS — ColModernVBERT (3,230 pages, 180 queries, ~1,011 vectors/page)

Approach	R@1	R@5	R@10	R@20	% of Exact (R@1)
Exact MaxSim (brute-force)	40.6%	76.7%	83.9%	89.4%	100%
MUVERA + MaxSim rerank (4x)	37.2%	66.7%	73.3%	78.3%	91.7%

MUVERA+rerank recovers 90-97% of exact MaxSim quality across all three datasets, including visual documents with ~1,000 multi-vectors per page. Mean pooling retains only 42-52%, demonstrating why MUVERA is needed.

Full results: https://github.com/praveenMprasad/OpenSearch-k-NN/blob/feature/muvera-benchmark/benchmarks/muvera/BENCHMARK_RESULTS.md

[mingshl]

quick question, in step 5,

Step 5: Search why not using the new muvera_fde embedding for knn search? have you try running the knn query with the lateInteractionScore rerank is working?

[praveenMprasad]

quick question, in step 5,

Step 5: Search why not using the new muvera_fde embedding for knn search? have you try running the knn query with the lateInteractionScore rerank is working?

The search query in Step 5 intentionally uses script_score with match_all + lateInteractionScore — the MUVERA search pipeline intercepts this and rewrites it under the hood. Here's what happens:

1. User sends script_score with match_all and query_vectors in params
2. The muvera_query search processor:
   • Extracts query_vectors from the script params
   • MUVERA-encodes them into a query FDE
   • Replaces match_all with knn on muvera_fde (k = size × oversample_factor)
   • Keeps the lateInteractionScore script wrapper intact
3. OpenSearch executes the rewritten query: KNN prefetch on FDE → MaxSim rerank on original multi-vectors

So the user doesn't need to know about FDE encoding — they just send their raw query multi-vectors and the pipeline handles the FDE encoding + KNN rewrite transparently. The muvera_fde field is used for KNN search, just not explicitly in the user's query.

And yes, this is verified end-to-end — the integration tests confirm that the pipeline rewrites the query correctly and lateInteractionScore reranking produces proper MaxSim scores on the KNN candidates. The benchmark results above also confirm quality (90-97% of exact MaxSim).

[wdongyu]

Curious, can we introduce a new query like neural query for multi_vector? So user can explicitly specify any query(knn query or match_all query). Why we need to rewrite the query transparently?

[heemin32]

We could simplify the user interface by implementing a new field like the semantic text field.

1. Index creation

PUT muvera-index
{
  "settings": {
    "index.knn": true
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "colbert_vectors": {
        "type": "muvera",
        "dimention": 128,
        "k_sim": 4, //optional
        "dim_proj" : 8, //optional
        "r_reps": 20 //optional
        "seed": 42 //optional
        "method": {
          "name": "hnsw",
          "space_type": "innerproduct",
          "engine": "faiss"
        }
      },
      "title": { "type": "text" }
    }
  }
}

2. Ingestion

POST muvera-index/_doc/1
{
  "title": "example document",
  "colbert_vectors": [
    [0.1, 0.2, ...],
    [0.3, 0.4, ...],
    [0.5, 0.6, ...]
  ]
}

3. Query

PUT _search/pipeline/muvera-search
{
 "muvera": {
  "colbert_vectors": {
    "oversample_factor": 4 // optional
    "k": 10, // optional
    "vectors": [[0.1, 0.4, ...], [2.3, 3.3, ...], [1.1, 3.2, ...]]
  }
 }
}