marqo_original_docs_merged.txt

List Indexes

GET /indexes
List indexes
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl http://localhost:8882/indexes


Response: 200 OK

{
  "results": [
    {
      "indexName": "Book Collection"
    },
    {
      "indexName": "Animal facts"
    }
  ]
}

---

Delete index
Delete an index.

Index creation and deletion can not be called concurrently. If you try to delete an index when there is an ongoing creation or deletion of an index, the request will fail, you will receive a 409 OperationConflictError.

Note: This operation cannot be undone, and the deleted index can't be recovered


DELETE /indexes/{index_name}
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XDELETE http://localhost:8882/indexes/my-first-index

Response: 200 OK

{"acknowledged": true}

---

Modify index
In Marqo Cloud you can modify the number of inference nodes numberOfInferences, the type of inference node inferenceType, the number of storage shards numberOfShards, the number of replicas numberOfReplicas. Support to update the type of storage class storageClass will be added soon. These are body parameters that are specific to Marqo Cloud only and so this page is specifically for Marqo Cloud.

If you're looking to modify documents in either Marqo Open Source or Marqo Cloud, please see Update Documents.


Marqo Cloud
You can modify the settings of an existing index, such as the number of inference nodes or the type of inference node.


PUT https://api.marqo.ai/api/v2/indexes/{index_name}
Example

cURL

curl -XPUT 'https://api.marqo.ai/api/v2/indexes/my-first-index' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"numberOfInferences": 1,
"inferenceType": "marqo.CPU.large",
"numberOfShards": 2,
"numberOfReplicas": 1,
"storageClass": "marqo.basic"
}'

Response: 200 OK

{"acknowledged":true}
Path parameters
Name  Type  Description
index_name  String  name of the index
Body Parameters
The settings for the index. The settings are represented as a nested JSON object.

Name  Type  Default value Description
inferenceType String  marqo.CPU.small Type of inference for the index. Options are "marqo.CPU.small"(deprecated), "marqo.CPU.large", "marqo.GPU".
numberOfInferences  Integer 1 Defines the number of inference nodes for the index. The minimum value is 0, and the maximum value is 5 by default, but this is dependent on your account limits.
numberOfShards  Integer 1 Defines the number of shards for the index. The minimum value is equal to the current number of shards for the index, and the maximum value is 5 by default, but this is dependent on your account limits.
numberOfReplicas  Integer 1 Defines the number of replicas for the index. The minimum value is equal to the current number of replicas for the index, and the maximum value is 1 by default, but this is dependent on your account limits.
storageClass  String  marqo.basic Defines the storage class for the index. Permisible values are marqo.basic, marqo.balanced and marqo.performance. The value must be equal to the current storage class for the index, but support to change this value is coming soon.


---


Create Index
Create index with (optional) settings. This endpoint accepts the application/json content type.


POST /indexes/{index_name}
Index creation and deletion can not be called concurrently. If you try to create an index when there is an ongoing creation or deletion of an index, the request will fail, you will receive a 409 OperationConflictError.

Marqo Cloud creates dedicated infrastructure for each index. Using the create index endpoint, you can specify the type of storage for the index storageClass and the type of inference inferenceType. The number of storage instances is defined by numberOfShards, the number of replicas numberOfReplicas and the number of Marqo inference nodes by numberOfInferences. This is only supported for Marqo Cloud, not Marqo Open Source.

Example

Marqo Open Source
Marqo Cloud
This is an example of creating an index with Marqo Open Source:


cURL
Python

curl -X POST 'http://localhost:8882/indexes/my-first-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "type": "unstructured"
}'

Response: 200 OK

{"acknowledged":true, "index":"my-first-index"}

Path parameters
Name	Type	Description
index_name
String	name of the index
Body Parameters
The settings for the index are represented as a nested JSON object that contains the default settings for the index. The parameters are as follows:

Name	Type	Default value	Description
treatUrlsAndPointersAsImages	Boolean	False	Fetch images from pointers
treatUrlsAndPointersAsMedia*	Boolean	False	Fetch images, videos, and audio from pointers
model	String	hf/e5-base-v2	The model to use to vectorise doc content in add_documents() calls for the index. To create index with a Marqtuned model, please specify the model name as marqtune/{model_id}/{released_checkpoint_name}. More details can be found here
modelProperties	Dictionary	""	The model properties object corresponding to model (for custom models). Check here on how to bring your own models.
normalizeEmbeddings
Boolean	true	Normalize the embeddings to have unit length
textPreprocessing	Dictionary	""	The text preprocessing object
imagePreprocessing	Dictionary	""	The image preprocessing object
videoPreprocessing	Dictionary	""	The video preprocessing object
audioPreprocessing	Dictionary	""	The audio preprocessing object
annParameters	Dictionary	""	The ANN algorithm parameter object
type	String	unstructured	Type of the index
vectorNumericType	String	float	Numeric type for vector encoding
filterStringMaxLength	Int	50	Specifies the maximum character length allowed for strings used in filtering queries within unstructured indexes. This means that any string field you intend to use as a filter in these indexes should not exceed 50 characters in length.
textChunkPrefix	String	"" or model default	The prefix added to indexed text document chunks when embedding.
textQueryPrefix	String	"" or model default	The prefix added to text queries when embedding.
* treatUrlsAndPointersAsMedia is a new parameter introduced in Marqo 2.12 to support the new modalities of video and audio. Here is how it interacts with treatUrlsAndPointersAsImages:

Both False: All content is processed as text only.
treatUrlsAndPointersAsImages True, treatUrlsAndPointersAsMedia False:
Processes URLs and pointers as images
Does not process other media types (video, audio)
treatUrlsAndPointersAsImages False, treatUrlsAndPointersAsMedia True:
Invalid state since this is a conflict.
Both True:
Processes URLs and pointers as various media types (images, videos, audio)
Note: these body parameters are used in both Marqo Open Source and Marqo Cloud. Marqo Cloud also has additional body parameters. Let's take a look at those now.

Additional Marqo Cloud Body Parameters
Marqo Cloud creates dedicated infrastructure for each index. Using the create index endpoint, you can specify the type of storage for the index storageClass and the type of inference inferenceType. The number of storage instances is defined by numberOfShards, the number of replicas numberOfReplicas and the number of Marqo inference nodes by numberOfInferences. This is only supported for Marqo Cloud, not Marqo Open Source.

Name	Type	Default value	Description	Open Source	Cloud
inferenceType	String	marqo.CPU.small	Type of inference for the index. Options are "marqo.CPU.small"(deprecated), "marqo.CPU.large", "marqo.GPU".	❌	✅
storageClass	String	marqo.basic	Type of storage for the index. Options are "marqo.basic", "marqo.balanced", "marqo.performance".	❌	✅
numberOfShards	Integer	1	The number of shards for the index.	❌	✅
numberOfReplicas	Integer	0	The number of replicas for the index.	❌	✅
numberOfInferences	Integer	1	The number of inference nodes for the index.	❌	✅
Text Preprocessing Object
The textPreprocessing object contains the specifics of how you want the index to preprocess text. The parameters are as follows:

Name	Type	Default value	Description
splitLength	Integer	2	The length of the chunks after splitting by split_method
splitOverlap
Integer	0	The length of overlap between adjacent chunks
splitMethod	String	sentence	The method by which text is chunked (character, word, sentence, or passage)
Image Preprocessing Object
The imagePreprocessing object contains the specifics of how you want the index to preprocess images. The parameters are as follows:

Name	Type	Default value	Description
patchMethod
String	null	The method by which images are chunked (simple or frcnn)
Video Preprocessing Object
The videoPreprocessing object contains the specifics of how you want the index to preprocess videos. The last chunk in the video file will have a start time of the total length of the video file minus the split length.

The parameters are as follows:

Name	Type	Default value	Description
splitLength	Integer	20	The length of the video chunks in seconds after splitting by split_method
splitOverlap
Integer	3	The length of overlap in seconds between adjacent chunks
Audio Preprocessing Object
The audioPreprocessing object contains the specifics of how you want the index to preprocess audio. The last chunk in the audio file will have a start time of the total length of the audio file minus the split length.

The parameters are as follows:

Name	Type	Default value	Description
splitLength	Integer	10	The length of the audio chunks in seconds after splitting by split_method
splitOverlap
Integer	3	The length of overlap in seconds between adjacent chunks
ANN Algorithm Parameter object
The annParameters object contains hyperparameters for the approximate nearest neighbour algorithm used for tensor storage within Marqo. The parameters are as follows:

Name	Type	Default value	Description
spaceType
String	prenormalized-angular	The function used to measure the distance between two points in ANN (angular, euclidean, dotproduct, geodegrees, hamming, or prenormalized-angular).
parameters	Dict	""	The hyperparameters for the ANN method (which is always hnsw for Marqo).
HNSW Method Parameters Object
parameters can have the following values:

Name	Type	Default value	Description
efConstruction
int	512	The size of the dynamic list used during k-NN graph creation. Higher values lead to a more accurate graph but slower indexing speed. It is recommended to keep this between 2 and 800 (maximum is 4096)
m	int	16	The number of bidirectional links that the plugin creates for each new element. Increasing and decreasing this value can have a large impact on memory consumption. Keep this value between 2 and 100.
Model Properties Object
This flexible object, used by modelProperties is used to set up models that aren't available in Marqo by default ( models available by default are listed here). The structure of this object will vary depending on the model.

For OpenCLIP models, see here for modelProperties format and example usage.

For Generic SBERT models, see here for modelProperties format and example usage.

Prefixes in Index Settings
Parameters: textChunkPrefix, textQueryPrefix

Expected value: A string.

Default value: ""

These fields override the model's default prefixes for text documents and queries. URLs pointing to images are not affected by these prefixes. If these fields are left undefined, Marqo will use the model's default prefixes. Currently, only the e5 series models have default prefixes defined.

Indexes built on Marqo 2.5 and below will not have prefixes added to any new documents, embeddings, or queries when read with Marqo 2.6 and above, even if the index’s model has default prefixes set.

Currently, Marqo adds the prefixes by default to e5 models since these are trained on data with prefixes. So, adding them to text chunks before embedding improves the quality of the embeddings. For more information, refer to the model card here

Example: Setting text chunk and query prefixes during index creation

Marqo Open Source
Marqo Cloud

cURL
Python

curl -X POST 'http://localhost:8882/indexes/my-first-index' \
-H "Content-Type: application/json" \
-d '{
    "textChunkPrefix": "override passage: ",
    "textQueryPrefix": "override query: ",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "type": "unstructured"
}'


Example Settings Object
Below is a sample index settings JSON object. When using the Python client, pass this dictionary as the settings_dict parameter for the create_index method.


{
  "type": "unstructured",
  "vectorNumericType": "float",
  "treatUrlsAndPointersAsImages": true,
  "model": "open_clip/ViT-L-14/laion2b_s32b_b82k",
  "normalizeEmbeddings": true,
  "textPreprocessing": {
    "splitLength": 2,
    "splitOverlap": 0,
    "splitMethod": "sentence"
  },
  "imagePreprocessing": {
    "patchMethod": null
  },
  "annParameters": {
    "spaceType": "prenormalized-angular",
    "parameters": {
      "efConstruction": 512,
      "m": 16
    }
  },
  "filterStringMaxLength": 20
}

---

Create Structured Index
Structured indexes in Marqo are tailored for datasets with a defined schema and are particularly effective for complex queries like sorting, grouping, and filtering. They are designed for fast, in-memory operations.

To create your structured index:


POST /indexes/{index_name}
Create index with (optional) settings. This endpoint accepts the application/json content type.

Path parameters
Name  Type  Description
index_name
String  Name of the index
Body Parameters
The settings for the index are represented as a nested JSON object that contains the default settings for the index. The parameters are as follows:

Name  Type  Default value Description
allFields List  - List of fields that might be indexed or queried. Valid only if type is structured
tensorFields  List  []  List of fields that are treated as tensors
model String  hf/e5-base-v2 The model to use to vectorise doc content in add_documents() calls for the index
modelProperties Dictionary  ""  The model properties object corresponding to model (for custom models)
normalizeEmbeddings
Boolean true  Normalize the embeddings to have unit length
textPreprocessing Dictionary  ""  The text preprocessing object
imagePreprocessing  Dictionary  ""  The image preprocessing object
videoPreprocessing  Dictionary  ""  The video preprocessing object
audioPreprocessing  Dictionary  ""  The audio preprocessing object
annParameters Dictionary  ""  The ANN algorithm parameter object
type  String  unstructured  Type of the index. The default value is unstructured, but for the structured index this needs to be structured
vectorNumericType String  float Numeric type for vector encoding
Note: these body parameters are used in both Marqo Open-Source and Marqo Cloud. Marqo Cloud also has additional body parameters. Let's take a look at those now.

Additional Marqo Cloud Body Parameters
Marqo Cloud creates dedicated infrastructure for each index. Using the create index endpoint, you can specify the type of storage for the index storageClass and the type of inference inferenceType. The number of storage instances is defined by numberOfShards, the number of replicas numberOfReplicas and the number of Marqo inference nodes by numberOfInferences. This is only supported for Marqo Cloud, not Marqo Open-Source.

Name  Type  Default value Description Open Source Cloud
inferenceType String  marqo.CPU.small Type of inference for the index. Options are "marqo.CPU.small"(deprecated), "marqo.CPU.large", "marqo.GPU". ❌ ✅
storageClass  String  marqo.basic Type of storage for the index. Options are "marqo.basic", "marqo.balanced", "marqo.performance".  ❌ ✅
numberOfShards  Integer 1 The number of shards for the index. ❌ ✅
numberOfReplicas  Integer 0 The number of replicas for the index. ❌ ✅
numberOfInferences  Integer 1 The number of inference nodes for the index.  ❌ ✅
Fields
The allFields object contains the fields that might be indexed or queried. Each field has the following parameters:

Name  Type  Default value Description
name  String  - Name of the field
type  String  - Type of the field
features  List  []  List of features that the field supports
Available types are:

Field Type  Description Supported Features
text  Text field  lexical_search, filter
int 32-bit integer  filter, score_modifier
float 32-bit float  filter, score_modifier
long  64-bit integer  filter, score_modifier
double  64-bit float  filter, score_modifier
array<text> Array of text lexical_search, filter
array<int>  Array of 32-bit integers  filter
array<float>  Array of 32-bit floats  filter
array<long> Array of 64-bit integers  filter
array<double> Array of 64-bit floats  filter
bool  Boolean filter
multimodal_combination  Multimodal combination field  None
image_pointer Image URL. Must only be used with a multimodal model such as CLIP None
video_pointer Video URL. Must only be used with a multimodal model such as LanguageBind None
audio_pointer Audio URL. Must only be used with a multimodal model such as LanguageBind None
custom_vector Custom vector, with optional text for lexical/filtering lexical_search, filter
map<text, int>  Map of text to integers score_modifier
map<text, long> Map of text to longs  score_modifier
map<text, float>  Map of text to floats score_modifier
map<text, double> Map of text to doubles  score_modifier
Available features are:

lexical_search: The field can be used for lexical search
filter: The field can be used for exact and range (numerical fields) filtering
score_modifier: The field can be used to modify the score of the document
When using multimodal_combination fields, the dependentFields object is used to define the weights for the multimodal combination field and is required. The dependentFields object is a dictionary where the keys are the names of the fields that are used to create the multimodal combination field and the values are the weights for each field. Field names must refer to fields that are defined in allFields. See the example below for more details.

Text Preprocessing Object
The textPreprocessing object contains the specifics of how you want the index to preprocess text. The parameters are as follows:

Name  Type  Default value Description
splitLength Integer 2 The length of the chunks after splitting by split_method
splitOverlap
Integer 0 The length of overlap between adjacent chunks
splitMethod String  sentence  The method by which text is chunked (character, word, sentence, or passage)
Image Preprocessing Object
The imagePreprocessing object contains the specifics of how you want the index to preprocess images. The parameters are as follows:

Name  Type  Default value Description
patchMethod
String  null  The method by which images are chunked (simple or frcnn)
Video Preprocessing Object
The videoPreprocessing object contains the specifics of how you want the index to preprocess videos. The last chunk in the video file will have a start time of the total length of the video file minus the split length.

The parameters are as follows:

Name  Type  Default value Description
splitLength Integer 20  The length of the video chunks in seconds after splitting by split_method
splitOverlap
Integer 3 The length of overlap in seconds between adjacent chunks
Audio Preprocessing Object
The audioPreprocessing object contains the specifics of how you want the index to preprocess audio. The last chunk in the audio file will have a start time of the total length of the audio file minus the split length.

The parameters are as follows:

Name  Type  Default value Description
splitLength Integer 10  The length of the audio chunks in seconds after splitting by split_method
splitOverlap
Integer 3 The length of overlap in seconds between adjacent chunks
ANN Algorithm Parameter object
The annParameters object contains hyperparameters for the approximate nearest neighbour algorithm used for tensor storage within Marqo. The parameters are as follows:

Name  Type  Default value Description
spaceType
String  prenormalized-angular The function used to measure the distance between two points in ANN (angular, euclidean, dotproduct, geodegrees, hamming, or prenormalized-angular).
parameters  Dict  ""  The hyperparameters for the ANN method (which is always hnsw for Marqo).
HNSW Method Parameters Object
parameters can have the following values:

Name  Type  Default value Description
efConstruction
int 512 The size of the dynamic list used during k-NN graph creation. Higher values lead to a more accurate graph but slower indexing speed. It is recommended to keep this between 2 and 800 (maximum is 4096)
m int 16  The number of bidirectional links that the plugin creates for each new element. Increasing and decreasing this value can have a large impact on memory consumption. Keep this value between 2 and 100.
Model Properties Object
This flexible object, used by modelProperties is used to set up models that aren't available in Marqo by default ( models available by default are listed here). The structure of this object will vary depending on the model.

For OpenCLIP models, see here for modelProperties format and example usage.

For Generic SBERT models, see here for modelProperties format and example usage.

Example 1: Creating a structured index for combining text and images

Marqo Open-Source
Marqo Cloud

cURL
python

curl -X POST 'http://localhost:8882/indexes/my-first-structured-index' \
-H "Content-Type: application/json" \
-d '{
    "type": "structured",
    "vectorNumericType": "float",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": true,
    "textPreprocessing": {
        "splitLength": 2,
        "splitOverlap": 0,
        "splitMethod": "sentence"
    },
    "allFields": [
        {"name": "text_field", "type": "text", "features": ["lexical_search"]},
        {"name": "caption", "type": "text", "features": ["lexical_search", "filter"]},
        {"name": "tags", "type": "array<text>", "features": ["filter"]},
        {"name": "image_field", "type": "image_pointer"},
        {"name": "my_int", "type": "int", "features": ["score_modifier"]},
        {
            "name": "multimodal_field",
            "type": "multimodal_combination",
            "dependentFields": {"image_field": 0.9, "text_field": 0.1}
        }
    ],
    "tensorFields": ["multimodal_field"],
    "annParameters": {
        "spaceType": "prenormalized-angular",
        "parameters": {"efConstruction": 512, "m": 16}
    }
}'


Example 2: Creating a structured index with no model for use with custom vectors

Marqo Open-Source
Marqo Cloud

cURL
python

curl -X POST 'http://localhost:8882/indexes/my-hybrid-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "no_model",
    "modelProperties": {
        "type": "no_model",
        "dimensions": 3072
    },
    "type": "structured",
    "allFields": [
        {"name": "title", "type": "custom_vector", "features": ["lexical_search"]},
        {"name": "description", "type": "text", "features": ["lexical_search", "filter"]},
        {"name": "time_added_epoch", "type": "int", "features": ["score_modifier"]}
    ],
    "tensorFields": ["title"]
}'

---

Add or Replace Documents
Add an array of documents or replace them if they already exist.

If you send a document with an _id that corresponds to an existing document, the new document will overwrite the existing document.

This endpoint accepts the application/json content type.


POST /indexes/{index_name}/documents
Path parameters
Name  Type  Description
index_name  String  name of the index
Query parameters
Query Parameter Type  Default Value Description
device  String  null  The device used to index the documents. If device is not specified and CUDA devices are available to Marqo (see here for more info), Marqo will speed up the indexing process by using available CUDA devices. Otherwise, the CPU will be used. Options include cpu and cuda, cuda1, cuda2 etc. The cuda option tells Marqo to use any available cuda devices.
telemetry Boolean False If true, the telemetry object is returned in the add documents response body. This includes information like latency metrics. This is set at client instantiation time in the Python client: mq = marqo.Client(return_telemetry=True)
Body
In the RestAPI and for curl users these parameters are in lowerCamelCase, as presented in the following table. The Python client uses the pythonic snake_case equivalents.

Add documents parameters  Value Type  Default Value Description
documents Array of objects  n/a An array of documents. Each document is represented as a JSON object. You can optionally set a document's ID with the special _id field. The _id must be a string type. If an ID is not specified, marqo will generate one.
tensorFields
Array of Strings  []  Structured indexes only support tensor fields at the time of their creation, while unstructured indexes can include these fields during the document-adding process. The fields within these documents which will be tensor fields, and therefore will have vectors generated for them. Tensor search can only be performed on these fields for these documents. Pre-filtering and lexical search are still viable on text fields which are not included in the tensorFields parameter. For the best recall and speed performance, we recommend minimising the number of different tensor fields for your index. For production use cases where speed and recall are critical, we recommend only a single tensor field for the entire index.
useExistingTensors  Boolean false Setting this to true will get existing tensors for unchanged fields in documents that are indexed with an id. Note: Marqo analyses the field string for updates, so Marqo can't detect a change if a URL points to a different image.
imageDownloadHeaders (deprecated) Dict  null  An object that consists of key-value pair headers for image download. Can be used to authenticate the images for download.
mediaDownloadHeaders  Dict  null  An object that consists of key-value pair headers for media download. Can be used to authenticate the all types of media for download.
mappings  Dict  null  An object to handle object fields in documents. Check mappings for more information. Mappings are required to create multimodal combination and custom vector fields - see here for more information
modelAuth Dict  null  An object that consists of authorisation details used by Marqo to download non-publicly available models. Check here for more information.
clientBatchSize Integer null  A Python client only helper parameter that splits up very large lists of documents into batches of a more manageable size for Marqo.
textChunkPrefix String  null  The prefix added to indexed text document chunks when embedding. Setting this field overrides the textChunkPrefix set in the index settings during index creation. If it unset by the user, it defaults to the prefixes defined in the index settings. For more information on default values for index settings, see create_index.
Additional Marqo Cloud Body Parameters
Marqo Cloud creates dedicated infrastructure for each index. Using the create index endpoint, you can specify the type of storage for the index storageClass and the type of inference inferenceType. The number of storage instances is defined by numberOfShards, the number of replicas numberOfReplicas and the number of Marqo inference nodes by numberOfInferences. This is only supported for Marqo Cloud, not Marqo Open Source.

Name  Type  Default value Description Open Source Cloud
inferenceType String  marqo.CPU.small Type of inference for the index. Options are "marqo.CPU.small"(deprecated), "marqo.CPU.large", "marqo.GPU". ❌ ✅
storageClass  String  marqo.basic Type of storage for the index. Options are "marqo.basic", "marqo.balanced", "marqo.performance".  ❌ ✅
numberOfShards  Integer 1 The number of shards for the index. ❌ ✅
numberOfReplicas  Integer 0 The number of replicas for the index. ❌ ✅
numberOfInferences  Integer 1 The number of inference nodes for the index.  ❌ ✅
Response
The response of the add_or_replace_documents endpoint in Marqo operates on two levels. Firstly, a status code of 200 in the overall response indicates that the batch request has been successfully received and processed by Marqo. The response has the following fields:

Field Name  Type  Description
errors  Boolean Indicates whether any errors occurred during the processing of the batch request.
items Array An array of objects, each representing the processing status of an individual document in the batch.
processingTimeMs  Integer The time taken to process the batch request, in milliseconds.
index_name  String  The name of the index to which the documents were added.
However, a 200 status does not necessarily imply that each individual document within the batch was processed without issues. For each document in the batch, there will be an associated response code that specifies the status of that particular document's processing. These individual response codes provide granular feedback, allowing users to discern which documents were successfully processed, which encountered errors, and the nature of any issues encountered.

Each item in the items array has the following fields:

Field Name  Type  Description
_id String  The ID of the document that was processed.
status  Integer The status code of the document processing.
message String  A message that provides additional information about the processing status of the document. This field only exists when the status is not 200.
Here is the HTTP status code of the individual document responses (non-exhaustive list of status codes):

Status Code Description
200 The document is successfully added to the index.
400 Bad request. Returned for invalid input (e.g., invalid field types). Inspect message for details.
429 Marqo vector store receives too many requests. Please try again later.
500 Internal error.
Example
For unstructured index:


Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-index/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [ 
    {
        "Title": "The Travels of Marco Polo",
        "Description": "A 13th-century travelogue describing the travels of Polo",
        "Genre": "History"
        }, 
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection",
        "_id": "article_591",
        "Genre": "Science"
    }
],
"tensorFields": ["Description"]
}'


For structured index:


Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-structured-index/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [ 
    {
        "Title": "The Travels of Marco Polo",
        "Description": "A 13th-century travelogue describing the travels of Polo",
        "Genre": "History"
        }, 
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection",
        "_id": "article_591",
        "Genre": "Science"
    }
]
}'


Response: 200 OK

{
  "errors": false,
  "items": [
    {
      "_id": "5aed93eb-3878-4f12-bc92-0fda01c7d23d",
      "status": 200
    },
    {
      "_id": "article_591",
      "status": 200
    }
  ],
  "processingTimeMs": 6,
  "index_name": "my-first-index"
}
The first document in this example had its _id generated by Marqo. In this example, there was already a document in Marqo with _id = article_591, so it was updated rather than created. In unstructured index we want Description to be a searchable with tensor search (Marqo's default search), so we explicitly declare it as a tensor field. In structured index the tensor fields are specified during index creation, so we don't need to specify them here. Tensor fields are stored alongside vector representation of the data, allowing for multimodal and semantic searches.

If you would like to see an example of adding video and audio documents, please visit this section.

Documents
Parameter: documents

Expected value: An array of documents (default maximum length: 128). Each document is a JSON object that is to be added to the index. Each key is the name of a document's field and its value is the content for that field. See here for the allowed field data types. The optional _id key can be used to specify a string as the document's ID.

Map Fields
Only flat numeric dictionaries with int, long, float, and double values are currently supported as document fields.


[
  {
    "Title": "The Travels of Marco Polo",
    "Description": "A 13th-century travelogue describing Polo's travels",
  },
  {
    "Title": "Extravehicular Mobility Unit (EMU)",
    "Description": "The EMU is a spacesuit that provides environmental protection",
    "_id": "article_591"
  },
  {
    "Title": "The Travels of Marco Polo",
    "Description": "A 13th-century travelogue describing Polo's travels",
    "map_numeric_field": {
        "popularity": 56.4,
        "availability": 0.9,
        "year_published": 1300,
    }
  },
]
Mappings
Parameter: mappings

Expected value: JSON object with field names as keys, mapped to objects with type (currently only multimodal_combination and custom_vector are supported). Multimodal combination mappings also have weights, which is an object that maps each nested field to a relative weight.

Default value: null

The mappings object allows adding special fields, such as: multimodal fields, custom vector fields, and map score modifiers.

With multimodal fields, child fields are vectorised and combined into a single tensor via weighted-sum approach using the weights object. The combined tensor will be used for tensor search.

With custom vector fields, vectors can be directly inserted into documents. This is useful if you are generating your vectors outside of marqo.

With map score modifiers, child fields values can be of type int, long, float, or double. These values will be used in the score modifier computation during search.

All multimodal combination or custom vector fields must be in tensor_fields.

Dependent fields can be used for lexical search or vector search with filtering. Dependent fields can only have content of type str, representing a text or a pointer (URL) to an image.

The mappings is optional with structured indexes and is only needed if the user needs to override default multimodal weights defined at index creation time. Additionally, custom vector fields should not be declared in mappings for structured indexes.

Read more about using mappings and special fields here

Example: Multimodal Combination
Unstructured Index (Default)

Marqo Open Source
Marqo Cloud

cURL
Python

# Create an unstructured index (default)
curl -X POST 'http://localhost:8882/indexes/my-first-index' \
-H "Content-Type: application/json" \
-d '{
    "treatUrlsAndPointersAsImages": true,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k"
}'

# Add documents with mappings to specify multimodal combination fields
curl -X POST 'http://localhost:8882/indexes/my-first-index/documents' \
-H "Content-Type: application/json" \
-d '{
    "documents":[
        {
            "img": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg",
            "caption": "A man riding horse"
        },
        {
            "img": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg",
            "caption": "An airplane flying in the sky"
        }
    ],
    "mappings": {
        "my_combination_field": {
            "type": "multimodal_combination",
            "weights": {
                "img": 0.9, "caption": 0.1
            }
        }
    },
    "tensorFields": ["my_combination_field"]
}'


Structured Index

Marqo Open Source
Marqo Cloud

cURL
Python

# Alternatively you can create a structured index with multimodal combination fields
curl -X POST 'http://localhost:8882/indexes/my-first-structured-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "type": "structured",
    "allFields": [
        {"name": "caption", "type": "text"}, 
        {"name": "img", "type": "image_pointer"},
        {"name": "my_combination_field", "type": "multimodal_combination", 
        "dependentFields": {"caption": 0.5, "img": 0.5}}
    ],
    "tensorFields": ["my_combination_field"]
}'

# Add documents
# The mappings object is optional with structured indexes and is only needed if the user needs to
# override default multimodal weights defined at index creation time.
curl -X POST 'http://localhost:8882/indexes/my-first-structured-index/documents' \
-H "Content-Type: application/json" \
-d '{
    "documents":[
        {
            "img": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg",
            "caption": "A man riding horse"
        },
        {
            "img": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg",
            "caption": "An airplane flying in the sky"
        }
    ],
    "mappings": {
        "my_combination_field": {
            "type": "multimodal_combination",
            "weights": {
                "img": 0.6, "caption": 0.4
            }
        }
    }
}'


Example: Custom Vectors
(Replace 'vector' field value with your own vectors!)

Unstructured Index (Default)

Marqo Open Source
Marqo Cloud

cURL
Python

# Create an index with the model that has the dimensions of your custom vectors. For example: "open_clip/ViT-B-32/laion2b_s34b_b79k" (dimension is 512). 
# Only the model dimension matters, as we are not vectorising anything when using custom vector fields.
# Space type CANNOT be 'prenormalized-angular' for custom vectors, as they are not normalized.
curl -X POST 'http://localhost:8882/indexes/my-first-index' \
-H "Content-Type: application/json" \
-d '{
    "treatUrlsAndPointersAsImages": true,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "annParameters": {
        "spaceType": "angular",
        "parameters": {"efConstruction": 512, "m": 16}
    }
}'

# We add the custom vector documents into our index (with mappings)
curl -X POST 'http://localhost:8882/indexes/my-first-index/documents' \
-H "Content-Type: application/json" \
-d '{
    "documents":[
        {
            "_id": "doc1",
            "my_custom_vector": {
                "vector": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511],
                "content": "Singing audio file"
            }
        },
        {
            "_id": "doc2",
            "my_custom_vector": {
                "vector": [1.0, 0.5, 0.3333, 0.25, 0.2, 0.1667, 0.1429, 0.125, 0.1111, 0.1, 0.0909, 0.0833, 0.0769, 0.0714, 0.0667, 0.0625, 0.0588, 0.0556, 0.0526, 0.05, 0.0476, 0.0455, 0.0435, 0.0417, 0.04, 0.0385, 0.037, 0.0357, 0.0345, 0.0333, 0.0323, 0.0312, 0.0303, 0.0294, 0.0286, 0.0278, 0.027, 0.0263, 0.0256, 0.025, 0.0244, 0.0238, 0.0233, 0.0227, 0.0222, 0.0217, 0.0213, 0.0208, 0.0204, 0.02, 0.0196, 0.0192, 0.0189, 0.0185, 0.0182, 0.0179, 0.0175, 0.0172, 0.0169, 0.0167, 0.0164, 0.0161, 0.0159, 0.0156, 0.0154, 0.0152, 0.0149, 0.0147, 0.0145, 0.0143, 0.0141, 0.0139, 0.0137, 0.0135, 0.0133, 0.0132, 0.013, 0.0128, 0.0127, 0.0125, 0.0123, 0.0122, 0.012, 0.0119, 0.0118, 0.0116, 0.0115, 0.0114, 0.0112, 0.0111, 0.011, 0.0109, 0.0108, 0.0106, 0.0105, 0.0104, 0.0103, 0.0102, 0.0101, 0.01, 0.0099, 0.0098, 0.0097, 0.0096, 0.0095, 0.0094, 0.0093, 0.0093, 0.0092, 0.0091, 0.009, 0.0089, 0.0088, 0.0088, 0.0087, 0.0086, 0.0085, 0.0085, 0.0084, 0.0083, 0.0083, 0.0082, 0.0081, 0.0081, 0.008, 0.0079, 0.0079, 0.0078, 0.0078, 0.0077, 0.0076, 0.0076, 0.0075, 0.0075, 0.0074, 0.0074, 0.0073, 0.0072, 0.0072, 0.0071, 0.0071, 0.007, 0.007, 0.0069, 0.0069, 0.0068, 0.0068, 0.0068, 0.0067, 0.0067, 0.0066, 0.0066, 0.0065, 0.0065, 0.0065, 0.0064, 0.0064, 0.0063, 0.0063, 0.0063, 0.0062, 0.0062, 0.0061, 0.0061, 0.0061, 0.006, 0.006, 0.006, 0.0059, 0.0059, 0.0058, 0.0058, 0.0058, 0.0057, 0.0057, 0.0057, 0.0056, 0.0056, 0.0056, 0.0056, 0.0055, 0.0055, 0.0055, 0.0054, 0.0054, 0.0054, 0.0053, 0.0053, 0.0053, 0.0053, 0.0052, 0.0052, 0.0052, 0.0052, 0.0051, 0.0051, 0.0051, 0.0051, 0.005, 0.005, 0.005, 0.005, 0.0049, 0.0049, 0.0049, 0.0049, 0.0048, 0.0048, 0.0048, 0.0048, 0.0047, 0.0047, 0.0047, 0.0047, 0.0047, 0.0046, 0.0046, 0.0046, 0.0046, 0.0045, 0.0045, 0.0045, 0.0045, 0.0045, 0.0044, 0.0044, 0.0044, 0.0044, 0.0044, 0.0043, 0.0043, 0.0043, 0.0043, 0.0043, 0.0043, 0.0042, 0.0042, 0.0042, 0.0042, 0.0042, 0.0041, 0.0041, 0.0041, 0.0041, 0.0041, 0.0041, 0.004, 0.004, 0.004, 0.004, 0.004, 0.004, 0.004, 0.0039, 0.0039, 0.0039, 0.0039, 0.0039, 0.0039, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002],
                "content": "Podcast audio file"
            }
        }
    ],
    "mappings": {
        "my_custom_vector": {
            "type": "custom_vector"
        }
    },
    "tensorFields": ["my_custom_vector"]
}'
---

Get multiple documents
Gets a selection of documents based on their IDs.

This endpoint accepts the application/json content type.


GET /indexes/{index_name}/documents
Path parameters
Name  Type  Description
index_name  String  name of the index
Query parameters
Search parameter  Type  Default value Description
expose_facets
Boolean False If true, the documents' tensor facets are returned. This is a list of objects. Each facet object contains document data and its associated embedding (found in the facet's _embedding field)
Response
The response of the get_multiple_documents endpoint in Marqo operates on two levels. Firstly, a status code of 200 in the overall response indicates that the batch request has been successfully received and processed by Marqo.

The response has the following fields:

Field Type  Description
results Array An array of objects, each representing a document. Each object contains the document's data.
However, a 200 status does not necessarily imply that each individual document within the batch was processed without issues. For each document in the batch, there will be an associated response code that specifies the status of that particular document's processing. These individual response codes provide granular feedback, allowing users to discern which documents were successfully processed, which encountered errors, and the nature of any issues encountered. If Marqo finds the document, the document will be returned with the _found field set to true in an object. For documents not found, the _found field will be set to false, with the document ID returned in the _id field and details of the error in the message field.

For this endpoint, a 200 status code is not used to indicate successful document retrieval, as we aim to avoid adding extra fields to the returned documents. Here is the HTTP status code of the individual document responses (non-exhaustive list of status codes):

Status Code Description
400 Bad request. Returned for invalid input (e.g., invalid field types). Inspect message for details.
404 The target document is not in the index.
429 Marqo vector store receives too many requests. Please try again later.
500 Internal error.
Body
An array of IDs. Each ID is a string.


["article_152", "article_490", "article_985"]
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XGET http://localhost:8882/indexes/my-first-index/documents -H 'Content-Type: application/json' -d '
    ["article_152", "article_490", "article_985"]
'


Response 200 OK

{'results': [{'Blurb': 'A rocket car is a car powered by a rocket engine. This '
                       'treatise proposes that rocket cars are the inevitable '
                       'future of land-based transport.',
              'Title': 'Treatise on the viability of rocket cars',
              '_found': true,
              '_id': 'article_152'},
             {'_found': false, '_id': 'article_490'},
             {'Blurb': "One must maintain one's space suite. It is, after all, "
                       'the tool that will help you explore distant galaxies.',
              'Title': 'Your space suit and you',
              '_found': true,
              '_id': 'article_985'}]}
In this response, the index has no document with and ID of article_490. As a result, the _found field is false.

---


Mappings
The mappings object is a parameter (mappings) for an add_documents call. Mappings can be used for granular control over a field. Currently, it is only supported for the multimodal_combination and custom_vector field types.

When creating a structured index you define weights for a multimodal field under dependent fields. When adding documents mappings is optional with structured indexes and is only needed if the user needs to override default multimodal weights defined at index creation time.

Mappings is used to define custom_vector fields for unstructured indexes only. For structured indexes, do not include custom_vector fields in mappings. Instead, declare them as fields during index creation.

Mappings object
Multimodal Combination Mappings
Defining the mapping for multimodal_combination fields:


my_mappings = {
    "my_combination_field": {
        "type": "multimodal_combination",
        "weights": {"My_image": 0.5, "Some_text": 0.5},
    },
    "my_2nd_combination_field": {
        "type": "multimodal_combination",
        "weights": {"Title": -2.5, "Description": 0.3},
    },
}
Custom Vector Mappings
Defining the mapping for custom_vector fields (in an unstructured index):


my_mappings = {
    "my_custom_audio_vector_1": {"type": "custom_vector"},
    "my_custom_audio_vector_2": {"type": "custom_vector"},
}
Adding custom vector documents using that mapping object:

Unstructured Index

Marqo Open Source
Marqo Cloud

# Random vectors for example purposes. replace these with your own.
example_vector_1 = [i for i in range(512)]
example_vector_2 = [1 / (i + 1) for i in range(512)]

# Create the unstructured index
mq = marqo.Client("http://localhost:8882", api_key=None)
settings = {
    "type": "unstructured",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
}
mq.create_index("my-custom-vector-index", settings_dict=settings)

# Add the custom vectors
mq.index("my-custom-vector-index").add_documents(
    documents=[
        {
            "_id": "doc1",
            "my_custom_audio_vector_1": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_1,
                "content": "Singing audio file",
            },
        },
        {
            "_id": "doc2",
            "my_custom_audio_vector_2": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_2,
                "content": "Podcast audio file",
            },
        },
    ],
    tensor_fields=["my_custom_audio_vector_1", "my_custom_audio_vector_2"],
    mappings=my_mappings,
)

Structured Index

Marqo Open Source
Marqo Cloud

# Random vectors for example purposes. replace these with your own.
example_vector_1 = [i for i in range(512)]
example_vector_2 = [1 / (i + 1) for i in range(512)]

# Create the structured index
mq = marqo.Client("http://localhost:8882", api_key=None)
settings = {
    "type": "structured",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "allFields": [
        {"name": "my_custom_audio_vector_1", "type": "custom_vector"},
        {"name": "my_custom_audio_vector_2", "type": "custom_vector"},
    ],
    "tensorFields": ["my_custom_audio_vector_1", "my_custom_audio_vector_2"],
}
mq.create_index("my-structured-custom-vector-index", settings_dict=settings)

# Add the custom vectors
mq.index("my-structured-custom-vector-index").add_documents(
    documents=[
        {
            "_id": "doc1",
            "my_custom_audio_vector_1": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_1,
                "content": "Singing audio file",
            },
        },
        {
            "_id": "doc2",
            "my_custom_audio_vector_2": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_2,
                "content": "Podcast audio file",
            },
        },
    ]
)

---

Structured Index

Marqo Open Source
Marqo Cloud

cURL
Python

# For structured indexes, the custom vector field should be declared upon index creation (with type `custom_vector`).
# Create an index with the model that has the dimensions of your custom vectors. For example: "open_clip/ViT-B-32/laion2b_s34b_b79k" (dimension is 512). 
# Only the model dimension matters, as we are not vectorising anything when using custom vector fields.
# Space type CANNOT be 'prenormalized-angular' for custom vectors, as they are not normalized.

curl -X POST 'http://localhost:8882/indexes/my-first-structured-index' \
-H "Content-Type: application/json" \
-d '{
    "type": "structured",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "allFields": [
        {"name": "my_custom_vector", "type": "custom_vector"}
    ],
    "tensorFields": ["my_custom_vector"],
    "annParameters": {
        "spaceType": "angular",
        "parameters": {"efConstruction": 512, "m": 16}
    }
}'

# We add the custom vector documents into our structured index.
# We do NOT use mappings for custom vectors here.
curl -X POST 'http://localhost:8882/indexes/my-first-structured-index/documents' \
-H "Content-Type: application/json" \
-d '{
    "documents":[
        {
            "_id": "doc1",
            "my_custom_vector": {
                "vector": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511],
                "content": "Singing audio file"
            }
        },
        {
            "_id": "doc2",
            "my_custom_vector": {
                "vector": [1.0, 0.5, 0.3333, 0.25, 0.2, 0.1667, 0.1429, 0.125, 0.1111, 0.1, 0.0909, 0.0833, 0.0769, 0.0714, 0.0667, 0.0625, 0.0588, 0.0556, 0.0526, 0.05, 0.0476, 0.0455, 0.0435, 0.0417, 0.04, 0.0385, 0.037, 0.0357, 0.0345, 0.0333, 0.0323, 0.0312, 0.0303, 0.0294, 0.0286, 0.0278, 0.027, 0.0263, 0.0256, 0.025, 0.0244, 0.0238, 0.0233, 0.0227, 0.0222, 0.0217, 0.0213, 0.0208, 0.0204, 0.02, 0.0196, 0.0192, 0.0189, 0.0185, 0.0182, 0.0179, 0.0175, 0.0172, 0.0169, 0.0167, 0.0164, 0.0161, 0.0159, 0.0156, 0.0154, 0.0152, 0.0149, 0.0147, 0.0145, 0.0143, 0.0141, 0.0139, 0.0137, 0.0135, 0.0133, 0.0132, 0.013, 0.0128, 0.0127, 0.0125, 0.0123, 0.0122, 0.012, 0.0119, 0.0118, 0.0116, 0.0115, 0.0114, 0.0112, 0.0111, 0.011, 0.0109, 0.0108, 0.0106, 0.0105, 0.0104, 0.0103, 0.0102, 0.0101, 0.01, 0.0099, 0.0098, 0.0097, 0.0096, 0.0095, 0.0094, 0.0093, 0.0093, 0.0092, 0.0091, 0.009, 0.0089, 0.0088, 0.0088, 0.0087, 0.0086, 0.0085, 0.0085, 0.0084, 0.0083, 0.0083, 0.0082, 0.0081, 0.0081, 0.008, 0.0079, 0.0079, 0.0078, 0.0078, 0.0077, 0.0076, 0.0076, 0.0075, 0.0075, 0.0074, 0.0074, 0.0073, 0.0072, 0.0072, 0.0071, 0.0071, 0.007, 0.007, 0.0069, 0.0069, 0.0068, 0.0068, 0.0068, 0.0067, 0.0067, 0.0066, 0.0066, 0.0065, 0.0065, 0.0065, 0.0064, 0.0064, 0.0063, 0.0063, 0.0063, 0.0062, 0.0062, 0.0061, 0.0061, 0.0061, 0.006, 0.006, 0.006, 0.0059, 0.0059, 0.0058, 0.0058, 0.0058, 0.0057, 0.0057, 0.0057, 0.0056, 0.0056, 0.0056, 0.0056, 0.0055, 0.0055, 0.0055, 0.0054, 0.0054, 0.0054, 0.0053, 0.0053, 0.0053, 0.0053, 0.0052, 0.0052, 0.0052, 0.0052, 0.0051, 0.0051, 0.0051, 0.0051, 0.005, 0.005, 0.005, 0.005, 0.0049, 0.0049, 0.0049, 0.0049, 0.0048, 0.0048, 0.0048, 0.0048, 0.0047, 0.0047, 0.0047, 0.0047, 0.0047, 0.0046, 0.0046, 0.0046, 0.0046, 0.0045, 0.0045, 0.0045, 0.0045, 0.0045, 0.0044, 0.0044, 0.0044, 0.0044, 0.0044, 0.0043, 0.0043, 0.0043, 0.0043, 0.0043, 0.0043, 0.0042, 0.0042, 0.0042, 0.0042, 0.0042, 0.0041, 0.0041, 0.0041, 0.0041, 0.0041, 0.0041, 0.004, 0.004, 0.004, 0.004, 0.004, 0.004, 0.004, 0.0039, 0.0039, 0.0039, 0.0039, 0.0039, 0.0039, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0038, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0037, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0036, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0035, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.0031, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.003, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0029, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0027, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0024, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.0021, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002, 0.002],
                "content": "Podcast audio file"
            }
        }
    ]
}'


Note: Zero Magnitude Vector
When adding documents to a Marqo index, you may encounter an error related to a zero magnitude vector if normalizeEmbeddings is set to True during index creation and the custom vector provided for a document is a zero vector.

In such cases, Marqo will return the following message:

Error message: "Zero magnitude vector detected, cannot normalize"
HTTP status code: 400 Bad Request
This error occurs with the specific document that couldn't be added due to the zero vector. To avoid this error, ensure that non-zero vectors are provided as custom vectors when adding documents to the index.

Example: Map Score Modifiers
Structured Index

Marqo Open Source
Marqo Cloud

Python

mq = marqo.Client("http://localhost:8882", api_key=None)

settings = {
    "type": "structured",
    "vectorNumericType": "float",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
    "textPreprocessing": {
        "splitLength": 2,
        "splitOverlap": 0,
        "splitMethod": "sentence",
    },
    "imagePreprocessing": {"patchMethod": None},
    "allFields": [
        {"name": "text_field", "type": "text", "features": ["lexical_search"]},
        {
            "name": "map_score_mods",
            "type": "map<text, float>",
            "features": ["score_modifier"],
        },
        {
            "name": "map_score_mods_int",
            "type": "map<text, int>",
            "features": ["score_modifier"],
        },
    ],
    "tensorFields": ["text_field"],
    "annParameters": {
        "spaceType": "prenormalized-angular",
        "parameters": {"efConstruction": 512, "m": 16},
    },
}

mq.create_index("map-score-modifiers-index", settings_dict=settings)

docs = [
    {"_id": "1", "text_field": "a photo of a cat", "map_score_mods": {"a": 0.5}},
    {"_id": "2", "text_field": "a photo of a dog", "map_score_mods": {"b": 0.5}},
    {"_id": "3", "text_field": "a photo of a cat", "map_score_mods": {"c": 0.5}},
    {"_id": "4", "text_field": "a photo of a cat", "map_score_mods_int": {"a": 1}},
    {"_id": "5", "text_field": "a photo of a cat", "map_score_mods_int": {"b": 1}},
    {"_id": "6", "text_field": "a photo of a cat", "map_score_mods_int": {"c": 1}},
    {
        "_id": "7",
        "text_field": "a photo of a cat",
        "map_score_mods_int": {"c": 1},
        "map_score_mods": {"a": 0.5},
    },
]

res = mq.index("map-score-modifiers-index").add_documents(documents=docs)
print(res)


Unstructured Index

Marqo Open Source
Marqo Cloud

Python

mq = marqo.Client("http://localhost:8882", api_key=None)

mq.create_index("my-unstructured-index", model="open_clip/ViT-B-32/laion2b_s34b_b79k")
docs = [
    {"_id": "1", "text_field": "a photo of a cat", "map_score_mods": {"a": 0.5}},
    {"_id": "2", "text_field": "a photo of a dog", "map_score_mods": {"b": 0.5}},
    {"_id": "3", "text_field": "a photo of a cat", "map_score_mods": {"c": 0.5}},
    {"_id": "4", "text_field": "a photo of a cat", "map_score_mods_int": {"a": 1}},
    {"_id": "5", "text_field": "a photo of a cat", "map_score_mods_int": {"b": 1}},
    {"_id": "6", "text_field": "a photo of a cat", "map_score_mods_int": {"c": 1}},
    {
        "_id": "7",
        "text_field": "a photo of a cat",
        "map_score_mods_int": {"c": 1},
        "map_score_mods": {"a": 0.5},
    },
    {"_id": "8", "text_field": "a photo of a dog", "my_int": 2},
]

res = mq.index("my-unstructured-index").add_documents(
    documents=docs,
    tensor_fields=["text_field"],
)
print(res)


Media auth
Parameter: mediaDownloadHeaders

Expected value: An object that consists of key-value pair headers for image download. If set, Marqo will use this to authenticate the media download.

Default value: null

Example

mq.create_index(
    "my-first-index",
    treat_urls_and_pointers_as_images=True,
    model="open_clip/ViT-B-32/laion2b_s34b_b79k",
)
mq.index("my-first-index").add_documents(
    [
        {
            "img": "https://my-image-store.com/image_1.png",
            "title": "A lion roaming around...",
        },
        {
            "img": "https://my-image-store.com/image_2.png",
            "title": "Astronauts playing football",
        },
    ],
    media_download_headers={
        "my-image-store-api-key": "some-super-secret-image-store-key"
    },
    tensor_fields=["img", "title"],
)
Model auth
Parameter: modelAuth

Expected value: JSON object with either an s3 or an hf model store authorisation object.

Default value: null

The modelAuth object allows searching on indexes that use OpenCLIP and CLIP models from private Hugging Face and AWS S3 stores.

The modelAuth object contains either an s3 or an hf model store authorisation object. The model store authorisation object contains credentials needed to access the index's non publicly accessible model. See the example for details.

The index's settings must specify the non publicly accessible model's location in the setting's modelProperties object.

modelAuth is used to initially download the model. After downloading, Marqo caches the model so that it doesn't need to be redownloaded.

Example: AWS S3

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during add_documents to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_s3_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "oViT-B-32",
            "dimensions": 512,
            "model_location": {
                "s3": {
                    "Bucket": "<SOME BUCKET>",
                    "Key": "<KEY TO IDENTIFY MODEL>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# Specify the authorisation needed to access the private model during add_documents:
# We recommend setting up the credential's AWS user so that it has minimal 
# accesses needed to retrieve the model
mq.index("my-cool-index").add_documents(
    documents=[
        {'Title': 'The coolest moon walks'}
    ],
    model_auth={
        's3': {
            "aws_access_key_id": "<SOME ACCESS KEY ID>",
            "aws_secret_access_key": "<SOME SECRET ACCESS KEY>"
        }
    },
    tensor_fields=["Title"]
)
Example: Hugging Face (HF)

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during add_documents to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_hf_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B-32",
            "dimensions": 512,
            "model_location": {
                "hf": {
                    "repo_id": "<SOME HF REPO NAME>",
                    "filename": "<THE FILENAME TO DOWNLOAD>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# specify the authorisation needed to access the private model during add_documents:
mq.index("my-cool-index").add_documents(
    documents=[
        {'Title': 'The coolest moon walks'}
    ],
    tensor_fields=['Title'],
    model_auth={
        'hf': {
            "token": "<SOME HF TOKEN>",
        }
    }
)
Client batch size (Python client only)
Parameter: client_batch_size

Expected value: An Integer greater than 0 and less than or equal to 128.

Default value: None

A Python client only helper parameter that splits up very large lists of documents into batches of a more manageable size for Marqo. If very large documents are being indexed, it is recommended that this to be set lower. A client_batch_size=24 is a good place to start, and then adjust this for your use case as necessary.

Example

many_documents = [
    {"_id": f"doc_{i}", "Title": f"This is document number {i}"} for i in range(10000)
]
mq.index("my-first-index").add_documents(
    many_documents, client_batch_size=24, tensor_fields=["Title"]
)
Text Chunk Prefixes
Parameters: textChunkPrefix

Expected value: A string.

Default value: ""

This field overrides the text chunk prefix set during the index's creation.

If the user does not specify the textChunkPrefix field, the prefixes as defined in the index settings will be used. If these don't exist, the model's defaults will be used.

Note: Users do not need to provide textChunkPrefix for e5 models unless you want to override our default prefixes.

Example: Adding prefixes to the text document chunks and queries when embedding. Overrides index defaults

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/{index_name}/documents' \
-H 'Content-type:application/json' -d '
{
    "documents": [
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing the travels of Polo",
            "Genre": "History"
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection",
            "id": "article_591",
            "Genre": "Science"
        }
    ],
    "tensorFields": ["Description"],
    "textChunkPrefix": "override passage: "
}'


Adding Audio and Video documents
If you would like to index audio and/or video documents, you can do so by using the Languagebind models here. Make sure to set treatUrlsAndPointersAsMedia to true in the index settings for unstructured indexes. To run this example, please ensure that your system has at least 16GB of memory.

Unstructured Index Example

Marqo Open Source
Marqo Cloud

Python
cURL

mq.create_index(
    "my-index",
    treat_urls_and_pointers_as_media=True,
    model="LanguageBind/Video_V1.5_FT_Audio_FT_Image",
)

documents = [
    {
        "_id": "video",
        "video": "https://marqo-k400-video-test-dataset.s3.amazonaws.com/videos/---QUuC4vJs_000084_000094.mp4",
    },
    {
        "_id": "audio",
        "audio": "https://marqo-ecs-50-audio-test-dataset.s3.amazonaws.com/audios/4-145081-A-9.wav",
    },
    {
        "_id": "image",
        "image": "https://raw.githubusercontent.com/marqo-ai/marqo-api-tests/mainline/assets/ai_hippo_realistic.png",
    },
]

mq.index("my-index").add_documents(documents, tensor_fields=["video", "audio", "image"])


Structured Index Example

Marqo Open Source
Marqo Cloud

Python
cURL

settings = {
    "type": "structured",
    "vectorNumericType": "float",
    "model": "LanguageBind/Video_V1.5_FT_Audio_FT_Image",
    "normalizeEmbeddings": False,
    "tensorFields": ["video_field", "audio_field", "image_field", "multimodal_field"],
    "allFields": [
        {"name": "video_field", "type": "video_pointer"},
        {"name": "audio_field", "type": "audio_pointer"},
        {"name": "image_field", "type": "image_pointer"},
        {
            "name": "multimodal_field",
            "type": "multimodal_combination",
            "dependentFields": {
                "image_field": 0.3,
                "video_field": 0.4,
                "audio_field": 0.3,
            },
        },
    ],
}

mq.create_index("my-index", settings_dict=settings)

documents = [
    {
        "_id": "video",
        "video_field": "https://marqo-k400-video-test-dataset.s3.amazonaws.com/videos/---QUuC4vJs_000084_000094.mp4",
    },
    {
        "_id": "audio",
        "audio_field": "https://marqo-ecs-50-audio-test-dataset.s3.amazonaws.com/audios/4-145081-A-9.wav",
    },
    {
        "_id": "image",
        "image_field": "https://raw.githubusercontent.com/marqo-ai/marqo-api-tests/mainline/assets/ai_hippo_realistic.png",
    },
]

mq.index("my-index").add_documents(documents)

---

Search
Search for documents matching a specific query in the given index.


POST /indexes/{index_name}/search
Path parameters
Name  Type  Description
index_name  String  name of the requested index
Body
The body parameters below would be used for HTTP requests (if you were using cURL, for example). Python client users should use the pythonic snakecase equivalents (for example, searchable_attributes rather than searchableAttributes).

Search Parameter  Type  Default value Description
q String OR Dict  null  Query string, weighted query strings or custom vector object. Optional for tensor search if context parameter is used.
limit Integer 10  Maximum number of documents to be returned
offset  Integer 0 Number of documents to skip (used for pagination)
filter  String  null  Filter string in the Marqo DSL Language. In the Python client this parameter is called filter_string: mq.search("my query", filter_string="country:(United States)")
searchableAttributes  Array of strings  null  Attributes to be queried during the search
showHighlights
Boolean true  Return highlights for the document match. Only applicable for TENSOR search. With LEXICAL search, highlights will always be [].
searchMethod  String  "TENSOR"  The search method, can be LEXICAL, TENSOR or HYBRID.
hybridParameters  Dict  null  Parameters used for hybrid search.
attributesToRetrieve  Array of strings  null  Attributes to return in the search response
efSearch  Integer 2000  efSearch is the size of the dynamic list for the nearest neighbors (used during the search) - higher gives better recall at the cost of latency. Also efSearch must be greater than limit and limit is capped at 400
approximate Boolean True  Approximate toggles between exact KNN and approximate KNN (with HNSW)
reRanker  String  null  Method to use for reranking results
imageDownloadHeaders (deprecated) Dict  {}  Headers for the image download. Can be used to authenticate the images for download.
mediaDownloadHeaders  Dict  {}  Headers for the media download. Can be used to authenticate the media for download.
context Dict  null  Dictionary of "tensor":{List[{"vector": List[floats], "weight": (float)}]} to bring your own vectors into search.
scoreModifiers  Dict  null  A dictionary to modify the score based on field values. Check here for examples.
modelAuth Dict  null  Authorisation details used by Marqo to download non-publicly available models. Check here for examples.
textQueryPrefix String  null  The prefix added to text queries when embedding. This field overrides the textQueryPrefix set in the index settings during index creation. If it unset by the user, it defaults to the prefixes defined in the index settings. For more information on default values for index settings, see create_index.
rerankDepth Integer null  Number of hits to rerank with global score modifiers. If limit != rerankDepth, rerankDepth results are reranked but limit results are still returned. This value must be non-negative.
Note on Attributes to Retrieve per Query
It is beneficial to explicitly set the attributesToRetrieve parameter to limit the amount of data Marqo returns per document. Latency will increase as the number of attributes and documents retrieved increases. If you have documents with many fields that are not used by systems interfacing with Marqo's results, setting attributesToRetrieve to the minimal set of fields required can reduce latency and improve throughput.

Query parameters
Search Parameter  Type  Default value Description
device  String  null  The device used to search. If device is not specified and CUDA devices are available to Marqo (see here for more info), Marqo will speed up search by using an available CUDA device. Otherwise, the CPU will be used. Options include cpu and cuda, cuda1, cuda2 etc. The cuda option tells Marqo to use any available cuda devices.
telemetry Boolean False If true, the telemetry object is returned in the search response body. This includes information like latency metrics. This is set at client instantiation time in the Python client: mq = marqo.Client(return_telemetry=True)
Search result pagination
Use parameters limit and offset to paginate your results, meaning to query a certain number of results at a time instead of all at once.

The limit parameter sets the size of a page. If you set limit to 10, Marqo's response will contain a maximum of 10 search results. The offset parameter skips a number of search results. If you set offset to 20, Marqo's response will skip the first 20 search results.

Let's say you want each page to have 10 results, and you want to receive the 2nd page. Try setting limit and offset like so:


# Specify page properties
page_size = 10
page_num = 2

# Set limit and offset accordingly
limit = page_size
offset = (page_num - 1) * page_size
Pagination limitations

Search results can only be 10,000 results deep. This means limit + offset must be less than or equal to 10000. Also, efSearch must be greater than limit+offset.

Using pagination with search_method="TENSOR" may result in some results being skipped or duplicated (often near the edge of pages) within the first few pages if the page size is much smaller than the total search result count. Please keep this in mind when looking for particular results or when result order is essential.

Lexical search: exact matches
Use searchMethod="LEXICAL" to perform keyword search instead of tensor search. With lexical search, you can enable exact match searching using double quotes: "".

Any term enclosed in "" will be labeled a required term, which must exist in at least one field of every result hit. Note that terms enclosed in double quotes must also have a space between them and the terms before and after them, same as regular terms. Use this feature to filter your results to only documents containing certain terms. For example, if you want to search for results containing fruits, vegetables, or candy, but they must be green, you can construct your query as such:


Python


mq.index("my-first-index").search(
    q='fruit vegetable candy "green"',
    search_method="LEXICAL"
)
If you want to escape the double quotes (interpret them as text), use the escape key: backslash (\). For example:


curl -XPOST 'http://localhost:8882/indexes/my-first-index/search' -H 'Content-type:application/json' -d '
{
    "q": "Dwayne \"The Rock\" Johnson",
    "searchMethod": "LEXICAL"
}'
Note that to interpret backslashes as literal strings in Python, they must be escaped with another backslash, so the same query in the Python client would look like this:


mq.index("my-first-index").search(
    q='Dwayne \\"The Rock\\" Johnson',
    search_method="LEXICAL",
)
Note: syntax errors

If your use of "" does not follow proper syntax, Marqo will do its best interpretation of the quotes. Every 2 quotes (from left to right) will be paired and the text between them will be extracted as a required term if possible. If either of the quotes are badly formatted, both will be treated as whitespace and the text adjacent to them will be optional. Unpaired quotes will also be treated as whitespace. Here some examples of syntax errors:


# Quoted terms without spaces before/after
q = 'apples "oranges"bananas'
# Required terms: None
# Optional terms: apples, oranges, bananas

q = 'cucumbers "melons and watermelons""grapefruit"'
# Required terms: None
# Optional terms: cucumbers, melons, and, watermelons, grapefruit

# Unescaped quotes
q = 'There is a quote right"here'
# Required terms: None
# Optional terms: There, is, a, quote, right, here

# Unbalanced quotes
q = '"Dr. Seuss" "Thing 1" "Thing 2'
# Required terms: Dr. Seuss, Thing 1
# Optional terms: Thing, 2
Response
Name  Type  Description
hits  Array of objects  Results of the query
limit Integer Number of documents chunks specified in the query
offset  Integer Number of skipped results specified in the query
processingTimeMs  Number  Processing time of the query
query String  Query originating the response
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-index/search' -H 'Content-type:application/json' -d '
{
    "q": "what is the best outfit to wear on the moon?",
    "limit": 10,
    "offset": 0,
    "showHighlights": true,
    "searchMethod": "TENSOR",
    "attributesToRetrieve": ["Title", "Description"]
}'


Response: 200 Ok

{
  "hits": [
    {
      "Title": "Extravehicular Mobility Unit (EMU)",
      "Description": "The EMU is a spacesuit that provides environmental protection, mobility, life support, and communications for astronauts",
      "_highlights": [
        {
          "Description": "The EMU is a spacesuit that provides environmental protection, mobility, life support, and communications for astronauts"
        }
      ],
      "_id": "article_591",
      "_score": 1.2387788
    },
    {
      "Title": "The Travels of Marco Polo",
      "Description": "A 13th-century travelogue describing Polo's travels",
      "_highlights": [
        {
          "Title": "The Travels of Marco Polo"
        }
      ],
      "_id": "e00d1a8d-894c-41a1-8e3b-d8b2a8fce12a",
      "_score": 1.2047464
    }
  ],
  "limit": 10,
  "offset": 0,
  "processingTimeMs": 49,
  "query": "What is the best outfit to wear on the moon?"
}
Query (q)
Parameter: q

Expected value: Search string, a dictionary of weighted search strings. Optional for tensor search if context parameter is used.

Search strings are text or a url to an image, if the index has treatUrlsAndPointersAsImages set to True.

If queries are weighted, each weight act as a (possibly negative) multiplier for that query, relative to the other queries.

If your search method is TENSOR, this parameter is optional if you are using the context parameter. At least one of q or context must be specified for this search.

If you are using a custom vector you can also specify a dictionary of the form {'customVector': {'vector': [0.1,...,0], 'content': 'some string'}}.

Default value: null

Examples


# query string: 
q = "How do I keep my plant alive?"

# a dictionary of weighted query strings
q = {
    # a weighting of 1 gives this query a neutral effect:
    "Which dogs are the best pets": 1.0,
    # we give this a weighting of 2 because we really want results similar to this:
    "https://image_of_a_golden_retriever.png": 2.0,
    # we give this a negative weighting to make it less likely to appear: 
    "Poodle": -1
}

# providing a custom vector for tensor search
q = {
    "customVector" : {"vector": [0.1]*512}
}

# providing a custom vector and content for hybrid search
q = {
    # providing a custom vector and content if using hybrid search with a custom vector
    "customVector" : {"vector": [0.1]*512, "content": "some content that matches the vector"}
}
Limit
Parameter: limit

Expected value: Any positive integer

Default value: 10

Max: 1000

Sets the maximum number of documents returned by a single query.

Offset
Parameter: offset

Expected value: Any integer greater than or equal to 0

Default value: 0

Max: 10000

Sets the number of documents to skip. For example, if offset = 20, The first result returned will be the 21st result. Only set this parameter for single-field searches (multi-field support to follow).

Filter
Parameter: filter

Expected value: A filter string written in Marqo's query DSL.

Default value: null

Uses filter expressions to refine search results.

Read our guide on filtering, faceted search and filter expressions.

Example
You can write a filter expression in string syntax using logical connectives (see filtering in Marqo):


"(type:confectionary AND food:(ice cream)) OR animal:hippo"
Searchable attributes
Parameter: searchableAttributes

Expected value: An array strings

Default value: null

Configures which attributes will be searched for query matches.

If no value is specified, all fields will be searched.

Example
You can write the searchableAttributes as a list of strings, for example if you only wanted to search the "Description" field of your documents:


["Description"]
Reranker
Parameter: reRanker

Expected value: One of "owl/ViT-B/32", "owl/ViT-B/16", "owl/ViT-L/14"

Default value: null

Selects the method for reranking results. See the Models reference reranking section for more details.

If no value is specified, reRanker will be set to null and no reranking will occur.

Example
You can write reRanker as a string, for example:


"owl/ViT-B/32"
Context
Parameter: context

Expected value: Dictionary of "tensor":{List[{"vector": List[floats], "weight": (float)}]}

Default value: null

Context allows you to use your own vectors as context for your queries. Your vectors will be incorporated into the query using a weighted sum approach, allowing you to reduce the number of inference requests for duplicated content. The dimension of the provided vectors should be consistent with the index dimension.

Example

mq.index("my-first-index").search(
    q={"Chocolate chip cookies": 1},
    # the dimension of the vector (which is 768 here) should match the dimension of the index
    context={"tensor": [{"vector": [0.3, ] * 768, "weight": 2},  # custom vector 1
                        {"vector": [0.12, ] * 768, "weight": -1}, ]  # custom vector 2
             }
)
Score modifiers
Parameter: scoreModifiers

Expected value: An object with two optional keys: multiply_score_by and add_to_score. The value of each of these keys is an array of objects that each contain the name of a numeric field in the document as the field_name key and the weighting that should be applied to the numeric value, as the weight key, if it is found in the doc. If the score modifier field in the document is a map, access the subfield value using dot notation.

Default value: null

Score modifiers allows you to modify the initial score of the document by multiplying, and adding to, the initial search with values found within the document itself. This allows you to modify the search results based on metadata not included in the vectors.

The default weight value is 1 in the multiply_score_by object and 0 in the add_to_score object. The multiply_score_by modifiers will be applied to the document's score before the add_to_score modifiers. If a field specified in the score modification objects isn't found in the document, then the score modification will be skipped for that document's field.

For map score modifiers, avoid retrieving the score modifier fields in the query if they are not necessary for retrieval. For more information, see attributesToRetrieve.

There is negligible performance impact in performing queries with 1000 score modifiers against large dictionaries of upwards of 15,000 score modifiers per document.

Example

mq.index("my-first-index").add_documents(
    documents=[
        {
            "productImage": "https://my-images.com/cool-tshirt-1.png",
            "itemPopularity": 2.1,
            "negativeReviewCount": 4
        }],
    tensor_fields=['productImage']
)

mq.index("my-first-index").search(
    q="T-shirts with a cartoon character",
    score_modifiers={
        "multiply_score_by": [{"field_name": "itemPopularity", "weight": 1.8}],
        "add_to_score": [{"field_name": "negativeReviewCount", "weight": -0.1}]
    }
)

# if the initial score of the search query against this document is 0.67, then, after applying score modifiers,
# it will be modifed to 0.67 * (1.8 * 2.1) + (-0.1 * 4) = 2.13
Example Using Map Score Modifiers

import marqo
import json

mq = marqo.Client()

mq.create_index(index_name="map-modifiers-index",)

docs = [
    {"_id": "1", "text_field": "a photo of a cat", "map_score_mods": {"a": 0.5}},
    {"_id": "2", "text_field": "a photo of a dog", "map_score_mods": {"b": 0.5}},
    {"_id": "3", "text_field": "a photo of a cat", "map_score_mods": {"c": 0.5}},
    {"_id": "4", "text_field": "a photo of a cat", "map_score_mods_int": {"a": 1}},
    {"_id": "5", "text_field": "a photo of a cat", "map_score_mods_int": {"b": 1}},
    {"_id": "6", "text_field": "a photo of a cat", "map_score_mods_int": {"c": 1}},
    {"_id": "7", "text_field": "a photo of a cat", "map_score_mods_int": {"c": 1}, "map_score_mods": {"a": 0.5}},
    {"_id": "8", "text_field": "a photo of a dog", "my_int": 2},
]

res = mq.index("map-modifiers-index").add_documents(
    documents=docs, 
    tensor_fields=["text_field"],
)

# The same search syntax is used for both structured and unstructured indexes
res = mq.index("map-modifiers-index").search(
    q="",
    score_modifiers={
        "add_to_score": [{"field_name": "map_score_mods_int.c", "weight": 2}],
        "multiply_score_by": [{"field_name": "map_score_mods.a", "weight": 4}]
    },
    attributes_to_retrieve=["_id", "text_field"]
)
print(json.dumps(res, indent=2))
Global Score Modifiers
If the scoreModifiers parameter is used for HYBRID search, these will be treated as global score modifiers. This means that score modifiers will be applied after all ranking phases, and after fusion (if applicable). This global reranking only affects hits' _score value, and any initial scores like _tensor_score and _lexical_score will remain unchanged. This is also completely independent of the scoreModifiersTensor and scoreModifiersLexical parameters in hybridParameters, and they can be used in combination with each other.

The number of hits to be reranked by global score modifiers is determined by the rerankDepth parameter.

Global score modifiers are supported only for disjunction retrieval method and rrf ranking method. This is the order of operations for a HYBRID search with these parameters:

TENSOR search is done internally and reranked using scoreModifiersTensor (if any)
LEXICAL search is done internally and reranked using scoreModifiersLexical (if any)
Both result lists are fused using RRF algorithm.
The top rerankDepth results are reranked using scoreModifiers (global score modifiers).
The top limit results are returned.
Example of Hybrid Search with Global Score Modifiers

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-index/search' -H 'Content-type:application/json' -d '
{
    "q": "black shoes",
    "limit": 10,
    "rerankDepth": 5,
    "scoreModifiers": {
        "multiply_score_by": [{"field_name": "itemPopularity", "weight": 2}],
        "add_to_score": [{"field_name": "negativeReviewCount", "weight": -0.1}]
    },
    "searchMethod": "HYBRID",
    "hybridParameters": {
        "retrievalMethod": "disjunction",
        "rankingMethod": "rrf"
    }
}'

Model Auth
Parameter: modelAuth

Expected value: Dictionary with either an s3 or an hf model store authorisation object.

Default value: null

The ModelAuth object allows searching on indexes that use OpenCLIP and CLIP models from private Hugging Face and AWS S3 stores.

The modelAuth object contains either an s3 or an hf model store authorisation object. The model store authorisation object contains credentials needed to access the index's non publicly accessible model. See the example for details.

The index's settings must specify the non publicly accessible model's location in the setting's modelProperties object.

ModelAuth is used to initially download the model. After downloading, Marqo caches the model so that it doesn't need to be redownloaded.

Example: AWS s3

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during search to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_s3_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B/32",
            "dimensions": 512,
            "model_location": {
                "s3": {
                    "Bucket": "<SOME BUCKET>",
                    "Key": "<KEY TO IDENTIFY MODEL>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# Specify the authorisation needed to access the private model during search:
# We recommend setting up the credential's AWS user so that it has minimal 
# accesses needed to retrieve the model
mq.index("my-cool-index").search(
    q="Chocolate chip cookies",
    model_auth={
        's3': {
            "aws_access_key_id": "<SOME ACCESS KEY ID>",
            "aws_secret_access_key": "<SOME SECRET ACCESS KEY>"
        }
    }
)
Example: Hugging Face (HF)

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during search to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_hf_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B/32",
            "dimensions": 512,
            "model_location": {
                "hf": {
                    "repo_id": "<SOME HF REPO NAME>",
                    "filename": "<THE FILENAME TO DOWNLOAD>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# specify the authorisation needed to access the private model during search:
mq.index("my-cool-index").search(
    q="Chocolate chip cookies",
    model_auth={
        'hf': {
            "token": "<SOME HF TOKEN>",
        }
    }
)
Query Prefixes
Parameters: textQueryPrefix

Expected value: A string.

Default value: ""

This field overrides the text query prefix set during the index's creation.

Note: Users do not need to provide textQueryPrefix for e5 models unless you want to override our default prefixes.

Example: Adding prefixes to search queries. Overriding index defaults

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/{index_name}/search' \
-H 'Content-type:application/json' -d '
{
    "q": "Men shoes brown",
    "textQueryPrefix": "override query: "
}'


Rerank Depth
Parameters: rerankDepth

Expected value: An integer greater than or equal to 0.

Default value: null

The rerankDepth parameter specifies the number of hits to rerank with global score modifiers. If no value is specified, all possible hits will be reranked instead. If limit > rerankDepth, rerankDepth results are reranked then the remaining unranked limit - rerankDepth results (if any) are combined with the reranked result list before returning. If limit < rerankDepth, rerankDepth results are reranked then the result list is truncated to the top limit results.

This is currently only supported for HYBRID search method with disjunction retrieval method and rrf ranking method.

Hybrid parameters
Parameters: hybridParameters

Expected value: A Dictionary with parameters for hybrid search.

Default value: null

Hybrid parameter  Type  Default Description
retrievalMethod String  "disjunction" The method used for first stage retrieval. Can be "lexical" "tensor" or "disjunction" to use both lexical and tensor in the first stage.
rankingMethod String  "rrf" The method used for second stage retrieval. Can be "lexical" "tensor" or "rrf" for reciprocal rank fusion. You must use rrf if you specify disjunction for retrieval_method.
searchableAttributesLexical Array of strings  null  Attributes which are used for the lexical search.
searchableAttributesTensor  Array of strings  null  Attributes which are used for the tensor search.
scoreModifiersTensor  Dict  null  Score modifiers for tensor component of the query. Modifies the score based on field values. Check here for more details.
scoreModifiersLexical Dict  null  Score modifiers for lexical component of the query. Modifies the score based on field values. Check here for more details.
alpha Float 0.5 The linear weight of the tensor RRF score. A score of 1 would be 100% contribution from tensor component, and a score of 0 would be a 100% contribution from the lexical component.
rrfK  Integer 60  Smoothing factor for RRF. The higher rrfK, the lower the contribution of RRF to the ranking.
Example 1: Hybrid search with a structured index

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-structured-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "hf/e5-base-v2",
    "type": "structured",
    "allFields": [
        {"name": "title", "type": "text", "features": ["lexical_search"]},
        {"name": "description", "type": "text", "features": ["lexical_search", "filter"]},
        {"name": "time_added_epoch", "type": "float", "features": ["score_modifier"]}
    ],
    "tensorFields": ["title", "description"]
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-structured-index/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [ 
    {
        "title": "brown shoes",
        "description": "Mens brown shoes with laces",
        "time_added_epoch": 1421423142,
        "_id": "4231042142"
    }, 
    {
        "title": "red shirt",
        "description": "A red shirt with buttons",
        "time_added_epoch": 1421499942,
        "_id": "8988998589"
    }
]
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-structured-index/search' \
-H 'Content-type:application/json' -d '
{
    "q": "shirt that is red",
    "searchMethod": "HYBRID",
    "hybridParameters": {
        "retrievalMethod": "disjunction",                                                                                      
        "rankingMethod": "rrf",                                                                                                
        "alpha": 0.3,                                                                                                           
        "rrfK": 60,                                                                                                             
        "searchableAttributesLexical": ["description"],                                                                              
        "searchableAttributesTensor": ["description"],                                                                               
        "scoreModifiersTensor": { "add_to_score": [{"field_name": "epoch_timestamp", "weight": 0.01}] },  
        "scoreModifiersLexical": { "add_to_score": [{"field_name": "epoch_timestamp", "weight": 0.01}] }  
    }
}'


Example 2: Creating and searching an unstructured index, hybrid search with model deployed within Marqo

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "hf/e5-base-v2",
    "type": "unstructured"
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-index/documents' \
-H 'Content-type:application/json' -d '
{
    "documents": [ 
        {
            "title": "brown shoes",
            "description": "Mens brown shoes with laces",
            "time_added_epoch": 1421423142,
            "_id": "4231042142"
        }, 
        {
            "title": "red shirt",
            "description": "A red shirt with buttons",
            "time_added_epoch": 1421499942,
            "_id": "8988998589"
        }
    ],
    "tensorFields": ["title", "description"]
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-index/search' \
-H 'Content-type:application/json' -d '
{
    "q": "Men shoes brown",
    "searchMethod": "HYBRID",
    "hybridParameters": {
        "retrievalMethod": "disjunction",                                                                                      
        "rankingMethod": "rrf",                                                                                                
        "alpha": 0.3,                                                                                                           
        "rrfK": 10,                                                                                                              
        "scoreModifiersTensor": { "add_to_score": [{"field_name": "epoch_timestamp", "weight": 0.01}] },  
        "scoreModifiersLexical": { "add_to_score": [{"field_name": "epoch_timestamp", "weight": 0.01}] }  
    }
}'


Example 3: Creating a hybrid index with no model, hybrid search using custom vectors

Marqo Open Source
Marqo Cloud

cURL
Python

curl -X POST 'http://localhost:8882/indexes/my-hybrid-structured-index' \
-H "Content-Type: application/json" \
-d '{
    "model": "no_model",
    "modelProperties": {
        "type": "no_model",
        "dimensions": 3072
    },
    "type": "structured",
    "allFields": [
        {"name": "title", "type": "custom_vector", "features": ["lexical_search"]},
        {"name": "description", "type": "text", "features": ["lexical_search", "filter"]},
        {"name": "time_added_epoch", "type": "int", "features": ["score_modifier"]}
    ],
    "tensorFields": ["title"]
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-structured-index/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [ 
    {
        "title": {"vector": <replace with your custom 3072 dim vector>, "content": "brown shoes"},
        "description": "Mens brown shoes with laces",
        "time_added_epoch": 1421423142,
        "_id": "4231042142"
    }, 
    {
        "title": {"vector": <replace with your custom 3072 dim vector>, "content": "red shirt"},
        "description": "A red shirt with buttons",
        "time_added_epoch": 1421499942,
        "_id": "8988998589"
    }
]
}'

curl -XPOST 'http://localhost:8882/indexes/my-hybrid-structured-index/search' \
-H 'Content-type:application/json' -d '
{
    "q": {"customVector": {"vector": <replace with your custom 3072 dim vector>, "content": "Men shoes brown"}},
    "searchMethod": "HYBRID",
    "hybridParameters": {
        "retrievalMethod": "disjunction",                                                                                      
        "rankingMethod": "rrf",                                                                                                
        "alpha": 0.3,                                                                                                           
        "rrfK": 60,                                                                                                            
        "searchableAttributesLexical": ["title"],                                                                              
        "searchableAttributesTensor": ["title"],                                                                               
        "scoreModifiersTensor": { "add_to_score": [{"field_name": "time_added_epoch", "weight": 0.001}] },  
        "scoreModifiersLexical": { "add_to_score": [{"field_name": "time_added_epoch", "weight": 0.001}] }  
    }
}'

---

Embed
Vectorise a piece of content (string or weighted dictionary) or list of content and return the corresponding embeddings.


POST /indexes/{index_name}/embed
Path parameters
Name  Type  Description
index_name  String  name of the requested index
Body
The body parameters below would be used for HTTP requests (if you were using cURL, for example). Python client users should use the pythonic snakecase equivalents (for example, media_download_headers rather than mediaDownloadHeaders).

Search Parameter  Type  Default value Description
content String OR Dict OR List of (String OR Dict)  null  Content string to embed, weighted content strings (if Dict), or list of either of them. List or Dict cannot be empty.
imageDownloadHeaders(deprecated)  Dict  {}  Headers for the image download. Can be used to authenticate the images for download.
mediaDownloadHeaders  Dict  {}  Headers for the media download. Can be used to authenciate the all types of media for download.
modelAuth Dict  null  Authorisation details used by Marqo to download non-publicly available models. Check here for examples.
content_type  String  query The type of prefix to be added to all text content in the embed request. Can be either "query" (to use the default text query prefix), "document" (to use the default document text chunk prefix), or None (to have no prefix entirely). If the user would like a custom prefix, the prefix would have to be hardcoded into the document in the content field with content_type set to None.
Query parameters
Search Parameter  Type  Default value Description
device  String  null  The device used to embed. If device is not specified and CUDA devices are available to Marqo (see here for more info), Marqo will speed up embed by using an available CUDA device. Otherwise, the CPU will be used. Options include cpu and cuda, cuda1, cuda2 etc. The cuda option tells Marqo to use any available cuda devices.
telemetry Boolean False If true, the telemetry object is returned in the embed response body. This includes information like latency metrics. This is set at client instantiation time in the Python client: mq = marqo.Client(return_telemetry=True)
Response
Name  Type  Description
embeddings  Array of Array of floats  List of embeddings generated from the content (in the same order). If content was a single item, embeddings will contain a single element.
content String OR Dict OR List of (String OR Dict)  Content that was input for the request
processingTimeMs  Number  Processing time of the query
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-index/embed' \
 -H 'Content-type:application/json' -d '
{
    "content": [
        "Men shoes brown", 
        {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
    ],
    "content_type": null
}'


Response: 200 Ok

{
  "embeddings": [
    [-0.02587328478693962, -0.010530706495046616, 0.005109857302159071, ...],
    [0.013989399092603298, 0.015067596229836486, -0.02084693302954349, ...]
  ],
  "content": ["Men shoes brown", {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}],
  "processingTimeMs": 48
}
Content
Parameter: content

Expected value: Content string, a dictionary of weighted content strings, or a list of a combination of the previous types.

Content strings are text or a url to an image, if the index has treatUrlsAndPointersAsImages set to True.

If content is weighted, each weight acts as a (possibly negative) multiplier for that text, relative to the other text.

Default value: null

Examples


# content string: 
content = "How do I keep my plant alive?"

# a dictionary of weighted content strings
content = {
    # a weighting of 1 gives this query a neutral effect:
    "Which dogs are the best pets": 1.0,
    # we give this a weighting of 2 because we really want results similar to this:
    "https://image_of_a_golden_retriever.png": 2.0,
    # we give this a negative weighting to make it less likely to appear: 
    "Poodle": -1
}

# a list of content strings and weighted content strings
content = [
    "How do I keep my plant alive?",
    {
        "Which dogs are the best pets": 1.0,
        "https://image_of_a_golden_retriever.png": 2.0,
        "Poodle": -1
    }
]
Model Auth
Parameter: modelAuth

Expected value: Dictionary with either an s3 or an hf model store authorisation object.

Default value: null

The ModelAuth object allows embedding on indexes that use OpenCLIP and CLIP models from private Hugging Face and AWS S3 stores.

The modelAuth object contains either an s3 or an hf model store authorisation object. The model store authorisation object contains credentials needed to access the index's non publicly accessible model. See the example for details.

The index's settings must specify the non publicly accessible model's location in the setting's modelProperties object.

ModelAuth is used to initially download the model. After downloading, Marqo caches the model so that it doesn't need to be redownloaded.

Example: AWS s3

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_s3_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B-32",
            "dimensions": 512,
            "model_location": {
                "s3": {
                    "Bucket": "<SOME BUCKET>",
                    "Key": "<KEY TO IDENTIFY MODEL>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# Specify the authorisation needed to access the private model during embed:
# We recommend setting up the credential's AWS user so that it has minimal 
# accesses needed to retrieve the model
mq.index("my-cool-index").embed(
    content="Chocolate chip cookies",
    model_auth={
        's3': {
            "aws_access_key_id": "<SOME ACCESS KEY ID>",
            "aws_secret_access_key": "<SOME SECRET ACCESS KEY>"
        }
    }
)
Example: Hugging Face (HF)

# Create an index that specifies the non-public location of the model.
# Note the `auth_required` field in `modelProperties` which tells Marqo to use
# the modelAuth it finds during embedding process to download the model
mq.create_index(
    index_name="my-cool-index",
    settings_dict={
        "treatUrlsAndPointersAsImages": True,
        "model": 'my_hf_model',
        "normalizeEmbeddings": True,
        "modelProperties": {
            "name": "ViT-B-32",
            "dimensions": 512,
            "model_location": {
                "hf": {
                    "repo_id": "<SOME HF REPO NAME>",
                    "filename": "<THE FILENAME TO DOWNLOAD>",
                },
                "auth_required": True
            },
            "type": "open_clip",
        }
    }
)

# specify the authorisation needed to access the private model during embedding process:
mq.index("my-cool-index").embed(
    content="Chocolate chip cookies",
    model_auth={
        'hf': {
            "token": "<SOME HF TOKEN>",
        }
    }
)
Prefixes in Embed
Parameters: content_type

Expected value: "query", "document", or None

Default value: ""

This field instructs Marqo to use the query or text chunk prefixes defined in the index settings when embedding text content.

To use a custom prefix, users must hardcode the prefix into the content field and explicitly set content_type=None.

setting content_type="query" or content_type="document" will add the prefixes defined in the index settings or the model's defaults to the content. setting content_type to any other string will raise an error.

Example: Embedding without prefixes
Embedding with content_type = None does not add any prefixes to the content.


Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
    "content": [
        "Men shoes brown", 
        {"Large grey hat": 0.7, "https://marqo-assets.s3.amazonaws.com/tests/images/image1.jpg": 0.3}
    ],
    "content_type": null
}'


Example: Embedding with default prefix
When embedding without specifying a content type, Marqo uses the index setting's query prefixes.


Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/{index_name}/embed' \
-H 'Content-type:application/json' -d '
{
    "content": ["Dogs are great"]
}'

---

Recommend
Input a list of existing document IDs or dict of IDs and weights, and the response will be a list of "recommendations", which are documents similar to the input. These similar documents are retrieved by searching using interpolated vectors from the input. No inference is done during this process.


POST /indexes/{index_name}/recommend
Path parameters
Name  Type  Description
index_name  String  name of the requested index
Body
The body parameters below would be used for HTTP requests (if you were using cURL, for example). Python client users should use the pythonic snakecase equivalents (for example, searchable_attributes rather than searchableAttributes).

Search Parameter  Type  Default value Description
documents Array of strings or Dict  null  Document IDs to get recommendations for. This is either a list of IDs or a dictionary of ID to weight pairs.
tensorFields
Array of Strings  []  Tensor fields within the documents to use to generate recommendations
excludeInputDocuments
Boolean true  If true, input documents will never be returned in the results.
limit Integer 10  Maximum number of documents to be returned
offset  Integer 0 Number of documents to skip (used for pagination)
filter  String  null  Filter string in the Marqo DSL Language. In the Python client this parameter is called filter_string: mq.recommend(["doc1", "doc2"], filter_string="country:(United States)")
searchableAttributes  Array of strings  null  Attributes to be queried during the search.
showHighlights
Boolean true  Return highlights for the document match.
interpolationMethod String  null  The interpolation method to use for combining document embeddings. Defaults to slerp if normalizeEmbeddings=True for the index, and lerp otherwise.
attributesToRetrieve  Array of strings  null  Attributes to return in the response
efSearch  Integer 2000  The size of the dynamic list for the nearest neighbors (used during the search) - higher gives better recall at the cost of latency. Also efSearch must be greater than limit and limit is capped at 400
approximate Boolean True  Toggles between exact KNN and approximate KNN (with HNSW)
reRanker  String  null  Method to use for reranking results
scoreModifiers  Dict  null  A dictionary to modify the score based on field values. Check here for examples.
Response
Name  Type  Description
hits  Array of objects  Results of the query
limit Integer Number of documents chunks specified in the query
offset  Integer Number of skipped results specified in the query
processingTimeMs  Number  Processing time of the query
query String  Query originating the response
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XPOST 'http://localhost:8882/indexes/my-first-index/recommend' -H 'Content-type:application/json' -d '
{
    "documents": ["doc1", "doc5"],
    "limit": 10,
    "offset": 0,
    "showHighlights": true,
    "attributesToRetrieve": ["Title", "Description"]
}'


Response: 200 Ok

{
    "hits": [
        {
            "Description": "A small breed of dog.",
            "Title": "Chihuahuas",
            "_highlights": [{"Description": "A small breed of dog."}],
            "_id": "doc4",
            "_score": 0.8582207950725244
        },
        {
            "Description": "A certain breed of dog.",
            "Title": "Huskies",
            "_highlights": [{"Description": "A certain breed of dog."}],
            "_id": "doc3",
            "_score": 0.8579511935990104
        },
        {
            "Description": "Our favorite feline friends.",
            "Title": "Cats",
            "_highlights": [{"Title": "Cats"}],
            "_id": "doc2",
            "_score": 0.8182416336023639
        }
    ],
    "limit": 10,
    "offset": 0,
    "processingTimeMs": 48,
    "query": null
}
Documents
Parameter: documents

Expected value: List of document IDs, a dictionary of weighted document IDs.

These are the document IDs that you want to get recommendations for. Their embeddings will be interpolated and used to search for similar documents in the index.

If document IDs are weighted, each weight acts as a (possibly negative) multiplier for that document's embeddings, relative to the other documents.

When providing weights, there must be at least one document with a non-zero weight. Document weights may have to meet other criteria, depending on the interpolation method used. See the Interpolation method section for more details.

Default value: null

Examples


# document list
documents = ["doc1", "doc2"]

# a dictionary of weighted documents
documents = {
    # a weighting of 1 gives this query a neutral effect:
    "doc1": 1.0,
    # we give this a weighting of 2 because we really want results similar to this:
    "doc2": 2.0,
    # we give this a negative weighting to make it less likely to appear: 
    "doc3": -1
}
Tensor fields
Parameter: tensorFields

Expected value: An array of strings

Default value: null

Configures which tensor fields will be used to generate recommendations.

If no value is specified, all tensor fields will be used.

Limit
Parameter: limit

Expected value: Any positive integer

Default value: 10

Max: 1000

Sets the maximum number of documents returned by a single query.

Offset
Parameter: offset

Expected value: Any integer greater than or equal to 0

Default value: 0

Max: 10000

Sets the number of documents to skip. For example, if offset = 20, The first result returned will be the 21st result. Only set this parameter for single-field searches (multi-field support to follow).

Filter
Parameter: filter

Expected value: A filter string written in Marqo's query DSL.

Default value: null

Uses filter expressions to refine search results.

Read our guide on filtering, faceted search and filter expressions.

Example
You can write a filter expression in string syntax using logical connectives (see filtering in Marqo):


"(type:confectionary AND food:(ice cream)) OR animal:hippo"
Searchable attributes
Parameter: searchableAttributes

Expected value: An array strings

Default value: null

Configures which attributes will be searched for query matches.

If no value is specified, all tensor fields will be searched.

Example
You can write the searchableAttributes as a list of strings, for example if you only wanted to search the "Description" field of your documents:


["Description"]
Interpolation method
Parameter: interpolationMethod

Expected value: One of "slerp", "lerp" or nlerp

Default value: null

Sets the method used to interpolate the document embeddings.

If no value is specified, the interpolation method will be set to slerp if normalizeEmbeddings=True for the index, and lerp otherwise.

SLERP
SLERP (Spherical Linear Interpolation) is a method of interpolating between two vectors on a hypersphere. It is suitable for interpolating between embeddings that are normalized. SLERP cannot interpolate between embeddings where the sum of the weights is equal to zero. Consequently, it is best to avoid consecutive pairs of weights that add up to zero in your input document weights, as such pairs can result in a 400 error.

LERP
LERP (Linear Interpolation) is a method of interpolating between two vectors in a linear fashion. LERP cannot interpolate between embeddings where the sum of all weights is equal to zero. A zero weight sum will result in a 400 error.

NLERP
NLERP (Normalized Linear Interpolation) is similar to LERP, but the interpolated vector is normalized. It can be used as an alternative to SLERP where normalizeEmbeddings=True for the index. As with LERP, a zero weight sum will result in a 400 error.

Reranker
Parameter: reRanker

Expected value: One of "owl/ViT-B/32", "owl/ViT-B/16", "owl/ViT-L/14"

Default value: null

Selects the method for reranking results. See the Models reference reranking section for more details.

If no value is specified, reRanker will be set to null and no reranking will occur.

Example
You can write reRanker as a string, for example:


"owl/ViT-B/32"
Score modifiers
Parameter: score_modifiers

Expected value: An object with two optional keys: multiply_score_by and add_to_score. The value of each of these keys is an array of objects that each contain the name of a numeric field in the document as the field_name key and the weighting that should be applied to the numeric value, as the weight key, if it is found in the doc.

Default value: null

Score modifiers allows you to modify the initial score of the document by multiplying, and adding to, the initial search with values found within the document itself. This allows you to modify the search results based on metadata not included in the vectors

The default weight value is 1 in the multiply_score_by object and 0 in the add_to_score object. The multiply_score_by modifiers will be applied to the document's score before the add_to_score modifiers. If a field specified in the score modification objects isn't found in the document, then the score modification will be skipped for that document's field.

---

Stats
Stats give information about indexes, including the number of documents and vectors in the index. Note that the number of vectors is not necessarily the same as the number of documents, as a document can have multiple vectors or zero vectors.

Please also be aware that the number of documents and the number of vectors may not be immediately consistent during indexing processes. However, these values will be eventually consistent with each other after indexing processes have finished.

Stats Object

{
  "numberOfDocuments": 4,
  "numberOfVectors": 4,
  "backend": {
    "memoryUsedPercentage": 0.73484113083,
    "storageUsedPercentage": 37.01321365493
  }
}
Name  Type  Description
numberOfDocuments Integer number of documents in the index
numberOfVectors Integer number of vectors in the index
Get index stats
Get statistics and information about an index


GET /indexes/{index_name}/stats
Path parameters
Name  Type  Description
index_name  String  name of the index
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XGET http://localhost:8882/indexes/my-first-index/stats


Response: 200

{
  "numberOfDocuments": 4,
  "numberOfVectors": 4,
  "backend": {
    "memoryUsedPercentage": 0.73484113083,
    "storageUsedPercentage": 37.01321365493
  }
}

---

Settings
Get settings of an index. For a conceptual overview of index settings, refer to our Index API Reference.

Get index settings

GET /indexes/{index_name}/settings
Path parameters
Name  Type  Description
index_name  String  name of the index
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XGET http://localhost:8882/indexes/my-first-index/settings


Response: 200

{
  "annParameters": {
    "parameters": {
      "efConstruction": 512,
      "m": 16
    },
    "spaceType": "prenormalized-angular"
  },
  "filterStringMaxLength": 20,
  "imagePreprocessing": {},
  "model": "hf/e5-base-v2",
  "normalizeEmbeddings": true,
  "textPreprocessing": {
    "splitLength": 2,
    "splitMethod": "sentence",
    "splitOverlap": 0
  },
  "treatUrlsAndPointersAsImages": false,
  "type": "unstructured",
  "vectorNumericType": "float"
}

---

Health
The indexes/{index_name}/health endpoint provides information about the health of a Marqo index.

Get health


GET /indexes/{index_name}/health
Gets the health of a Marqo index.

Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XGET http://localhost:8882/indexes/my-first-index/health


Response (Healthy Instance): 200 OK

{
  "status": "green",
  "inference": {
    "status": "green"
  },
  "backend": {
    "status": "green",
    "memoryIsAvailable": true,
    "storageIsAvailable": true
  }
}
Response (Full Storage): 200 OK

{
  "status": "yellow",
  "inference": {
    "status": "green"
  },
  "backend": {
    "status": "yellow",
    "memoryIsAvailable": false,
    "storageIsAvailable": false
  }
}

---

Get Models
This returns information about all the loaded models in cuda and cpu devices.

The /models endpoint provides necessary information relating to models loaded in your devices (cpu or cuda). You can check the currently loaded models, and eject a loaded model to free memory.


GET /models
Example

Marqo Open Source
Marqo Cloud

cURL
Python

curl  -XGET http://localhost:8882/models

Response: 200 OK

{"models": [
    {"model_name": "hf/e5-base-v2", "model_device": "cpu"},
    {"model_name": "hf/e5-base-v2", "model_device": "cuda"},
    {"model_name": "open_clip/ViT-B-32/laion2b_s34b_b79k", "model_device": "cpu"},
    {"model_name": "open_clip/ViT-B-32/laion2b_s34b_b79k", "model_device": "cuda"},
    {"model_name": "ViT-B/16", "model_device": "cpu"}]}


---

Get CUDA Information
This gives information about your cuda usage.


GET /device/cuda
Example
CUDA

Marqo Open Source
Marqo Cloud

cURL
Python

curl -XGET http://localhost:8882/device/cuda


Response: 200 OK

{
   "cuda_devices": [
     {
       "device_id": 0,
       "device_name": "Tesla T4",
       "memory_used": "1.7 GiB",
       "total_memory": "14.6 GiB",
       "utilization": "11.0 %",
       "memory_used_percent": "25.0 %"
     }
   ]
}

---


MARQO COOKBOOK BELOW:

Welcome to the Marqo Cookbook!
The Marqo Cookbook is a collection of recipes, best practices, and tips for using Marqo to build vector search backed applications.

The cookbook is designed to be a practical guide to help you get the most out of Marqo and answer common questions. The guides contain a mix of Marqo specific information as well as concepts and techniques that are generally applicable.

All guide expect that you have Marqo running on localhost using the default port of 8882 and have the Python client installed.

The client can be installed using pip:


pip install marqo
Marqo can be run using Docker:


docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
Table of Contents
Model Selection:
Text Search
Multimodal Search
Best Practices:
Score Modifier Best Practices
Multimodal Combination Field Best Practices
Text Pre-Processing Best Practices
Query Prompt Engineering
Multi-term Queries
Analysing Processing Time
Recipes:
Lexical Reciprocal Rank Fusion
Similar Item Recommendations
Diversifying Recommendations
Semantic Filtering Templates
Personalised Search and Recommendations
Embedding Static Context
Calculating Recall
Content Moderation
Duplicate Detection
Loading Sentence Transformers from Hugging Face
Index Types:
Unstructured Vs Structured Indexes
Creating an Unstructured Index:
Text Only Unstructured Index
Multimodal Unstructured Index
Filtering in Unstructured Indexes
Creating a Structured Index:
Text Only Structured Index
Multimodal Structured Index
Structured Index with Other Features
Understanding HNSW Parameters

---


Model Selection for Text Search
Marqo supports models which can be utilised via the sentence_transformers API. There is a large list of compatible models, we provide a number of good choices out of the box, but there is also flexibility to load other models as well.

In this guide we will provide some recommendations for selecting a model for text search.

Recommended Text Retrieval Models
For most use cases we recommend models from the E5 family as they benchmark consistently well across a range of tasks and are available in three sizes with an English only and multilingual version.

English only Models
The V2 E5 models are available in three sizes:

hf/e5-small-v2 (384 dimensional embeddings)
hf/e5-base-v2 (768 dimensional embeddings)
hf/e5-large-v2 (1024 dimensional embeddings)
The models benchmark better on retrieval tasks as the size increases, but the larger models are slower to encode text. The larger embedding dimensions also require more memory to store and process.

If you are using a CPU then the small or base models are recommended. If you are using a GPU then the base or large model can be used.

Multilingual Models
The multilingual E5 models are available in three sizes:

hf/multilingual-e5-small (384 dimensional embeddings)
hf/multilingual-e5-base (768 dimensional embeddings)
hf/multilingual-e5-large (1024 dimensional embeddings)
As with the English only models, the larger models benchmark better on retrieval tasks but are slower to encode text and require more memory for inference and within the index. The multilingual models are all larger than the English only models despite their equivalent embedding dimensions.

If you are using a CPU then the small or base models are recommended. If you are using a GPU then the base or large model can be used.

Model Prefixing and Best Practices
Some models are trained with specific prefixes for different tasks. Some notable examples are:

The E5 model family which expect queries to be prefixed with query: and documents to be prefixed with passage: for asymmetric retrieval.
E5 instruct models which expect queries to be prefixed with Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:
The BGE model family which expects the prefix Represent this sentence for searching relevant passages:.
Fortunately Marqo handles this for you, all models included in Marqo are automatically set up with the correct prefixing for asymmetric retrieval tasks. Asymmetric retrieval tasks are where you have a query and a document and you want to find the most relevant document for the query, whereas symmetric retrieval tasks are where you have a document and you intend to find similar documents.

If you need to override the prefixing you can do this when searching or adding documents like so:


# override search prefix
mq.index("my-first-index").search(
    "cats and dogs", text_query_prefix="my custom prefix:"
)

# override add documents text chunk prefix
mq.index("my-first-index").add_documents(
    documents, text_chunk_prefix="my custom prefix:"
)

---

Model Selection for Multimodal Search
Marqo supports a huge range of opensource models, deciding which one to use can be difficult and often getting the absolute best results for a specific dataset will require some experimentation. However, there are some models which tend to be the best choice for certain scenarios and we will cover these here.

When using multimodal CLIP models be sure to set treatUrlsAndPointersAsImages to true in the index settings. This will ensure that URLs and pointers are treated as images and not text.

For example:


import marqo

mq = marqo.Client()

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
}

mq.create_index("my-first-multimodal-index", settings_dict=settings)
OpenCLIP Model Naming Convention
OpenCLIP models are named with a loose convention to them that can help you understand what they are. Here is a breakdown of the naming convention.

For a model such as open_clip/ViT-L-14/laion2b_s32b_b82k:

ViT: Vision Transformer (the image tower architecture)
L: Large (the size of the model). Sizes are B (base), L (large), H (huge), g (gigantic)
14: 14x14 pixel patches for the vision tower. Other common values are 16 and 32
laion2b: the dataset used (this one is 2 billion image text pairs)
sXXb: the number of samples seen. For 2 billion examples in laion2b the s32b would mean 16 epochs
bXXk: the global batch size using in training, in this case 82 thousand
Some models such as open_clip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k also specify the text tower architecture in the name (xlm-roberta-base in this case).

I want a balanced model
For a good balanced of speed and relevancy we recommend open_clip/ViT-L-14/laion2b_s32b_b82k as a good starting point. This model uses a 224x224px image with 14x14px patches and has strong image and text understanding from the diverse laion2b dataset. This model is best used with a GPU however it can be used on a CPU.

If compute is more constrained or latency is more critical then the smaller open_clip/ViT-B-16/laion2b_s34b_b88k is a good choice. This model is very close in performance however it uses a smaller architecture and slightly larger patches.

I want the best image understanding
As a general rule, models with smaller patches and/or larger input images will tend to exhibit better image understand; image understanding is also heavily influenced by data and training though so this is not a hard and fast rule.

The following models are strong choices for image understanding:

Vision Transformer Models
The vision towers for these models are ViT architectures.

open_clip/ViT-L-14/laion2b_s32b_b82k: A large model with 14x14px patches and a 224x224px image size.
open_clip/ViT-H-14/laion2b_s32b_b79k: A huge model with 14x14px patches and a 224x224px image size. (GPU strongly recommended)
open_clip/ViT-g-14/laion2b_s34b_b88k: A gigantic model with 14x14px patches and a 224x224px image size. (GPU strongly recommended)
ConvNeXT Models
The vision towers for these models are ConvNeXT architectures.

open_clip/convnext_base_w_320/laion_aesthetic_s13b_b82k_augreg: A base model with 320x320px images.
open_clip/convnext_large_d_320/laion2b_s29b_b131k_ft_soup: A large model with 320x320px images. (GPU strongly recommended)
ResNet Models
The vision towers for these models are ResNet architectures. While they are often not as strong in image-text search tasks as other models, these models can be quite good at image-image search tasks.

open_clip/RN50x64/openai: A ResNet model with 448x448px images. This models is quite large so a GPU is recommended.
I want the best text understanding
In use cases where the text is more important than the image it is important to select a model with a strong text tower. Typically models that use models pretrained independant of the image tower are good choice.

XLM-RoBERTa text towers are pretrained on a large amount of multilingual data and have strong text understanding. These models are significantly better at multilingual search than english only models however the dedicated multilingual CLIP models perform better.

open_clip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k: This is a pretrained base XLM-RoBERTa which was then further trained alongside an untrained ViT-B-32 vision tower.
open_clip/xlm-roberta-large-ViT-H-14/frozen_laion5b_s13b_b90k: This is a pretrained large XLM-RoBERTa which was then further trained alongside a frozen ViT-H-14 vision tower from open_clip/ViT-H-14/laion2b_s32b_b79k.
I want fast inference
For fast inference times smaller models with larger patches are typically a better choice.

open_clip/ViT-B-32/laion2b_s34b_b79k: A base model with 32x32px patches and a 224x224px image size.
I want multilingual image-text search
Marqo also supports a selection of multilingual CLIP models.

multilingual-clip/XLM-Roberta-Large-Vit-L-14: A large XLM-RoBERTa text tower trained alongside a large ViT-L-14 vision tower. This models is quite large so ensure you have sufficient RAM/VRAM (>6GB).

---


Score Modifier Best Practices
Score modifiers are a way to influence the ranking of search results. Score modifiers let you adjust the score of each document in the result set based on the value of a scalar field in the document (int, float, double, long).

Typical use cases include:

Boosting documents based on some measure of "goodness" (e.g. popularity, previous sales, aesthetic score).
Reducing duplication in search results for user generated content (e.g. users copy each other but copies score worse that originals).
Score Modifiers Under the Hood
Score modifiers are only applied to your limit results. This means that if you have a limit of 10 then the score modifiers will only be applied to the 10 results (changing their ranking).

This is important to know because changing your limit later on can have a significant impact on the ranking of your results.

Using Score Modifiers
Marqo provides two modifications: "multiply_score_by" and "add_to_score". For almost all use cases we recommend using "add_to_score" as it has much more intuitive behavior.

Score modifiers as specified in search requests can be be different for each request:


mq.index("my-first-index").search(
    q="T-shirts with a cartoon character",
    score_modifiers={
        "multiply_score_by": [{"field_name": "itemPopularity", "weight": 1.8}],
        "add_to_score": [{"field_name": "sellerScore", "weight": 0.01}],
    },
)
Best Practices with "add_to_score"
While variables in your documents which are used for score modifiers can be of any magnitude it is recommended to use values in the 0-1 range for simplicity.

For "add_to_score" the final score for a document (_score in the response) is calculated as:


_score = _score + (field_value * weight)
Where _score is the distance between the query vector and the document vector and field_value is the value of the field used for score modification. For a document like:


{
    "_id": "document1",
    "sellerScore": 0.7,
    "title": "this is a t-shirt"
}
With a search request like:


mq.index("my-first-index").search(
    q="T-shirts with a cartoon character",
    score_modifiers={"add_to_score": [{"field_name": "sellerScore", "weight": 0.01}]},
)
Would result in a score of 0.7 * 0.01 being added to the score of the document.

Deciding on an amount to add
The amount to add ("weight") will largerly depend on your data, the range of values in the field and your desired effect. Typically lower weights such as 0.01 are a sensible starting point as this does not overpower the original ranking entirely.

Best Practices with "multiply_score_by"
For "multiply_score_by" the final score for a document (_score in the response) is calculated as:


_score = _score * (field_value * weight)
Where _score is the distance between the query vector and the document vector and field_value is the value of the field used for score modification. For a document like:


{
    "_id": "document1",
    "itemPopularity": 0.7,
    "title": "this is a t-shirt"
}
With a search request like:


mq.index("my-first-index").search(
    q="T-shirts with a cartoon character",
    score_modifiers={
        "multiply_score_by": [{"field_name": "itemPopularity", "weight": 1.2}]
    },
)
The score of the document would be multiplied by 0.7 * 1.2. This might make sense in some use cases but can have unintuitive effects on the ranking of results if your data is not well understood.

Because documents with no value in the field will have this treated as a value of 0 this will mean that all these documents receive a score of 0 which can be undesirable.

Multiply score modifiers for positive weight and field values are not strictly increasing as this depends on which side of 1 the field value multiplied by the weight falls. This can make it harder to reason about the effect of the score modifier.

That being said, "multiply_score_by" can work well in cases where your score modifier values are greater than 1 as for prenormalized-angular vector spaces the original score is in the range [0, 1] and so this will weight the modifier value by the similarity of the document to the query.

Score Modifiers in Scaled Deployments
If you are running Marqo with a scaled deployment where there are multiple shards for the index then the score modifiers will be applied to the results from each shard. This means that if you set limit=10 and have 3 shards then the score modifiers will be applied to all 30 results before the top 10 are selected from all shards.

This means that the impact of score modifiers can be very different for deployments with many shards compared to deployments with fewer shards. Keep this in mind when comparing smaller dev/test/staging configurations with larger production configurations.

---


Lexical Reciprocal Rank Fusion
In some applications, lexical search can be a better approach to certain queries despite its known deficiencies on others. Typical cases include queries with measurements (e.g. Rolex with 35mm face) or exact model names (e.g. iPhone 12 Pro Max 256GB).

To get the best of both lexical and tensor search, the results from both can be combined using a technique called Reciprocal Rank Fusion (RRF). This technique combines the results from both lexical and tensor search and boosts the results that are common to both.

How does Reciprocal Rank Fusion work?
Reciprocal Rank Fusion (RRF) is a technique that combines the results from two or more search methods by assigning a score to each result based on its rank in the result list. The score is calculated as the reciprocal of the rank of the result in the list. The scores from all the result lists are then combined to produce a final list of results. The results are then sorted based on the combined scores.

The score of a document in RRF if calculated as 1 / (k + rank), where k is a constant and rank is the position of the document in the result list. The constant k is used to control the importance of the rank in the final score. A higher value of k gives more importance to the rank, while a lower value gives more importance to the document itself. Many implementations of RRF use a value of k = 60 as a default.

Recipe for Lexical Reciprocal Rank Fusion with Structured Indexes
For structured indexes Marqo supports hybrid search out of the box through the search API and can be used as follows:


results = mq.index("my-structured-index").search(
    q="Rolex with 35mm face",
    search_method="HYBRID",
    limit=100,
    hybrid_parameters={
        "retrievalMethod": "disjunction",
        "rankingMethod": "rrf",
        "alpha": 0.5,
        "rrfK": 60,
    },
)
In this example we set the following parameters:

retrievalMethod: disjunction retrieves both lexical and tensor results
rankingMethod: rrf combines the results using reciprocal rank fusion
alpha: 0.5 weights the lexical and tensor results (0 for pure lexical, 1 for pure tensor)
rrfK: 60 the constant k in the RRF formula, default is 60
Recipe for Lexical Reciprocal Rank Fusion with Unstructured Indexes
The following implementation is a generic implementation of RRF for unstructured Marqo indexes. Native hybrid search will be coming to unstructured indexes in a future release.

You may wish to modify the search requests in marqo_rrf_search to use different score modifiers or searchable attributes.


import marqo
from typing import List, Dict


def reciprocal_rank_fusion(
    list1: List[str], list2: List[str], k: int = 60
) -> List[str]:
    rrf_scores = {}

    def update_scores(result_list: List[str], scores_dict: Dict[str, float]) -> None:
        for rank, item in enumerate(result_list, start=1):
            if item not in scores_dict:
                scores_dict[item] = 0
            scores_dict[item] += 1 / (k + rank)

    update_scores(list1, rrf_scores)
    update_scores(list2, rrf_scores)

    sorted_items = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)

    sorted_result_ids = [item[0] for item in sorted_items]

    max_length = max(len(list1), len(list2))
    return sorted_result_ids[:max_length]


def marqo_rrf_search(
    mq: marqo.Client,
    index_name: str,
    query: str,
    limit: int,
):
    lexical_results = mq.index(index_name).search(
        q=query, limit=limit, search_method="LEXICAL"
    )
    vector_results = mq.index(index_name).search(q=query, limit=limit)
    lexical_ids = [r["_id"] for r in lexical_results["hits"]]
    vector_ids = [r["_id"] for r in vector_results["hits"]]

    search_data = {}
    for hit in lexical_results["hits"] + vector_results["hits"]:
        search_data[hit["_id"]] = hit

    rrf_results = reciprocal_rank_fusion(lexical_ids, vector_ids)

    rrf_hits = []
    for _id in rrf_results:
        rrf_hits.append(search_data[_id])

    return {"hits": rrf_hits}


---


Similar Item Recommendations
Search and recommendations are very tightly coupled, especially for vector search. This means that Marqo is also a powerful recommendations tool. In this recipe we will show you how to use Marqo to generate similar item recommendations.

How Similar Item Recommendations work in Marqo
Similar item recommendations are effectively a "reverse document search". Instead of searching with a query, the goal is to search using a document that is already in the index.

Marqo comes with an endpoint for recommendations that allows you to generate similar item recommendations based on a selection of one or more existing documents. Documents used as the query can be represented as a list of document IDs or as a dictionary with the document ID as the key and a weight as the value. There are two included methods for combining the vectors of the documents: slerp and lerp.

slerp is a spherical linear interpolation between the vectors of the documents.
lerp is a linear interpolation between the vectors of the documents.
Marqo will select this automatically based off the index settings, if you are using normalized embeddings then slerp will be used, otherwise lerp will be used.

Recipe for Similar Item Recommendations

import marqo

recommendations = mq.index("my-first-index").recommend(
    documents=["doc1", "doc5"], limit=10, interpolation_method="slerp"
)

---

Semantic Filtering Templates
Semantic filtering is a way to provide a filtering effect on a set of images without the need for any metadata to filter on.

It is important to remember how models are trained. CLIP models are trained on captioned images, this means that the best search results typically come from queries that resemble a caption. For example a search for an image of a dog will typically return better results than dog. A great example of this can be seen in how prompting is done for zero-shot classification with CLIP models in this notebook

This concept can be taken a step further and we can use semantic filtering to provide a filtering effect with just images.

Semantic Filtering is Like Prompt Engineering
Prompt engineering has become popular with the rise of Large Language Models (LLMs). Semantic filtering is a very similar process whereby you aim to take some task like searching for a dog and tune the query to give more specific or better behaviour, like searching for a dog in a cartoon style.

Recipe for Semantic Filtering Templates
Below we provide a number of templates which can be used in a variety of domains. To use these, simply replace the <QUERY> with your search term. For example, a search for dog with the template "An image of a <QUERY> set in a neon-lit, futuristic cyberpunk world" would become "An image of a dog set in a neon-lit, futuristic cyberpunk world".

Stock Image / Generic Image Semantic Filtering Templates

{
    "photorealistic": "A high-resolution, lifelike stock photo of a <QUERY>",
    "stylized": "An image of a <QUERY> rendered in a unique, stylized manner",
    "lowpoly": "A 3D low polygon angular render of a <QUERY>",
    "artistic": "An image resembling a masterpiece, portraying a <QUERY>",
    "cartoon": "An image in the style of Saturday morning cartoons featuring a <QUERY>",
    "abstract": "An image inspired by modern art, abstractly representing a <QUERY>",
    "pixelart": "An 8-bit or 16-bit pixel art representation of a <QUERY>",
    "vector": "A clean, scalable vector graphic of a <QUERY>",
    "minimalistic": "A photo emphasizing simplicity and minimalism, showing a <QUERY>",
    "grayscale": "A black and white, monochromatic image of a <QUERY>",
    "vintage": "An image of a <QUERY> with a retro or vintage aesthetic",
    "cyberpunk": "An image of a <QUERY> set in a neon-lit, futuristic cyberpunk world",
    "steampunk": "An image blending Victorian and industrial elements, depicting a <QUERY>",
    "realism": "A hyper-realistic, detailed image of a <QUERY>",
    "impressionist": "An image mimicking the impressionist art movement, showing a <QUERY>",
    "surreal": "A surreal, dream-like image of a <QUERY>",
    "sketch": "A hand-drawn sketch or pencil drawing of a <QUERY>",
    "popart": "An image in the style of pop art, featuring a <QUERY>",
    "watercolor": "A watercolor painting-like image of a <QUERY>",
    "oilpaint": "An oil painting-like image, rich in texture, of a <QUERY>",
    "glitch": "A glitch-art styled, distorted image of a <QUERY>",
    "neon": "An image of a <QUERY> with vibrant, neon colors",
    "noir": "A film-noir inspired, high-contrast image of a <QUERY>",
    "fantasy": "A fantastical, otherworldly image of a <QUERY>",
    "comic": "An image resembling a comic book panel, featuring a <QUERY>",
    "isometric": "An isometrically projected image of a <QUERY>",
    "outline": "An outline line art depiction of a <QUERY>",
}
E-commerce Semantic Filtering Templates

{
    "bright": "A brightly lit, high-resolution image of a <QUERY>",
    "bohemian": "An image of a <QUERY> with a bohemian style, emphasizing free-spirited aesthetics",
    "colorful": "An image of a <QUERY> featuring a wide range of colors, appealing to diverse tastes",
    "formal": "An elegantly presented image of a <QUERY>, suitable for formal contexts",
    "dynamic": "A dynamic, action-oriented photo of a <QUERY>, emphasizing its use in real-life situations",
    "lifestyle": "A lifestyle image of a <QUERY>, showing it in a realistic setting or being used",
    "60s": "An image of a <QUERY> with 1960s style, showcasing mid-century modern aesthetics",
    "retro": "A retro-inspired image of a <QUERY>, with a nod to past decades",
    "closeup": "A close-up, detailed image of a <QUERY>, highlighting textures and quality",
    "minimalist": "An image of a minimalist <QUERY>, focusing on clean lines and simplicity",
    "vibrant": "A photo of a <QUERY> with saturated, vibrant colors to catch the customer's eye",
    "textured": "A detailed image showing the texture of a <QUERY>, giving a sense of material",
}
Fashion Semantic Filtering Templates

{
    "streetwear": "An image showcasing a <QUERY> in a streetwear style, perfect for urban fashion",
    "high_fashion": "A high-fashion, editorial image of a <QUERY>, emphasizing luxury and trend-setting designs",
    "comfy": "A cozy, comfortable-looking image of a <QUERY>, ideal for casual wear",
    "summer_fashion": "A summer-inspired image of a <QUERY>, light and perfect for warm weather",
    "winter_fashion": "A winter-themed image of a <QUERY>, highlighting warmth and layering",
    "vintage_fashion": "A retro-styled image of a <QUERY>, with a vintage or nostalgic flair",
    "runway": "A runway image of a <QUERY>, capturing the essence of high fashion shows",
    "bohemian": "An image of a <QUERY> with a bohemian style, emphasizing free-spirited aesthetics",
    "sporty": "A sporty, athletic image of a <QUERY>, suitable for active lifestyles",
    "elegant": "An elegantly styled image of a <QUERY>, perfect for formal occasions",
    "casual": "A casual, everyday image of a <QUERY>, for a relaxed look",
    "avant_garde": "An avant-garde, boundary-pushing image of a <QUERY>, for cutting-edge styles",
}
Furniture and Interior Design Semantic Filtering Templates

{
    "modern": "A modern style image of a <QUERY>, showcasing contemporary design trends",
    "traditional": "A traditional style image of a <QUERY>, with classic details and timeless appeal",
    "minimalist": "A minimalist image of a <QUERY>, emphasizing clean lines and simplicity in design",
    "scandinavian": "A Scandinavian style image of a <QUERY>, highlighting functionality, simplicity, and craftsmanship",
    "industrial": "An industrial style image of a <QUERY>, with a focus on raw materials and sleek lines",
    "boho": "A bohemian (boho) style image of a <QUERY>, rich in patterns, colors, and textures",
    "rustic": "A rustic style image of a <QUERY>, emphasizing natural beauty and ruggedness",
    "luxury": "A luxury image of a <QUERY>, depicting opulence and high-end design",
    "eclectic": "An eclectic style image of a <QUERY>, mixing various styles and periods for a unique look",
    "art_deco": "An Art Deco style image of a <QUERY>, featuring bold geometric patterns and luxurious materials",
}

---


Personalised Search and Recommendations
Search and recommendations are largerly similar tasks. Vector search enables real time personalisation of both by injecting additional query terms into the search.

A note on implementation: Typically the desired effect (a slight tailoring of results) is best achieved when the additional context you inject is diverse with regards to your data. For example if you have a broad range of documents (like Amazon) then you want to inject a range of items that the user has shown interest in rather than only t-shirts. Conversly if all you have is t-shirts then diversity can come in the form of patterns and styles. Without diversity search and recommendations may steer away from the relevant results and towards to known priors.

Personalisation with Query Terms
Where additional text or images are known, these can use used to condition any subsequent search or recommendation. This is useful for personalisation based on user history, profiles, images, free text context, or other unstuctured sources of contextual information.

We aim to build a multi-term query with contextual information contributing a low enough weight to not overpower the original query. We will us the weighted multi-term queries support in marqo, some more info on best practices for these can be found here.

A simple example of code to do this is as follows:


import marqo

mq = marqo.Client()

personalisation_context = [
    "I love bold patterns and bright colours",  # free text about a user
    "It's the 60s! Pattern living room rug",  # a product title they interacted with
    "https://example.com/image.jpg",  # a product image they interacted with
    # etc
]

# something sufficiently low to not overpower the original query
# will most likely require tuning
total_personalisation_weight = 0.2

# the actual query for what the user wants
query = "t-shirt"


def personalise_query(query, personalisation_context, total_personalisation_weight):
    personalisation_weight = total_personalisation_weight / len(personalisation_context)

    composed_query = {}

    for context in personalisation_context:
        composed_query[context] = personalisation_weight

    composed_query[query] = 1.0

    return composed_query


mq.index("my-first-index").search(
    q=personalise_query(query, personalisation_context, total_personalisation_weight)
)
Personalisation with Existing Documents
The same principle can be applied to personalisation with existing documents. This is useful for personalisation based on user interaction history, current checkout cart, or historical purchases.

The key difference is that we will pull the vectors out of Marqo and use them to condition the search or recommendation.


import marqo
from typing import List, Dict

mq = marqo.Client()

personalisation_context = [
    "document1",  # a document id
    "document2",  # another document id
    # etc
]

# something sufficiently low to not overpower the original query
# will most likely require tuning
total_personalisation_weight = 0.2

# the actual query for what the user wants
query = "t-shirt"


def get_personalisation_vectors(
    mq: marqo.Client,
    index_name: str,
    personalisation_context: List[str],
    total_personalisation_weight: float,
) -> Dict[str, List[Dict[str, float]]]:
    personalisation_weight = total_personalisation_weight / len(personalisation_context)

    items_with_facets = mq.index(index_name).get_documents(
        document_ids=personalisation_context, expose_facets=True
    )

    vecs = []

    for item_with_facets in items_with_facets["results"]:
        try:
            for facet in item_with_facets["_tensor_facets"]:
                vecs.append(facet["_embedding"])
        except KeyError:
            pass

    context_tensor = []

    for vec in vecs:
        context_tensor.append({"vector": vec, "weight": personalisation_weight})

    return {"tensor": context_tensor}


mq.index("my-first-index").search(
    q={query: 1.0},
    context=get_personalisation_vectors(
        mq, "my-first-index", personalisation_context, total_personalisation_weight
    ),
)

---

Embedding Static Context
Some use cases require frequent re-use of embeddings in queries. For example terms that influence style or quality as outlined in the multi-term queries section. In these cases it can be beneficial to pre-compute the embeddings. This allows for faster retrieval and avoids the need to recompute the embeddings for each query.

Using the Embed Endpoint and Context
The embed endpoint can be used to generate embeddings from any content and get the vectors back from Marqo. This can be used to generate embeddings for static context that can be used in queries.


import marqo

mq = marqo.Client()

# Generate an embedding for a static context
context_embeddings = mq.index("my-first-index").embed(
    ["Low quality, bad, jpeg artifacts"]
)
The context_embeddings can then be used in queries to influence the results. For example, to search for items that are not low quality:


results = mq.index("my-first-index").search(
    q={"t-shirt": 1.0},
    context={
        "tensor": [{"vector": context_embeddings["embeddings"][0], "weight": -0.5}]
    },
)
Doing this means that inference is only done once for the context vector, resulting in faster search.

Implementation Notes
In practice you will most likely want to cache these vectors in a separate persistent store, fast KV databases with disk storage for presistance are a good choice. Other use cases for this pattern includes applications like embedding a taxonomy of style descriptions for re-use in future searches.

---


Calculating Recall
Marqo provides configurable parameters for the underlying HNSW Approximate Nearest Neighbours (ANN) search algorithm. These parameters can be tuned to balance recall, latency, and memory usage. The key parameters are efConstruction, m, and efSearch. More details on these can be found in the Understanding HNSW Parameters section.

While the defaults provided are able to provide >99% recall on average for most datasets, advanced users or people with use cases that are incredibly sensitive to recall may wish to tune these parameters to achieve higher recall.

Recall in this context is the proportion of items returned by the approximate search that are returned by an exact search. For example, if the result sets of the approximate and exact searches are A and B respectively, then recall is calculated as |A ∩ B| / |B|.

Recipe for Calculating Recall with Marqo
Marqo allows you to toggle between approximate and exact search. This allows you to calculate the recall for any index you have in Marqo.

The following function which takes an instance of the client, an index name, a limit, and a list of queries, can be used to calculate the recall for a given index:


import marqo
from typing import List


def calculate_average_recall(
    mq: marqo.Client, index_name: str, limit: int, queries: List[str]
) -> float:
    recalls = []
    for query in queries:
        approximate_results = mq.index(index_name).search(q=query, limit=limit)
        exact_results = mq.index(index_name).search(
            q=query, limit=limit, approximate=False
        )
        approximate_ids = [result["_id"] for result in approximate_results["hits"]]  # A
        exact_ids = [result["_id"] for result in exact_results["hits"]]  # B
        intersection = set(approximate_ids).intersection(exact_ids)  # A ∩ B
        recall = len(intersection) / len(exact_ids)  # |A ∩ B| / |B|
        recalls.append(recall)

    return sum(recalls) / len(recalls)  # average recall over the set of queries
Example usage:


import marqo

mq = marqo.Client()

index_name = "my-first-index"
limit = 10
queries = ["apple", "banana", "cherry"]

average_recall = calculate_average_recall(mq, index_name, limit, queries)
print(f"Average Recall@{limit}: {average_recall}")

---


Loading Sentence Transformers from Hugging Face
Loading other Models
Other models can be loaded from Hugging Face into Marqo, there are just a few things to check first. For this example we will use BAAI/bge-small-en-v1.5.

Checking API Compatibility
Once you have found a model you would like to try, check that it is compatible with the sentence_transformers API. You can do this by checking the model's page on Hugging Face and looking for the "Use in libraries" or "Use in sentence-transformers" button above the downloads graph. If sentence-transformers appears then the model should work in Marqo.

Checking Model Details
To load a model into Marqo you will need to know its embedding dimension and max tokens. This can be found by navigating to the "Files and versions" tab and opening the config.json file.

The embedding dimension is the value of hidden_size and the max tokens is the value of max_position_embeddings. Below is an example of the config for BAAI/bge-small-en-v1.5:


{
  "_name_or_path": "/root/.cache/torch/sentence_transformers/BAAI_bge-small-en/",
  "architectures": [
    "BertModel"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 384,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "intermediate_size": 1536,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "torch_dtype": "float32",
  "transformers_version": "4.30.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}
Loading the Model
To use the model in Marqo you can use the modelProperties object in the index settings with the model name from Hugging Face and the other details we extracted from the config.json. e.g.


import marqo

settings = {
    "modelProperties": {
        "name": "BAAI/bge-small-en-v1.5",
        "type": "hf",
        "dimensions": 384,
        "tokens": 512,
    },
    "model": "bge-small-en-v1.5",
}

mq = marqo.Client()

mq.create_index("my-first-index", settings_dict=settings)

# trigger the model download
mq.index("my-first-index").search("query: apple")

---


Unstructured Vs Structured Indexes
Marqo lets you utilise both structured and unstructured indexes. While much of the functionality is shared between these two index types there are some key differences which influence the decision of which to use.

Definition
New Unstructured Index: In Marqo 2.13.0, we introduced a significant improvement for unstructured indexes to support searchable attributes for both tensor and lexical search. New tensor and lexical fields are dynamically added to the index settings during indexing. Therefore, unstructured indexes created with Marqo 2.13.0 and later behave similarly to structured indexes in terms of search functionality.
Legacy Unstructured Index: Unstructured indexes created prior to Marqo 2.13.0 remain unchanged and do not support searchable attributes.
Structured Index: When creating a structured index, you must define all the fields. The field definition, including name, type, and features, is immutable throughout the lifecycle of the structured index.
Key Differences
Searchable Attributes: Both structured indexes and new unstructured indexes allow you to specify which attributes to search at query time. Legacy unstructured indexes cannot do this and search all attributes by default.
HNSW Behaviour: Structured indexes and new unstructured indexes place each tensor field in its own HNSW graph, whereas legacy unstructured indexes use a single HNSW graph for all tensor fields. This distinction means that searchable attributes offer additional performance benefits and enable targeted searches within specific fields.
Lexical Search: Structured indexes allow you to specify which fields are available for lexical search, while unstructured indexes treat all text fields as lexical search fields.
Mutability: Structured indexes have a fixed schema that cannot be changed once created. The schema must be a superset of the fields in each document, but a document does not have to contain all the fields defined in the schema. In contrast, unstructured indexes can have fields added at any time.
Partial Updates: Partial updates are support for both however they are significantly faster for structured indexes. Partial updates for unstructured indexes are identical to adding the document with useExistingTensors set to true.
Filtering: Structured indexes allow you to specify which fields are filterable. Unstructured indexes will automatically make fields filterable, if a field contains text then filterStringMaxLength will be used to determine if it is filterable using the length of the string.
Performance: Structured indexes are faster in general, the largest performance difference is that structured indexes will consume less memory space. Partial updates to document metadata is also significantly faster for structured indexes.
Error Handling: Structured indexes will throw an error if you try to add a document with a field that is not in the schema. Unstructured indexes will add the field to the schema and continue. The strictness of structured indexes can help catch errors early.
When to Use Unstructured Indexes
Unstructured indexes are recommended in the following situations:

Getting Started: If you are new to Marqo and want to get started quickly, unstructured indexes are the best choice due to their ease of use.
Dynamic Schema: If you have a dynamic schema where fields are added frequently, unstructured indexes are the best choice.
When to Use Structured Indexes
Structured indexes are recommended in the following situations:

Performance: If you require the best maximum performance, structured indexes are the best choice. Expecially for large indexes with continuous updates to documents.
Production/Enterprise: If you are using Marqo in a production or enterprise environment, structured indexes are often a better choice due to the strictness of the schema and the ability to catch malformed documents early.
Advanced Usage: If you require better control over searchable attributes, lexical search, and other features, then structured indexes are the best choice.

---

Text Only Unstructured Indexes
For documentation on structured indexes see here.

In this section we will go over how to create a simple unstructured index with a text field to vectorise. This is the simplest form of an index.

Creating a very basic text only unstructured index is as simple as the following:


import marqo

mq = marqo.Client(url="http://localhost:8882")

mq.create_index("my-simple-unstructured-index")
Example Add Documents Usage
Because we are using an unstructured index we need to specify the tensor fields during indexing.


documents = [
    {
        "_id": "1",
        "text_field": "New York",
    },
    {
        "_id": "2",
        "text_field": "Los Angeles",
    },
]

mq.index("my-simple-unstructured-index").add_documents(
    documents, tensor_fields=["text_field"]
)
Example search usage
This index will allow us to search the text_field field using the LEXICAL search method and the TENSOR search method.

Tensor Search (search_method="TENSOR" is the default):


results = mq.index("my-simple-unstructured-index").search(q="New York")
Lexical Search:


results = mq.index("my-simple-unstructured-index").search(
    q="New York", search_method="LEXICAL"
)

---

Multimodal Unstructured Indexes
For documentation on unstructured indexes see here.

Unstructured indexes have the multimodal combination fields defined during indexing. Multimodal combination field mappings can be different for each document though for most use cases it is best to keep them consistent.

In this section we will go over how to create a simple unstructured index with a multimodal field to vectorise.

Minimal example of creating an untructured multimodal index
For Marqo to know that it needs to identify image URLs and download the images we need to set the treatUrlsAndPointersAsImages setting to True. Marqo will automatically detect image URLs, fetch the image, and vectorise it with the CLIP model.


import marqo

settings = {
    "model": "open_clip/ViT-L-14/laion2b_s32b_b82k",
    "treatUrlsAndPointersAsImages": True,
}

mq = marqo.Client(url="http://localhost:8882")

mq.create_index("my-mm-unstructured-index", settings_dict=settings)
Index Settings: model
The model field is used to specify the model to use for the vectorisation. To do multimodal search this must be a CLIP model.

Example Add Documents Usage
Because we are using an unstructured index we need to specify the tensor fields or the multimodal mappings in add documents.


documents = [
    {
        "_id": "1",
        "text_field": "New York",
        "image_field": "https://example.com/image.jpg",
    },
    {
        "_id": "2",
        "text_field": "Los Angeles",
        "image_field": "https://example.com/image2.jpg",
    },
]

mq.index("my-mm-unstructured-index").add_documents(
    documents,
    mappings={
        "multimodal_field": {
            "type": "multimodal_combination",
            "weights": {"text_field": 0.1, "image_field": 0.9},
        }
    },
    tensor_fields=["multimodal_field"],
)
Example Search Usage
This index will allow us to search the multimodal_field with the TENSOR search method and the text_field field using the LEXICAL search method.

Tensor Search (search_method="TENSOR" is the default):


results = mq.index("my-mm-unstructured-index").search(q="New York")
Lexical Search:


results = mq.index("my-mm-unstructured-index").search(
    q="New York", search_method="LEXICAL"
)

---

Duplicate Detection
Vector search excels at duplicate detection, especially with multimodal indexes where copies of images may have very slight variations due to compression, resizing, or other minor transformations.

Finding Duplicates
The simplest way to find duplicates is to search with the vector of a known document. This will return the most similar documents in the index. Documents with a similarity score above a sufficiently high threshold can be considered duplicates. As this is a symmetric retrieval task the threshold can be set quite high; ultimately it requires some experimentation to find the best threshold for your data and model, however a good starting point is 0.99 for strong duplicates.


import marqo

mq = marqo.Client()

index_name = "my-first-index"

base_document_id = "document1"

item_with_facets = mq.index(index_name).get_document(
    document_id=base_document_id, expose_facets=True
)

# NOTE: we assume on tensor field in this example with no text chunking (multimodal index)
# adjust accordingly for your data (_tensor_facets is a list with an entry for each tensor field / text chunk)
vec = item_with_facets["_tensor_facets"][0]["_embedding"]

results = mq.index(index_name).search(
    q=None,  # no query, just context vectors
    context={"tensor": [{"vector": vec, "weight": 1.0}]},
)

---

Query Prompt Engineering
Queries with multimodal models can be engineered into prompts to help guide search behaviour. This can be used to control result style, supress low quality content, or to implement semantic filtering.

The goal of query prompt engineering is to engineer a generic modification to a query that will steer search results in a desired direction. This works best with CLIP models as longer caption style prompts bear similarities to the data on which they are trained. This is similar to the techniques used to repurpose CLIP for zero shot classification, an example of prompt templates can be seen in this notebook.

Typically query are best modified either with natural language prefixes or by adding tags to the query. The goal is to make the query resemble a caption that might be found in the training data of the multimodal model. Those familiar with text to image models like Stable Diffusion will be familiar with the comma separated tag style of prompts, Stable Diffusion uses CLIP as a text encoder for the embeddings with condition the generation.

Controlling Style
Modifying queries in your backend implementation can help tailor results to a specific style without sacrificing relevance. It is important to think back to how these models were trained, with captioned images. The models have an understanding of various styles, quality descriptions, brands, and other caption information that can be leveraged to control the style of search results.

Prefixing Examples
Prefixes work well as they resemble image captions.

An image of <QUERY> - A generic prompt popular in zero-shot tasks
A high resolution photo of <QUERY> - For high quality images
A stock image of <QUERY> - Stock image style to the results
A colorful vibrant image of <QUERY> - For colorful images
An Amazon product image of <QUERY> - Higher scores to images that look like ones you would find on Amazon
Tag Style Examples
Comma separated tag style modifications are also effective.

<QUERY>, high resolution, high quality
<QUERY>, colorful, vibrant
<QUERY>, stock image
<QUERY>, e-commerce listing, Amazon product image
Supressing Low Quality Content
There are two approaches to supressing low quality content, one is to modify the query to steer the model towards high quality content, the other is to add an additional weighted term to the query that will penalize low quality content.

Query Modification
This is similar to the style control example above but with a focus on quality. Examples might include:

A high resolution photo of <QUERY> - For high quality images
A high quality image of <QUERY> - Stock image style to the results
A professional photo of <QUERY> - For professional quality images
Penalizing Low Quality Content
Marqo supports multi-term queries with weighted terms. This can be used to inject an additional query term which penalises low quality content. Examples might include:

{"<QUERY>": 1.0, "low resolution, blurry, jpeg artifacts": -0.4} - Penalizes low quality images
{"<QUERY>": 1.0, "NSFW, nudity": -0.4} - Penalizes NSFW content
Semantic Filtering
For detailed examples and more information we provide a selection of useful templates in the Semantic Filtering Recipe.

An example use case where semantic filtering is powerful is in stock image search, it can be effectively applied in almost any domain though. If you have an index of stock images it can be powerful to filter on styles and aesthetics, things that typically do not have metadata. A template for outline art styles might look like "An artwork of a <QUERY> in a clean black and white outline style". Where <QUERY> is the search term entered by an end user. Semantic Filtering can also be done with tag style prompts such as <QUERY>, outline, black and white, clean.

Recommended UI for Semantic Filtering
Typically the user interface is implemented as a dropdown selector which makes it appear to a use as if it were a traditional filter. The dropdown would contain the various styles or aesthetics that can be applied to the search query. When a user selects an option the query is modified without the user knowing what is happening behind the scenes.

---

Multi-term Queries
Marqo supports weighted multi-term queries. This can be used to inject additional semantics into the query vectors to influence the results, uses include advanced search, personalisation, and quality control.

Multi-term queries are specified as a dictionary where the query term is the key and the weight is the value. The weight can be any floating point number, positive or negative. Vectors for each term are combined as a weighted average using the weights provided, as such a negative weight move search away from that term and a positive term moves towards it.


mq.index("my-first-index").search(
    {
        "red t-shirt": 1.0,
        "short sleeve": 0.3,
        "buttons": -0.4,
        "low resolution, blurry, jpeg artifacts": -0.2,
    }
)
Advanced Search Queries
Our demo website has an implementation of weighted multi-term queries via the "more of" and "less of" input fields. For example you could search "shirt" which will return results with some business shirts, but if you enter "business" into the "less of" field it will remove business shirts from the results.

Under the hood the query construction is implemented as follows (actual implementation here):


def compose_query(
    query: str,
    more_of: str,
    less_of: str,
) -> Dict[str, float]:
    composed_query = {}

    if more_of:
        more_term = query + ", " + more_of  # this trick is explained below
        composed_query[more_term] = 0.75

    if less_of:
        composed_query[less_of] = -1.1

    if query:
        composed_query[query] = 1.0

    if not composed_query:
        return {"": 1}

    return composed_query
This means that a search for "shirt" with more of "buttons" and less of "business" would be transformed into:


{
    "shirt": 1.0,
    "shirt, buttons": 0.75,
    "business": -1.1
}
Prepended Query Term Trick: In this implementation we prepend the query term to the "more of" term. This trick allows for a higher weight on the "more of" term without risking the "more of" term derailing the search and taking over the intended primary search keyword.

Personalisation
Where you have additional information about a user this can be incorporated with low weights into a weighted query to personalise the search results in real-time. For example a users current shopping cart, interaction history, or favourited item list could be used to source text and images to influence search.

A modified version of the advanced search query implementation could be used to incorporate personalisation through favourited items:


def compose_query(
    query: str, more_of: str, less_of: str, favourites: List[str]
) -> Dict[str, float]:
    composed_query = {}

    if more_of:
        more_term = query + ", " + more_of  # this trick is explained below
        composed_query[more_term] = 0.75

    if less_of:
        composed_query[less_of] = -1.1

    if query:
        composed_query[query] = 1.0

    total_fav_weight = 0.2  # we will equally distribute the weight of the favourites
    # favourites could be text or image urls
    for favourite in favourites:
        composed_query[favourite] = total_fav_weight / len(favourites)

    if not composed_query:
        return {"": 1}

    return composed_query
Quality Control
Additional terms can be used to steer search away from low quality, NSFW, or other undesirable results. More examples are provided in the Query Prompt Engineering tips.

---

Analysing Processing Time
Marqo provides a telemetry query parameter on the documents and search routes to help you better understand the processing time of your requests.

Getting Telemetry
If using the Python client then the telemetry is enabled for all requestst with it during the client instantiation.


import marqo

mq = marqo.Client(return_telemetry=True)
If using the REST API directly then you can enable the telemetry by adding the telemetry query parameter to the request. For search and add documents.


curl -XPOST 'http://localhost:8882/indexes/my-first-index/documents?telemtry=true' \
-H 'Content-type:application/json' -d '
{
  "documents": [ 
      {
           "Title": "The Travels of Marco Polo",
           "Description": "A 13th-century travelogue describing the travels of Polo",
           "Genre": "History"
        }, 
      {
          "Title": "Extravehicular Mobility Unit (EMU)",
          "Description": "The EMU is a spacesuit that provides environmental protection",
          "_id": "article_591",
          "Genre": "Science"
      }
  ],
  "tensorFields": ["Description"]
}'

curl -XPOST 'http://localhost:8882/indexes/my-first-index/search?telemetry=true' \
-H 'Content-type:application/json' -d '
{
    "q": "what is the best outfit to wear on the moon?",
    "limit": 10,
    "offset": 0,
    "showHighlights": true,
    "searchMethod": "TENSOR",
    "attributesToRetrieve": ["Title", "Description"]
}'
Understanding Telemetry
Search
For search requests, the telemetry will return a new telemetry key in the response with data as follows:


{
    "timesMs": {
      "search.vector_inference_full_pipeline": 11.1809599998905,
      "search.vector.processing_before_vespa": 12.090376000131,
      "search.vector.vespa": 17.501166999863926,
      "search.vector.postprocess": 1.4460830000189162,
      "POST /indexes/test-index/search": 84.4371719998087
    }
}
search.vector_inference_full_pipeline: The total time taken for the full pipeline of the vector search, this is the time it takes Marqo to do inference on your query.
search.vector.processing_before_vespa: The time taken for processing before Vespa this includes any steps around the inference.
search.vector.vespa: The time taken for the Vespa database to perform the HNSW search.
search.vector.postprocess: The time taken for post-processing the request.
POST /indexes/test-index/search: The total time taken for the search request.
Documents
For adding documents the telemetry will return a new telemetry key in the response with data as follows:


{
    "timesMs": {
        "image_download.imageserver.com/image1.jpg": 1185.17400199994,
        "image_download.thread_time": [
            1186.053542999844,
            524.2454180001914
        ],
        "image_download.imageserver.com/image2.jpg": 524.1457510001055,
        "image_download.full_time": 1186.757126999964,
        "add_documents.create_vectors": [
            173.112292000058,
            238.6752099999976
        ],
        "add_documents.processing_before_vespa": 1425.432337999883,
        "add_documents.vespa._bulk": 17.816249999896172,
        "add_documents.postprocess": 0.007333999974434846,
        "POST /indexes/test-index/documents": 1443.611254000052
    }
}
image_download.imageserver.com/image1.jpg: The time taken to download the image image1.jpg. (Images are downloaded in parallel)
image_download.imageserver.com/image2.jpg: The time taken to download the image image2.jpg. (Images are downloaded in parallel)
image_download.thread_time: The time taken to download the media files in parallel.
image_download.full_time: The total time taken to download the media files.
add_documents.create_vectors: The time taken to create the vectors for the documents, this is the pre-processing and inference with the model as well as any weighted combinations.
add_documents.processing_before_vespa: The time taken for processing before Vespa this includes all image downloading and inference.
add_documents.vespa._bulk: The time taken for Vespa to ingest the documents, this is the insertion of the data and the vector indexing in the HNSW index.
add_documents.postprocess: The time taken for post-processing the request.
POST /indexes/test-index/documents: The total time taken for the add documents request.

---

Query filtering in Marqo
You can use Marqo's query DSL to refine search results. The filters are executed efficiently as a pre-filter across an HNSW graph, if you're using a neural search query (the default query for Marqo).

Filters have several use cases, for example, restricting the results a specific user has access to or creating faceted search interfaces.

Query filters use a syntax to parse and split the provided query string based on operators, such as AND or NOT. The query then analyzes each split text independently before returning matching documents.

Example
In the following example, Marqo's filter query analyzer splits the query into two components, "country:USA" and "state:NY".


Python
cURL

results = mq.index("my-first-index").search(
    q="New York", filter_string="country:(United States) OR state:NY"
)

Marqo's query DSL is based on Lucene but with some differences.

all term queries must be connected to a field, e.g., city:(New York)

efficient range queries are supported for numeric types without manually specifying the type

fuzzy/approximate searches are not supported

Let's dig into some of the specifics:

Fields
Marqo's search terms must be fielded. You can search any field by typing the field name followed by a colon ":" and then the term you are looking for.

As an example, let's assume a Marqo index contains two fields, title and text and text is the default field. If you want to find the document entitled "The Right Way" which contains the text "go", you can enter:


title:(The Right Way) AND text:go
Since text is the default field, the field indicator is not required.

Note: The field is only valid for the term that it directly precedes, so the query


title:Do it right
Will only find "Do" in the title field. It will find "it" and "right" in the default field (in this case the text field).

Range Queries
Marqo supports efficient execution of range queries on numeric types.

Numbers from 0..100:


some_numeric:[0 TO 100]
Greater than or equal to 0:


some_numeric:[0 TO *]
IN Queries
Marqo supports the IN operator for restricting a field to a list of values. The value list must be enclosed in parentheses and comma-separated. Values with spaces must be enclosed in parentheses.

Currently IN is only supported for structured indexes. This operator is also only supported for the following field types: text, int, long, array<text>, array<int>, array<long>, and custom_vector. The _id field can also be filtered on with IN.

Text field example:


text_field IN (apple, banana, (wild cherry))
Integer field example:


int_field IN (1, 2, 3)
Boolean Queries
Marqo supports execution of boolean queries, if you reference true or false within the filter then it will be treated as a boolean.


some_bool:true
Boolean Operators
Boolean operators allow terms to be combined through logic operators. Marqo supports AND, "+", OR, NOT and "-" as Boolean operators(Note: Boolean operators must be ALL CAPS).

The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.


food:(ice cream) OR type:confectionary
The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND.

To search for documents that exactly match "ice cream" and "confectionary" use the query:


food:(ice cream) AND type:confectionary
The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.

To search for documents that match a type of "confectionary" but not "ice cream" use the query:


type:confectionary AND NOT food:(ice cream)
Grouping
Marqo filtering supports using parentheses to group clauses to form sub queries. This can be very useful if you want to control the boolean logic for a query.

To search for either type is "confectionary" or food is "ice cream" and sweetness is 10 use the query:


(type:confectionary OR food:(ice cream)) AND sweetness:10
Escaping Special Characters
Marqo supports escaping special characters that are part of the query syntax. The current list special characters are


+ - && || ! ( ) { } [ ] ^ " ~ * ? : \
To escape these character use backslash (\) before the character. For example to filter for (1+1):2 in the field myField use the query:


myField:\(1\+1\)\:2
Note that in languages like Python, you also need to escape the backslashes with their own backslashes. So the above filter string looks like the following, in the Python client:


my_index.search(q="what colour are plants?", filter_string="myField:\\(1\\+1\\)\\:2")
You also need to escape characters such as the special characters and spaces in fieldnames:


my\ field:hello
Filtering with array fields
Marqo supports filtering over array fields. This can be useful for usecases such as filtering over document tags. See here for more details.

Filtering on multimodal objects
Marqo supports filtering on multimodal object. The dot notation is used to search fields within a multimodal combination object. For the object with the following structure.


{
   "_id": "article_1",
   "my_combination_field": {
      "my_interior_field": "hello there",
      "img": "https://my-img-store.jpg"
   }
}
You can filter for this document with following filter string:


my_combination_field.my_interior_field:(hello there)
Filtering on custom vector fields
Marqo supports filtering on custom vector fields. This will only work on the given content field. You cannot filter on the vector of a custom_vector field.


{
    "_id": "custom_audio_doc_1",
    "my_custom_vector": {
            "vector": [0.1, 0.2, 0.3...],
            "content": "Singing audio file"
    }
}
You can filter for this document with following filter string:


my_custom_vector:(Singing audio file)

---

Overview
The size of the indexed data will depend on the number of vectors, the dimension of the vectors and the amount of meta-data that is indexed alongside the vectors.

The number of vectors will depend on the settings chosen, the dimension of the vector will depend on the model and the size of meta-data will depend on what else is indexed.

The exact amount will vary from use case to use case but we provide some examples below to help estimate and understand the storage requirements. For the most accurate estimates, a small but representative amount of data (1000 documents) should be indexed and the per document storage estimated from this.

Examples
Below are some prototypical examples to help understand the storage requirements.

Example 1. Indexing text with the default model
In this example, each document has a single text field (”text”) and a short amount of text. For example:


document = {"text": "this is an example. here is some more text."}
Each of these occupies between 5-10 kB depending on the model used. Thus, indexing 1M documents would require approximately 5-10GB of storage.

Example 2. Indexing images with CLIP ViT-B/32
In this example, each document has a single image field (”image”) that contains a uri for the images. For example:


document = {"image": "https://some.domain/an.image/image.webp"}
Each of these occupies between 15-20 kB depending on the model used. Thus, indexing 1M documents would require approximately 15-20GB of storage.

Example 3. Indexing text with the default model and multiple fields
In this example, each document has a multiple text fields (”text1” and “text2”) and a short amount of text. For example:


document = {
    "text1": "this is an example. here is some more text.",
    "text2": "this is an example. here is some more text.",
}
Each of these occupies between 10-20 kB depending on the model used. Thus, indexing 1M documents would require approximately 10-20GB of storage.

Reducing storage
There are several strategies that can be used to reduce the amount of storage required. These are listed below:

Include specific fields to be transformed into vector fields. The non-tensor fields can still be stored, filtered and searched with lexical search. An example below will turn the field “Description” into vectors but will exclude “Title” and “Genre”. This would reduce the amount of storage by ~3x.

mq.index("my-first-index").add_documents(
    [
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing the travels of Polo",
            "Genre": "History",
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection",
            "_id": "article_591",
            "Genre": "Science",
        },
    ],
    tensor_fields=["Description"],
)
Modify the internal segmentation settings for text (see here for details). By default, blocks of text that are two sentences long will be turned into 1 vector. To create an index with modified settings, change the text processing parameters. For fields that are longer than 2 sentences the settings below would reduce storage by ~2x. Note already existing indexes cannot have their settings modified.

settings = {
    "textPreprocessing": {
        "splitLength": 4,
        "splitOverlap": 0,
        "splitMethod": "sentence",
    }
}

response = mq.create_index("my-index", settings_dict=settings)

---

Using Marqo with a GPU
This section outlines how to use Marqo with GPU's as well as some troubleshooting. Note that the following configurations are for development only, if you would like to run Marqo in production, we recommend you use Marqo Cloud or follow our Marqo on Kubernetes guide.

Deploying Marqo on a single GPU instance within AWS
Navigate into the ec2 console, select Instances on the left panel and then select Launch instances.
Select the correct AMI. The recommended AMI is "Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.3". Select the Ubuntu version of this AMI. The AMI ID in us-east-1 is ami-03db1a48758a57ae6.
Configure the access key as needed, then select an instance type with an NVIDIA GPU. We recommend g4dn.xlarge (due to its price performance) for development. Ensure you configure the instance with sufficient storage for your dataset (100-200GB of disk space should give you a decent margin).
Connect to the instance, and in the terminal, run the following command to start Marqo:

docker run --name marqo --gpus all -p 8882:8882 marqoai/marqo:latest
Note that --gpus all has been added.
In another window connected to the instance you can run the following command to check Marqo is running:


curl -XGET 'http://localhost:8882/'
You should see the following response:


{"message":"Welcome to Marqo","version":"2.10.0"}
Deploying single instance Marqo with GPU on other machines and providers
Currently, only CUDA based (Nvidia) GPU's are supported. If you have a GPU on the host machine and want to use it with Marqo, there are two things to do;

Install nvidia-docker2.
Add a --gpus all flag to the Docker run command. Note that this flag should appear after the run command but before the end. See the full Docker command in step 2 below.
Detailed instructions
Install nvidia-docker2 which is required for the GPU to work with Docker. The three steps below will install it for a Ubuntu based machine (refer to the original instructions for more details);


distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-docker2
Once nvidia-docker2 is installed, a simple modification to the Docker command is all that is needed. This is achieved by adding a --gpus all flag to the docker run command. For example, the Docker command would become,


docker run --name marqo --gpus all -p 8882:8882 marqoai/marqo:latest
Note that --gpus all has been added.
Using Marqo outside of Docker
Marqo outside Docker will rely on the system setup to use the GPU. If you can use a GPU normally with pytorch then it should be good to go. The usual caveats apply though, the CUDA version of pytorch will need to match that of the GPU drivers (see below on how to check).

Troubleshooting
Drivers
In order for the GPU to be used within Marqo, the underlying host needs to have NVIDIA drivers installed. The current driver can be easily accessed by typing


nvidia-smi
in a terminal. If there is no output then there may be something wrong with the GPU setup and installing or updating drivers may be necessary.

CUDA
Aside from having the correct drivers installed, a matching version of CUDA is required. The marqo Dockerfile comes setup to use CUDA 11.4.2 by default. The Dockerfile can be easily modified to support different versions of CUDA.

Checking the status of your GPU and CUDA
To see if a GPU is available when using pytorch, the following can be used to check (from python);


import torch

torch.cuda.is_available()  # is a GPU available
torch.version.cuda  # get the CUDA version
torch.cuda.device_count()  # get the number of devices
To check your driver and maximum CUDA version supported, type the following into the terminal;

nvidia-smi
Pytorch comes with its own bundled CUDA which allows many different CUDA versions to be used. Follow the getting started guide to see how to install different versions of pytorch and its respective CUDA version.
Marqo will use CUDA if available, you can test if CUDA is working by forcing CUDA with the device="cuda" argument.


mq.index("my-first-index").add_documents(
    [
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing Polo's travels",
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection, "
            "mobility, life support, and communications for astronauts",
            "_id": "article_591",
        },
    ],
    tensor_fields=["Title", "Description"],
    device="cuda",
)

---

Check Marqo Version
To check the version of Marqo you are using, send a get request to the root endpoint. If you are using the Python client, use the Index.get_marqo() method


Python
cURL

mq.index("my-first-index").get_marqo()

Response: 200 OK

{'message': 'Welcome to Marqo', 'version': '2.0.0'}

---

Changing the default storage location for Docker
Docker comes with a default location for storing artefacts locally (e.g. /var/lib/docker on linux) like containers and images. It can be convenient to store these in alternative places, like an attached storage volume. The following steps can be performed to change the default location;


sudo mkdir /the_new_directory/to_store/docker_artefacts/
sudo systemctl stop docker
sudo mount --rbind /the_new_directory/to_store/docker_artefacts /var/lib/docker
sudo systemctl start docker
where /the_new_directory/to_store/docker_artefacts/ should be replaced with the new location. Note that the defualt docker location may be different to what is shown above and should be determined from the operating system that docker is running on.

---

Configuring Marqo
Marqo is configured through environment variables passed to the Marqo container when it is run.

Configuring usage limits
Limits can be set to protect the resources of the machine Marqo is running on.

Configuration name  Default Description
MARQO_MAX_DOC_BYTES 100000  Maximum document size allowed to be indexed
MARQO_MAX_RETRIEVABLE_DOCS  10000 Maximum number of documents allowed to be returned in a single request. The maximum value this can be set to is 10000.
MARQO_MAX_CUDA_MODEL_MEMORY 4 Maximum CUDA memory usage (GB) for models in Marqo. For multi-GPU, this is the max memory for each GPU.
MARQO_MAX_CPU_MODEL_MEMORY  4 Maximum RAM usage (GB) for models in Marqo.
MARQO_MAX_VECTORISE_BATCH_SIZE  16  Maximum size of batch size to process in parallel (when, for example, adding documents ).
MARQO_MEDIA_DOWNLOAD_THREAD_COUNT_PER_REQUEST 5 Maximum number of threads to download media in parallel.
MARQO_IMAGE_DOWNLOAD_THREAD_COUNT_PER_REQUEST 20  Maximum number of threads to download images in parallel.
MARQO_MAX_SEARCH_VIDEO_AUDIO_FILE_SIZE  387973120 Maximum size of video or audio file to be searched in a single request in bytes.
MARQO_MAX_ADD_DOCS_VIDEO_AUDIO_FILE_SIZE  387973120 Maximum size of video or audio file to be added to an index in bytes.
MARQO_MAX_DOCUMENTS_BATCH_SIZE  128 Maximum number of documents that can be added or updated in a single request.
MARQO_MAX_DELETE_DOCS_COUNT 10000 Maximum number of documents that can be deleted in a single request.
VESPA_POOL_SIZE 10  The size of the connection pool for Vespa operations including search. This should be set to a value at least as large as MARQO_MAX_CONCURRENT_SEARCH.
VESPA_FEED_POOL_SIZE  10  Maximum Vespa feed concurrency per indexing batch.
VESPA_GET_POOL_SIZE 10  Maximum Vespa get concurrency per request when retrieving documents by ID.
VESPA_DELETE_POOL_SIZE  10  Maximum Vespa delete concurrency per request.
VESPA_PARTIAL_UPDATE_POOL_SIZE  10  Maximum Vespa update concurrency per request.
VESPA_SEARCH_TIMEOUT_MS 1000  Amount of time before search request to Vespa times out (milliseconds).
Example

docker run --name marqo -p 8882:8882 \
    -e "MARQO_MAX_DOC_BYTES=200000" \
    -e "MARQO_MAX_RETRIEVABLE_DOCS=600" \
    -e "MARQO_MAX_CUDA_MODEL_MEMORY=5" \
    -e "VESPA_SEARCH_TIMEOUT_MS=2000" marqoai/marqo:latest
In the above example a marqo container is being run with the following limits:

The max number of fields per index is capped at 400

The max size of an indexed document is 0.2mb

The max number of documents allowed to be returned in a single request is 600

The max CUDA memory usage for models in Marqo is 5GB.

The max number of replicas allowed when creating an index is 2.

The search timeout for Vespa is 2 seconds.

Configure backend communication
This section describes the environment variables that can be used to configure Marqo's communication with the backend. It can be helpful to set these variables when Marqo is running in a container and needs to communicate with a Vespa running on a separate container or a difference host machine.

Note: Regularly upgrade Vespa when hosting it yourself. New releases of Marqo leverage features and bug fixes introduced in the latest versions of Vespa. If you are running Marqo 2.13.0, please upgrade Vespa to version 8.396.18 or later. This helps prevent potential issues, such as long response times when adding documents to an unstructured index or unexpected behavior during Marqo upgrades. For more details or if you encounter any issues, please refer to the Troubleshooting Guide.

Configuration name  Default Description
VESPA_CONFIG_URL  "http://localhost:19071"  URL for Vespa configuration.
VESPA_QUERY_URL "http://localhost:8080" URL for querying the Vespa instance.
VESPA_DOCUMENT_URL  "http://localhost:8080" URL for document operations in the Vespa instance.
VESPA_CONTENT_CLUSTER_NAME  "content_default" Name of the Vespa content cluster.
ZOOKEEPER_HOSTS null  Hosts for the Zookeeper server, no "https" or "http" required in the string. If not set, Marqo will skip the connection to the Zookeeper server.
Example: Running Marqo on a standalone Vespa container
In this example, we will start a Vespa container, initialise it with an application package, and run Marqo container on that Vespa container.

Step 1: Initialize Vespa Container Environment
Start a Vespa container using the latest Vespa image. Make sure to expose the necessary ports for the config server, container server, and Zookeeper.


docker run --detach --name vespa -p 8080:8080 -p 19071:19071 -p 2181:2181 vespaengine/vespa:8
Step 2: Deploy an Application Package to Configure Vespa
Clone the Marqo repository and deploy an application package for local runs. This setup ensures that the vector store is configured correctly.


git clone https://github.com/marqo-ai/marqo.git
cd marqo/scripts/vespa_local
zip -r - * | curl --header "Content-Type:application/zip" --data-binary @- http://localhost:19071/application/v2/tenant/default/prepareandactivate
You can verify that the vector store has been set up correctly by visiting http://localhost:8080 in your browser. The vector store can take a few minutes to start responding after the initial configuration.

Step 3: Launch Marqo with Vespa Configuration
With your external vector store ready, you can now run Marqo configured to use it:


docker run --name marqo -p 8882:8882 --add-host host.docker.internal:host-gateway \
    -e VESPA_CONFIG_URL="http://host.docker.internal:19071" \
    -e VESPA_DOCUMENT_URL="http://host.docker.internal:8080" \
    -e VESPA_QUERY_URL="http://host.docker.internal:8080" \
    -e ZOOKEEPER_HOSTS="host.docker.internal:2181" \
    marqoai/marqo:latest
Enhancing Your Vespa Setup with Kubernetes
For a more robust and scalable setup, follow the instructions provided in marqo-on-kubernetes Github repo. This guide offers detailed steps for setting up a Vespa cluster using Kubernetes across various cloud providers.

Configuring preloaded patch models
Variable: MARQO_PATCH_MODELS_TO_PRELOAD

Default value: '[]'

Expected value: A string of comma-separated patch model names. Currently supported patch models are: 'simple', 'overlap', 'fastercnn', 'frcnn', 'marqo-yolo', 'yolox', 'dino-v1', 'dino-v2', 'dino/v1', 'dino/v2'.

This is a list of patch models to load and pre-warm as Marqo starts. This prevents a delay during initial image processing.

Configuring preloaded models
Variable: MARQO_MODELS_TO_PRELOAD

Default value: '["hf/e5-base-v2", "open_clip/ViT-B-32/laion2b_s34b_b79k"]'

Expected value: A JSON-encoded array of strings or objects.

This is a list of models to load and pre-warm as Marqo starts. This prevents a delay during initial search and index commands in actual Marqo usage.

Models in string form must be names of models within the model registry. You can find these models here

Models in object form must have model and modelProperties keys.

Model Object Example (OPEN CLIP model)

'{
    "model": "my-open-clip-1",
    "modelProperties": {
        "name": "ViT-B-32-quickgelu",
        "dimensions": 512,
        "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_avg-8a00ab3c.pt",
        "type": "open_clip"
    }
}'
Model Object Example (CLIP model)

'{
    "model": "generic-clip-test-model-2",
    "modelProperties": {
        "name": "ViT-B/32",
        "dimensions": 512,
        "type": "clip",
        "url": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt"
    }
}'
Marqo Run Example (containing both string and object)

export MY_MODEL_LIST='[
    "sentence-transformers/stsb-xlm-r-multilingual",
    "hf/e5-base-v2",
    {
        "model": "generic-clip-test-model-2",
        "modelProperties": {
            "name": "ViT-B/32",
            "dimensions": 512,
            "type": "clip",
            "url": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt"
        }
    }
]'

docker run --name marqo -p 8882:8882 \
    -e MARQO_MODELS_TO_PRELOAD="$MY_MODEL_LIST" \
    marqoai/marqo:latest
Configuring media download threads
Marqo provides environment variables and parameters to control the number of threads used for downloading media during processing.

Configuration name  Default Description
MARQO_MEDIA_DOWNLOAD_THREAD_COUNT_PER_REQUEST 5 Maximum number of threads to download media in parallel.
MARQO_IMAGE_DOWNLOAD_THREAD_COUNT_PER_REQUEST 20  (Deprecated) Maximum number of threads to download images.
Thread Count Determination
Marqo determines the number of threads for media downloads in the following order of priority:

If media_download_thread_count is set in the add_documents parameters and is different from the default, this value is used.
If the MARQO_MEDIA_DOWNLOAD_THREAD_COUNT_PER_REQUEST environment variable is explicitly set and is different from the default, this value is used.
If the model type is languagebind, the thread count is set to 5.
If image_download_thread_count is explicitly set in the add_documents parameters and is different from the default, this value is used.
If the MARQO_IMAGE_DOWNLOAD_THREAD_COUNT_PER_REQUEST environment variable is explicitly set and is different from the default, this value is used.
If none of the above conditions are met, the default value (for MARQO_IMAGE_DOWNLOAD_THREAD_COUNT_PER_REQUEST) is used.
MARQO_MEDIA_DOWNLOAD_THREAD_COUNT_PER_REQUEST - This is the preferred parameter for controlling media download threads. It applies to all types of media, including images, videos, and audio files.

MARQO_IMAGE_DOWNLOAD_THREAD_COUNT_PER_REQUEST - This environment variable is deprecated and will be removed in future versions. It's maintained for backward compatibility but only affects image downloads.

Configuring log level
Variable: MARQO_LOG_LEVEL

Default value: INFO

Expected value: a str from one of ERROR, WARNING, INFO, DEBUG.

This environment variable will change the log level of timing logger and uvicorn logger. A higher log level (e.g., ERROR) will reduce the amount of logs in Marqo, while a lower log level (DEBUG) will record more detailed information in the logs.

The default log level is INFO and is recommended for production environments. Setting log level toDEBUG can have performance implications and is not recommended for production environments.

Example

docker run --name marqo -p 8882:8882 \
    -e MARQO_LOG_LEVEL='warning' \
    marqoai/marqo:latest
Configuring throttling
Configuration name  Default Description
MARQO_ENABLE_THROTTLING "TRUE"  Adds throttling if "TRUE". Must be a str: Either "TRUE" or "FALSE".
MARQO_MAX_CONCURRENT_INDEX  8 Maximum allowed concurrent indexing threads
MARQO_MAX_CONCURRENT_SEARCH 8 Maximum allowed concurrent search threads
MARQO_MAX_CONCURRENT_PARTIAL_UPDATE 100 Maximum allowed concurrent partial update threads
These environment variables set Marqo's allowed concurrency across index and search. If these limits are reached, then Marqo will return 429 on subsequent requests. These should be set with respect to available resources of the machine Marqo will be running on.

Example

docker run --name marqo -p 8882:8882 \
    -e MARQO_ENABLE_THROTTLING='TRUE' \
    -e MARQO_MAX_CONCURRENT_SEARCH='10' \
    marqoai/marqo:latest
Marqo inference cache configuration
Configuration name  Default Description
MARQO_INFERENCE_CACHE_SIZE  0 (disabled)  The size (measured by the number of query-embedding pairs) of the marqo inference cache. Set it to a positive integer to enable this feature
MARQO_INFERENCE_CACHE_TYPE  "LRU" (least recently used) The eviction policy of the marqo inference cache. Supported types are "LRU", "LFU" (least frequently used)
These environment variables configure the size and eviction policy of the Marqo inference cache, which stores results from inference queries to improve search latency. Note that this cache does not apply on the add_documents endpoint. Consider enabling this feature if you frequently encounter a high volume of identical queries. By default, this feature is disabled.

Example

docker run --name marqo -p 8882:8882 \
    -e "MARQO_INFERENCE_CACHE_SIZE=20" \
    -e "MARQO_INFERENCE_CACHE_TYPE=LRU" \
    marqoai/marqo:latest
Marqo Video GPU Acceleration Configuration
Configuration Name  Default Description
MARQO_ENABLE_VIDEO_GPU_ACCELERATION None  Controls whether GPU acceleration is enabled for video decoding. Accepted values are TRUE or FALSE.
The environment variable MARQO_ENABLE_VIDEO_GPU_ACCELERATION determines whether Marqo uses GPU acceleration for video decoding.

Default Behavior: If this variable is not set, Marqo automatically decides based on the availability of a GPU on the host machine.
Set to TRUE: Forces Marqo to use GPU acceleration for video decoding. An error will be raised if GPU acceleration is not available.
Set to FALSE: Disables GPU acceleration for video decoding, ensuring CPU-based decoding is used.
Note: In addition to a compatible GPU, the NVIDIA drivers on the host must be version 550.54.14 or newer for GPU acceleration to function properly.

Example Usage
To enable GPU acceleration for video decoding, run the following Docker command:


docker run --name marqo --gpus all -p 8882:8882 \
    -e "MARQO_ENABLE_VIDEO_GPU_ACCELERATION=TRUE" \
    marqoai/marqo:latest
Advanced configuration
These are additional advanced configurations that can be set to customize Marqo's behavior. Most users will not need to change these values.

Configuration name  Default Description
MARQO_DEFAULT_EF_SEARCH 2000  Default HNSW efSearch value
MARQO_MAX_SEARCHABLE_TENSOR_ATTRIBUTES  null  The maximum allowed number of tensor fields to be searched in a single tensor search query. By default, there is no limit
MARQO_MAX_SEARCH_LIMIT  1000  The maximum allowed limit for search requests. This can be set up to 1000000.
MARQO_MAX_SEARCH_OFFSET 10000 The maximum allowed offset for search requests. This can be set up to 1000000.
MARQO_MAX_TENSOR_FIELD_COUNT_UNSTRUCTURED 100 The maximum allowed number of tensor fields to be added to a unstructured index created with Marqo 2.13.0+
MARQO_MAX_LEXICAL_FIELD_COUNT_UNSTRUCTURED  100 The maximum allowed number of lexical fields to be added to a unstructured index created with Marqo 2.13.0+
MARQO_THREAD_EXPIRY_TIME  1800  When throttling is enabled, this is the time in seconds after which a request thread's slot is automatically freed up
MARQO_ROOT_PATH null  Disk path where Marqo stores runtime artifacts such as downloaded models
ZOOKEEPER_CONNECTION_TIMEOUT  null  Connection timeout when connecting to Zookeeper
Third party environment variables
The following environment variables are managed by dependencies of Marqo rather than Marqo itself. They are intended for advanced users and should be configured with caution. These variables may be modified or deprecated in future Marqo versions.

Configuration Name  Default Value Description
HF_HUB_ENABLE_HF_TRANSFER null  Set this to 1 to enable faster downloads from Hugging Face on high-bandwidth networks. See the documentation for details.
HF_HUB_OFFLINE  null  Set this to 1 to skip HTTP requests when loading a Hugging Face model. This can be useful if you want to run Marqo in offline mode. Refer to the documentation for more details.


---


Disabling client logging
The Marqo Python client generates INFO-level messages during events like indexing batches of documents. To disable these, set Marqo's logging level to warning (or higher). Like so:


marqo.set_log_level('WARN')

---

Document field types
Strings
These are vectorised, if the field is specified is in tensor_fields during index time.

Floats
These aren't vectorised, but can be used to filter search results.

Bools
These aren't vectorised, but can be used to filter search results.

Ints
These aren't vectorised, but can be used to filter search results.

Array
Currently, only arrays of strings are supported.

Array fields must not be a tensor field during index time, else an error will be thrown.

This type of field can be used to filter search results and for lexical search.

Example

# index an array field called "my_tags", making sure it is not a tensor field 
mq.index("my-index").add_documents(documents=[
    {"Title": "Cool summer t-shirt", "_id": "1234", 'my_tags': ['summer', 'yellow']}], 
    tensor_fields=['Title']
)

# do a search request that filters based on the tags
mq.index("my-index").search(
    q="Something to wear in warm weather",
    filter_string="(my_tags:yellow) AND (my_tags:summer)"
)
Multimodal combination object
The multimodal combination object works with mappings. This field can consist of multiple child fields. The contents of these child fields will be vectorized and combined into a single tensor using a weighted-sum approach. The weights are specified in mappings. Each child field must have an assigned weight.

The combined tensor will be used for tensor search. The multimodal combination field must be in tensor_fields.

Child fields can be used for lexical search or tensor search with filtering. All the child fields and child fields content must be str.

Note that only a single vector is generated for a multimodal combination object per document. No chunking is applied.

Example

Python


# Create an index with "open_clip/ViT-B-32/laion2b_s34b_b79k" that can vectorise both text and images.
settings = {
    "treat_urls_and_pointers_as_images": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
}

mq.create_index("my-index", **settings)

# We add the document into our index
mq.index("my-index").add_documents(
    documents=[
        {
            "_id": "111",
            "Title": "my document",
            "my_text_attribute_1": "Riding horse",
            "my_image_attribute_1": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg",
        }
    ],
    tensor_fields=["combo_text_image"],
    mappings={
        "combo_text_image": {
            "type": "multimodal_combination",
            "weights": {"my_text_attribute_1": 0.5, "my_image_attribute_1": 0.5},
        }
    },
)

# tensor search
res = mq.index("my-index").search(q="Riding horse")

# lexical search
res = mq.index("my-index").search(
    q="Riding horse",
    search_method="LEXICAL",
)

# filter search
res = mq.index("my-index").search(
    q="Riding horse",
    filter_string="my_text_attribute_1:(Riding horse)",
)
Results
Here we have the search results for the filter search:


{
    "hits": [
        {
            "Title": "my document",
            "_highlights": [
                {
                    "combo_text_image": "{'my_text_attribute_1': 'Riding horse"
                    "'my_image_attribute_1': 'https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg'}"
                }
            ],
            "_id": "111",
            "_score": 0.726511350523296,
            "my_image_attribute_1": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg",
            "my_text_attribute_1": "Riding horse",
        }
    ],
    "limit": 10,
    "offset": 0,
    "processingTimeMs": 45,
    "query": "Riding horse",
}
Custom vector object
The custom vector object allows you to insert your own vectors into a Marqo index. This is especially useful if you have vectors generated from another model that you want to use for search in Marqo.

For unstructured indexes, this field type requires mappings. For structured indexes, this field type should only be declared upon index creation. The mappings object for a custom vector should only have a type key, which should be set to custom_vector.


mappings = {"my_custom_vector": {"type": "custom_vector"}}
The document field content for a custom vector must be a dictionary with keys: vector (required) and content (optional). If content is left empty, it will be assigned as empty string "".

Name  Type  Default Description
vector  List[Float]   Required, List length must match the dimensions property of the model used in the index you are adding documents to.
content String  ""  Optional, used for lexical search, filtering, and search result highlight.
The custom vector field must be in tensor_fields.

Note that only a single chunk (containing the given vector) is generated for a custom vector object per document. No vectorisation in Marqo is done. Marqo does not support custom vector fields being dependent fields of multimodal combination fields.

No normalization is done on custom vectors. Because of this, using prenormalized-angular as your annParameters.spaceType will result in unexpected behavior when searching with custom vectors. Instead, set it to a different space type, such as angular, euclidean, etc.

Example

Python (Unstructured Index)
Python (Structured Index)

# Create an index with the model that has the dimensions of your custom vectors. For example: "open_clip/ViT-B-32/laion2b_s34b_b79k" (dimension is 512).
# Only the model dimension matters, as we are not vectorising anything when using custom vector fields.
# Space type CANNOT be 'prenormalized-angular' for custom vectors, as they are not normalized.

settings = {
    "treat_urls_and_pointers_as_images": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "ann_parameters": {
        "spaceType": "angular",
        "parameters": {"efConstruction": 512, "m": 16},
    },
}

mq.create_index("my-first-index", **settings)

# Random vectors for example purposes. replace these with your own.
example_vector_1 = [i for i in range(512)]
example_vector_2 = [1 / (i + 1) for i in range(512)]

# We add the custom vector documents into our index (with mappings)
res = mq.index("my-first-index").add_documents(
    documents=[
        {
            "_id": "doc1",
            "my_custom_vector": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_1,
                "content": "Singing audio file",
            },
        },
        {
            "_id": "doc2",
            "my_custom_vector": {
                # Put your own vector (of correct length) here.
                "vector": example_vector_2,
                "content": "Podcast audio file",
            },
        },
    ],
    mappings={"my_custom_vector": {"type": "custom_vector"}},
    tensor_fields=["my_custom_vector"],
)

# Tensor Search
# Use the `context` search parameter to search with your own vectors.
res = mq.index("my-first-index").search(
    q={"dummy text": 0},
    context={
        "tensor": [{"vector": example_vector_1, "weight": 1}]  # custom vector from doc1
    },
)
print(res)

# Lexical Search
# You can search for the text in the `content` field.
res = mq.index("my-first-index").search(q="Podcast audio file", search_method="lexical")
print(res)

# Filter search
res = mq.index("my-first-index").search(
    q="A rider is riding a horse jumping over the barrier.",
    filter_string="my_custom_vector:(Singing audio file)",
)
print(res)

---


Indexing images on a host machine
When indexing or searching with images, Marqo accepts a pointer to a local file or a url. For some use cases, images may be stored locally and referenced using a local file pointer. By default, Marqo running within Docker will not be able to access these. For example, a document given by the following:


{
    "My_Image": "/directory/where_you_keep_or_save_images_for_searching/images/image5.png",
    "Description": "The hippopotamus, also called the common hippopotamus or river hippopotamus, is a large semiaquatic mammal native to sub-Saharan Africa",
    "_id": "hippo-facts5",
}
will attempt to read the file from the local store "/directory/where_you_keep_or_save_images_for_searching/images/image5.png". However, the default settings may not allow the file to be read.
1. Using a simple http server
A simple http server can be used to allow access to files in a local directory through localhost. These can then be accessed from within the docker container. The setup is described below.

Navigate to the directory where the images for indexing and searching will be:


cd "/directory/where_you_keep_or_save_images_for_searching/"
and run the following:

python3 -m http.server 8222
Now modify the paths of the images in the data to be indexed or searched to start with: http://host.docker.internal:8222/ instead of the local path: /directory/where_you_keep_or_save_images_for_searching/. For example, a document that wants to use a file with a local address of /directory/where_you_keep_or_save_images_for_searching/images/image5.png would become:

{
    "My_Image": "http://host.docker.internal:8222/images/image5.png",
    "Description": "The hippopotamus, also called the common hippopotamus or river hippopotamus, is a large semiaquatic mammal native to sub-Saharan Africa",
    "_id": "hippo-facts5",
}
The original local path of the image was /directory/where_you_keep_or_save_images_for_searching/images/image5.png but now for docker to access it it becomes http://host.docker.internal:8222/images/image5.png.
If you want to view the files that are being served on localhost, you can use a web-browser to view them at http://localhost:8222/. Note this address is different to what docker uses.

The same pattern holds for images that are used for searching. Those images should reside (or be saved in) the directory (or a sub-directory) from where the http server was run.

2. Mounting the local drive to Docker
To enable local files to be read, the Docker run command can be updated to include mounting local directories so they are accessible from within the Docker using --mount via the following;


docker rm -f marqo;docker run --name marqo --mount type=bind,source=/user/someone/images/,target=/user/someone/images/ -it -p 8882:8882 marqoai/marqo:latest
The source and target above should be changed to correspond to your needs.

---


Optimising Search
Only use one index per Marqo cluster
For production use cases we recommend only a single Marqo index per Marqo cluster. This results in more predictable resource usage and consistent search performance.

Be selective about tensor fields
Tensor embeddings are used to power Marqo's tensor search. It is possible to index any string fields as tensor fields.

There is a natural tradeoff, however, between tensor search and storage size. For certain fields, it may not be worth using tensor search, and thus, storing full embeddings for each field and document. For example, categorical fields such as a song's genre or a book's category may be represented as a string, but are mainly useful in keyword/lexical search, or as conditions in pre-filtering tensor search.

Marqo provides the ability to tune this tradeoff. When adding documents, only fields explicitly added to the tensor_fields parameter are indexed for tensor search. This selective indexing allows for a balance between the benefits of tensor search and storage size efficiency.

The best practice is to only select fields that will benefit from semantic and multimodal search as tensor fields.

For example:


import marqo

mq = marqo.Client(url="http://localhost:8882")

mq.create_index("my-first-index")

mq.index("my-first-index").add_documents(
    [
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing Polo's travels",
            "Genre": "History",
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection, "
            "mobility, life support, and communications for astronauts",
            "Genre": "Science",
        },
    ],
    tensor_fields=["Description"],
)
The above example will not store tensors against the "Genre" field, but we can still use it, for example:

## Search all of a specific genre
result = mq.index("my-first-index").search("History", search_method="LEXICAL")

## Filter out search results
results = mq.index("my-first-index").search(
    q="spacesuits", filter_string="Genre:Science"
)

---

Transferring Marqo's state
Version compatibility note
Please be aware of the Marqo versions of the old and new Marqo instances. It is safest to transfer state between Marqo containers of the same version.

Marqo versions 0.x.x, 1.x.x, and 2.x.x are not compatible. Attempting to transfer state between these Marqo versions will result in unexpected behavior.

Minor version updates are backward compatible. However, it is still recommended to upgrade just a few minor versions (ideally no more than 2) at a time to reduce the risk of encountering unexpected issues during state transfer.

Change in version 2.9
We changed the volume mounting points of the Marqo image in version 2.9. Prior to version 2.9, we exposed the entire /opt/vespa folder as a volume. Starting with version 2.9, we expose two separate volumes: /opt/vespa/var, which contains data and configs, and /opt/vespa/logs, which contains Vespa logs. This change addresses an issue that can cause Vespa to crash due to missing dependencies when an older version volume is mounted.

Guide
Please follow the steps below to transfer the state to a new Marqo docker container (with embedded Vespa)

From Marqo 2.9 release
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. You can create named volumes to persist data and transfer state across different Marqo versions.


docker volume create --name opt_vespa_var
docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var marqoai/marqo:<version>
If you forget to specify the volume mapping when you run marqo the first time, you can find the name of the anonymous volume using following command if the marqo container is not removed from docker.


export VESPA_VAR_VOLUME=$(docker inspect marqo | jq -r '.[0].Mounts.[] | select (.Destination == "/opt/vespa/var") | .Name')

# To upgrade to a newer version of Marqo
docker rm marqo
docker run --name marqo -it -p 8882:8882 -v $VESPA_VAR_VOLUME:/opt/vespa/var marqoai/marqo:<version>
Optionally, you can also bind-mount a local folder to expose vespa logs for debugging.


docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var -v $(pwd)/logs:/opt/vespa/logs marqoai/marqo:<version>
Prior to 2.9 release
In order to mount a volume or a local folder created prior to Marqo 2.9 to be used by any newer version, you will need to follow these steps:

If you use bind mounts, you can simply run this:


docker run --name marqo -it -p 8882:8882 -v <your local folder>/var:/opt/vespa/var marqoai/marqo:<version>
If you use volumes, you need to copy the var folder into a new volume to be used in newer versions of Marqo.


docker volume create --name opt_vespa_var
# copy var folder in the old volume to the new volume using marqo image. 
docker run --rm -it --entrypoint='' \
           -v <old volume name>:/opt/vespa_old \
           -v opt_vespa_var:/opt/vespa/var \
           marqoai/marqo:<version> sh -c "cd /opt/vespa_old/var ; cp -a . /opt/vespa/var"
docker run --name marqo -it -p 8882:8882 -v opt_vespa_var:/opt/vespa/var marqoai/marqo:<version>

---

Bring your own models
If the models in our registry do not meet your requirements, or you have a custom model that you want to use, you can bring your own model to Marqo. In this section, we will show you how to use your own OpenCLIP models and sentence transformers models in Marqo.

Bring your own OpenCLIP model
Marqo supports you to use your own OpenCLIP models fine-tune under the OpenCLIP framework. To load a custom OpenCLIP model, you need to provide the model properties in the index settings. A full details of the settings in modelProperties are listed below:

Field Name  Type  Default Value Description
name  String  No Default  The name of the model. It can be the architecture (e.g., "ViT-B-32") of the model or the Hugging Face model card starting with "hf-hub:".
dimensions  Integer No Default  The dimension of the embeddings generated by the model.
type  String  No Default  The type of the model. It should be "open_clip" since we are loading an OpenCLIP model here.
url String (Optional) None  The URL of the model checkpoint. Cannot be provided together with "modelLocation".
modelLocation Dict (Optional) None  The location of the model in S3 or Hugging Face. Cannot be provided together with "url".
jit Boolean (Optional)  False A boolean indicating whether the model is JIT compiled.
precision String (Optional) "fp32"  The precision of the model. It should be either "fp32" or "fp16".
tokenizer String (Optional) None  The Hugging Face tokenizer to be loaded. Provide this if you want to overwrite the tokenizer inferred from name.
imagePreprocessor String (Optional) "OpenCLIP"  The image preprocess configuration. Must be one of "SigLIP", "OpenAI", "OpenCLIP", "MobileCLIP", or "CLIPA".
mean  List[float] (Optional)  None  The mean of the image preprocessor. If provided, it will overwrite the loaded configuration.
std List[float] (Optional)  None  The standard deviation of the image preprocessor. If provided, it will overwrite the loaded configuration.
size  Integer (Optional)  None  The size of the image preprocessor. If provided, it will overwrite the loaded configuration.
note  String (Optional) None  A place to add notes to your model. This does not affect your model loading process.
pretrained  String (Optional) None  A place to indicate the pretrained dataset of your model. This does not affect your model loading process.
Most of the fields are optional and have default values. You can provide the fields you want to customize in the modelProperties. However, you need to provide at least the name, dimensions, and type fields to load a custom OpenCLIP model. There are two ways to load a custom OpenCLIP model in Marqo:

Load from a Hugging Face model card
To load a custom OpenCLIP model from a Hugging Face model card, you need to provide the model card name with the "hf-hub:" in the name, the dimensions of the model in dimensions, and the type of the model in type as "open_clip". Other fields are neglected in this case. This suits the case where you want to load a public model card from Hugging Face.

For example, instead of using loading the Marqo FashionCLIP model from the registry, you can load it from the Hugging Face with the following code:


settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "marqo-fashion-clip-custom-load",
    "modelProperties": {
        "name": "hf-hub:Marqo/marqo-fashionCLIP",
        "dimensions": 512,
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index(
    "marqo-fashion-clip-custom-load-index", settings_dict=settings
)
Load from a checkpoint file
This is the case where you have a custom OpenCLIP model checkpoint file and you want to load it in Marqo. This has the highest flexibility as you can load any custom model you have fine-tuned, from any source, and with any configurations, as long as the architecture is supported by OpenCLIP.

You need to provide the model name in name which is the architecture of the model (e.g., "ViT-B-32", "ViT-L-16-SigLIP"), the dimensions of the model in dimensions, and the type of the model in type as "open_clip".

You have two options to provide the checkpoint file: - 1. Provide the URL of the checkpoint file in url. The url should be accessible by Marqo and link to the checkpoint file with the format of *.pt.

Provide the location of the checkpoint file in S3 or Hugging Face in modelLocation. The modelLocation has the following fields:
Field Name  Type  Default Value Description
s3  Dict  No Default  A dictionary with "Bucket" and "Key" fields to locate the *.pt checkpoint
hf  Dict  No Default  A dictionary with "repoId" and "filename" fields to locate the *.pt checkpoint
authRequired  Bool  False A boolean indicating whether the authentication is required.
If authentication is required, you need to provide the authentication information in when you search or add documents to the index.

You can provide other fields like jit, precision, tokenizer, imagePreprocessor, mean, std, size, note, in the modelProperties to configure your model.

Examples
Here are some examples to load a custom OpenCLIP model in Marqo. Note that if your name has the "hf-hub:" prefix, we will try to load it from Hugging Face and ignore the url and modelLocation fields. Otherwise, if you provide the url or modelLocation, we will load the model from the provided location and treat the name as the model architecture.

Example 1: Load a custom OpenCLIP model from a public URL without configurations

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-own-clip-model",
    "modelProperties": {
        "name": "ViT-B-32",
        "dimensions": 512,
        "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_e32-46683a32.pt",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)
The above code loads a custom OpenCLIP model from a public URL. Note it is the same as loading the model open_clip/ViT-B-32/lainon400m_e32 from the registry. We use the public URL of the model checkpoint as an example.

Example 2: Load a custom OpenCLIP model from a public URL with custom configurations

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-own-clip-model",
    "modelProperties": {
        "name": "ViT-B-16-SigLIP",
        "dimensions": 768,
        "url": "https://huggingface.co/Marqo/marqo-fashionSigLIP/resolve/main/open_clip_pytorch_model.bin",
        "imagePreprocessor": "SigLIP",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)
The above code loads a custom OpenCLIP model from a public URL with custom configurations for the image preprocessor. It is very important to provide the correct imagePreprocessor configuration to match the model architecture as Marqo can not infer the correct configuration from the model name when you load a checkpoint file and will use the default configuration("OpenCLIP"). The imagePreprocessor is set to "SigLIP" in this example to match the model architecture ViT-B-16-SigLIP.

Note this is the same as loading the Marqo FashionSigLIP model from the registry. We use the public URL of the model checkpoint as an example.

Example 3: Load a custom OpenCLIP model from a private S3 bucket with authentication

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "my-private-clip-model",
    "modelProperties": {
        "name": "ViT-B-32",
        "dimensions": 512,
        "modelLocation": {
            "s3": {
                "Bucket": "my-prive-bucket",
                "Key": "my-private-model-checkpoint.pt",
            },
        },
        "authRequired": True,
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("my-own-clip-model", settings_dict=settings)

model_auth = {
    "s3": {
        "aws_secret_access_key": "my-secret-access-key",
        "aws_access_key_id": "my-access-key-id",
    }
}

mq.index("my-own-clip-model").search("test", model_auth=model_auth)
The above code loads a custom OpenCLIP model from a private S3 bucket with authentication. The authRequired is set to True and you need to provide the authentication information when you search or add documents to the index.

Bring your own Hugging Face Sentence Transformers models
Marqo supports you to use your own Hugging Face Sentence Transformers models to generate embeddings for your text data. You can use your own fine-tuned models to achieve better search results in domain-specific tasks. Generally, your model should follow the Hugging Face Sentence Transformers model format and can be loaded with AutoModel.from_pretrained and AutoTokenizer.from_pretrained in the transformers library.

A full details of the settings in modelProperties are listed below:

Field Name  Type  Default Value Description
name  String (Optional) None  The name of the model. This can be the huggingface repo name.
dimensions  Integer No Default  The dimension of the embeddings generated by the model.
type  String  No Default  The type of the model. It must be one of hf or hf_stella.
url String (Optional) None  The URL of the model checkpoint. Cannot be provided together with "modelLocation".
modelLocation Dict (Optional) None  The location of the model in S3 or Hugging Face. Cannot be provided together with "url".
poolingMethod String (Optional) "mean"  The pooling method to generate the sentence embeddings. It should be one of "mean", or "cls".
note  String (Optional) None  A place to add notes to your model. This does not affect your model loading process.
trustRemoteCode*  Bool (Optional) False Whether to trust the remote code when loading the model. Set this to True if you are loading a hf_stella models
tokens  Integer (Optional)  128 The maximum number of tokens to be used in the tokenizer.
text_query_prefix String (Optional) None  The default prefix to be added to the text query. Note this will be overwritten by the text_query_prefix parameter in search.
text_chunk_prefix String (Optional) None  The default prefix to be added to the documents chunks. Note this will be overwritten by the text_chunk_prefix parameter in add_document
* Enabling trustRemoteCode allows the model to execute code from remote sources, which can be necessary for custom functionality in hf_stella models. However, it introduces security risks, as unverified code may be executed. It is recommended to enable this flag only for trusted sources and use appropriate access controls and monitoring.

To load your model, at least one of the name, url, or modelLocation fields should be provided.

Load from a Hugging Face model card
The easiest way to load your fine-tuned model is to use the Hugging Face model card. After the fine-tuning, you can upload your model to the Hugging Face model hub and use the model card to load your model in Marqo. The model can be public or private. If the model is private, you need to provide the authentication information when you search or add documents to the index.

If the model is public, you can load it with the following code:


# Loading the model from a public Hugging Face model card:
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "name": "<your-public-huggingface-model-card-name>",
        "dimensions": 384,  # the dimension of the embeddings generated by the model
        "type": "hf",
        "tokens": 128,  # the maximum number of tokens to be used in the tokenizer
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)
If the model is private, you can load it with the following code:


settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "modalLocation": {
            "hf": {
                "repoId": "<your-private-huggingface-model-card-name>",
            },
            "authRequired": True,
        },
        "dimensions": 384,  # the dimension of the embeddings generated by the model
        "type": "hf",
        "tokens": 128,  # the maximum number of tokens to be used in the tokenizer
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)

model_auth = {"hf": {"token": "<your-hf-token>"}}

mq.index("test-custom-hf-model").search("test", model_auth=model_auth)
Load from a zip file
You can also provide a zip file containing all the your fine-tuned model and load it in Marqo. The zip file should contain the necessary files for the model, including the model checkpoint, tokenizer, configurations, etc. Note that these files should be in the root directory of the zip file.

The zip file can be provided through 3 ways: - 1. A public URL - 2. A public/private S3 bucket - 3. A public/private Hugging Face repository

Here is the code:


# Load from a public url
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "url": "https://path/to/your/sbert/model.zip",
        "dimensions": 384,  # the dimension of the embeddings generated by the model
        "type": "hf",
        "tokens": 128,  # the maximum number of tokens to be used in the tokenizer
    },
    "normalizeEmbeddings": True,
}

# Load from a s3 bucket
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "modelLocation": {
            "s3": {
                "Bucket": "<your-s3-bucket-name>",
                "Key": "<your-zip-file-key.zip>",  # a zip file
            }
        },
        "dimensions": 384,  # the dimension of the embeddings generated by the model
        "type": "hf",
        "tokens": 128,  # the maximum number of tokens to be used in the tokenizer
    },
    "normalizeEmbeddings": True,
}

# Load from a hf repository
settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "modelLocation": {
            "hf": {
                "repoId": "<your-hf-repo-name>",
                "filename": "<your-zip-file-key.zip>",  # a zip file
            }
        },
        "dimensions": 384,  # the dimension of the embeddings generated by the model
        "type": "hf",
        "tokens": 128,  # the maximum number of tokens to be used in the tokenizer
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)
Stella Models
Marqo also supports loading the Stella models. The dunzhang/stella_en_400M_v5 is included in the model registry, and you can load it by checking the guides here.

If you want to load your fine-tuned Stella model, you can use the same loading methods above with the type set to "hf_stella", and set the trustRemoteCode to True. The Stella models are trained by MRL so they can have multiple dimensions. However, Marqo requires you to concatenate the linear layers within the model. Your model must provide end-to-end embeddings with a single dimension.

Examples
Here are some examples to load your own Hugging Face Sentence Transformers model in Marqo.

Example 1: Load a model from a public Hugging Face model card
The sentence-transformers/nli-bert-base-cls-pooling is not included in the model registry. You can still load it from the Hugging Face model card with the following code:


settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "name": "sentence-transformers/nli-bert-base-cls-pooling",
        "dimensions": 768,
        "type": "hf",
    },
    "normalizeEmbeddings": True,
}
Example 2: Load a private model from a Hugging Face model card
Here, we load a private model from the Hugging Face model card with the following code:


settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "modelLocation": {
            "hf": {
                # This is a private model that you don't have access to
                "repoId": "Marqo/e5-base-v2-private-test",
            },
            "authRequired": True,
        },
        "dimensions": 768,
        "type": "hf",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)

model_auth = {"hf": {"token": "<your-hf-token>"}}

res = mq.index("test-custom-hf-model").search("test", model_auth=model_auth)
Example 3: Load a model from a public URL
Here, we load a model from a public URL to a zip file with the following code:


settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "url": "https://marqo-ecs-50-audio-test-dataset.s3.us-east-1.amazonaws.com/test-hf.zip",  # a public URL
        "dimensions": 384,
        "type": "hf",
    },
    "normalizeEmbeddings": True,
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)
Example 4: Load a Stella model from a public URL
Here, we load a Stella model from a public URL to a zip file with the following code. Note that this feature is not available for cloud users for security reasons.


settings = {
    "model": "your-own-sentence-transformers-model",
    "modelProperties": {
        "url": "https//:a-public-url-to-your-stella-model/model.zip",
        "dimensions": 1024,
        "type": "hf_stella",
        "trustRemoteCode": True,
    },
}

response = mq.create_index("test-custom-hf-model", settings_dict=settings)
Bring your own LanguageBind models
Marqo supports the use of custom LanguageBind models to embed your text, image, audio, and video data, enhancing the modalities in your search. Several LanguageBind models are included in the model registry and can be loaded directly. If you have your own LanguageBind model, you can load it using model properties. Full details of the settings in modelProperties are listed below:

Field Name  Type  Default Value Description
dimensions  Integer No Default  The dimensions of the embeddings generated by the model.
type  String  No Default  The type of the model, which should be "languagebind".
modelLocation Dict[str, modalityLocation](Optional) None  A dictionary specifying the model location for each modality.
supportedModalities List  No Default  A list of modalities supported by the model. Must include "text", and optionally "image", "audio", and "video".
Details of modelLocation
modelLocation should be a dictionary with the following keys: "tokenizer", "image", "audio", "video". At least one of "image", "audio", or "video" must be provided to load your model, while "tokenizer" is optional. The provided locations must match the supported modalities of the model. For example, if the model supports "image" and "audio", you must provide the model location for both modalities.

Each key in modelLocation must have a "modalityLocation" dictionary with the following fields:

Field Name  Type  Default Value Description
s3  Dict  No Default  A dictionary with "Bucket" and "Key" fields to locate a *.zip file.
hf  Dict  No Default  A dictionary with "repoId" and "filename" (optional) fields to locate a *.zip file or a Hugging Face repository.
url str No Default  The URL of the model. It should refer to a *.zip file containing all required components.
Examples
We recommend three ways to load your own LanguageBind model in Marqo: 1. Load from a public or private Hugging Face model card. 2. Load from a public URL. 3. Load from a public or private S3 bucket.

Below are examples for each method.

Example 1: Load from a Hugging Face model card
You can upload your LanguageBind model to the Hugging Face model hub and use the model card to load your model in Marqo. For example, you can load the video model, the audio model, and the image model with the following code:


settings = {
    "model": "your-own-languagebind-model",
    "treatUrlsAndPointersAsMedia": True,
    "modelProperties": {
        "dimensions": 768,
        "type": "languagebind",
        "supportedModalities": ["text", "audio", "video", "image"],
        "modelLocation": {
            "video": {
                "hf": {
                    "repoId": "Marqo/LanguageBind_Video_V1.5_FT",
                },
            },
            "audio": {
                "hf": {
                    "repoId": "Marqo/LanguageBind_Audio_FT",
                },
            },
            "image": {
                "hf": {
                    "repoId": "Marqo/LanguageBind_Image",
                },
            },
        },
    },
    "normalizeEmbeddings": True,
}

mq.create_index("test-custom-languagebind-model", settings_dict=settings)
If the Hugging Face repository is private, you need to provide authentication information when you search or add documents to the index:


mq.index("test-custom-languagebind-model").search(
    "test", model_auth={"hf": {"token": "<your-hf-token>"}}
)
Example 2: Load from a public URL
You can also provide a zip file containing all the necessary files for the model and load it in Marqo from a public URL. The zip file should contain the model checkpoint, tokenizer, configurations, etc. Ensure these files are in the root directory of the zip file.


# A model that supports text and audio modalities
settings = {
    "model": "your-own-languagebind-model",
    "treatUrlsAndPointersAsMedia": True,
    "modelProperties": {
        "dimensions": 768,
        "type": "languagebind",
        "supported_modalities": ["text", "audio"],
        "modelLocation": {
            "audio": {
                "url": "https://opensource-languagebind-models.s3.us-east-1.amazonaws.com/LanguageBind_Audio_FT.zip"
            },
        },
    },
    "normalizeEmbeddings": True,
}

mq.create_index("test-custom-languagebind-model", settings_dict=settings)
Example 3: Load from s3 bucket
You can load your model from a public or private S3 bucket:


settings = {
    "model": "your-own-languagebind-model",
    "treatUrlsAndPointersAsMedia": True,
    "modelProperties": {
        "dimensions": 768,
        "type": "languagebind",
        "supportedModalities": ["text", "audio", "video", "image"],
        "modelLocation": {
            "image": {
                "s3": {
                    "Bucket": "opensource-languagebind-models",
                    "Key": "LanguageBind_Image.zip",
                }
            },
            "audio": {
                "s3": {
                    "Bucket": "opensource-languagebind-models",
                    "Key": "LanguageBind_Audio_FT.zip",
                }
            },
            "video": {
                "s3": {
                    "Bucket": "opensource-languagebind-models",
                    "Key": "LanguageBind_Video_V1.5_FT.zip",
                }
            },
        },
    },
    "normalizeEmbeddings": True,
}

mq.create_index("test-custom-languagebind-model", settings_dict=settings)
You need to provide AWS credentials or ensure your host has the necessary permissions to access the bucket. You can provide the authentication information when you search or add documents to the index:


model_auth = {
    "s3": {
        "aws_access_key_id": "<your-access-key-id>",
        "aws_secret_access_key": "<your-secret-access-key>",
    }
}

mq.index("test-custom-languagebind-model").search("test", model_auth=model_auth)
Preloading your model
There may be cases wherein you want to preload (or prewarm, in other terms) your model before using it to index. This can be done by adding your model (with model and modelProperties) to the list of models on startup in your Marqo configuration.

The syntax for this can be found in Configuring preloaded models

---

Choosing a model for Marqo
This guide will explain tradeoffs and differences between Marqo's supported embedding models. See our blog post, Benchmarking Models for Multimodal Search and our Hugging Face space for more details.

The most fundamental component of any Marqo index is the embedding model used to represent the data. Marqo's embedding models take data like text or images as input and return an embedding (vector). This vector representation is indexed and searchable within Marqo by using approximate nearest neighbour algorithms along with a simililarty measure like L2 distance. You can use a varienty of different models to generate these vectors, depending on modality, language and performance requirements.

Text
The following models are supported by default (and primarily based on the excellent sbert and Hugging Face libraries and models).

Marqo/dunzhang-stella_en_400M_v5
sentence-transformers/all-MiniLM-L6-v1
sentence-transformers/all-MiniLM-L6-v2
sentence-transformers/all-MiniLM-L12-v2
sentence-transformers/all-mpnet-base-v1
sentence-transformers/all-mpnet-base-v2
sentence-transformers/stsb-xlm-r-multilingual
flax-sentence-embeddings/all_datasets_v3_MiniLM-L12
flax-sentence-embeddings/all_datasets_v3_MiniLM-L6
flax-sentence-embeddings/all_datasets_v4_MiniLM-L12
flax-sentence-embeddings/all_datasets_v4_MiniLM-L6
flax-sentence-embeddings/all_datasets_v3_mpnet-base
flax-sentence-embeddings/all_datasets_v4_mpnet-base
hf/e5-small
hf/e5-base
hf/e5-large
hf/e5-small-unsupervised
hf/e5-base-unsupervised
hf/e5-large-unsupervised
hf/e5-small-v2
hf/e5-base-v2
hf/e5-large-v2
hf/bge-small-en-v1.5
hf/bge-base-en-v1.5
hf/bge-large-en-v1.5
hf/bge-small-zh-v1.5
hf/bge-base-zh-v1.5
hf/bge-large-zh-v1.5
hf/multilingual-e5-small
hf/multilingual-e5-base
hf/multilingual-e5-large
hf/multilingual-e5-large-instruct
hf/GIST-large-Embedding-v0
hf/snowflake-arctic-embed-m
hf/snowflake-arctic-embed-m-v1.5
hf/snowflake-arctic-embed-l
hf/ember-v1
These models can be selected when creating the index and are illustrated by the example below:


# Import Marqo and create a client
settings = {
    "treatUrlsAndPointersAsImages": False,
    "model": "flax-sentence-embeddings/all_datasets_v4_MiniLM-L6",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-index", settings_dict=settings)
The model field is the pertinent field for selecting the model to use. Note that once an index has been created and a model has been selected, the model cannot be changed. A new index would need to be created with the alternative model.

The model will be applied to all relevant fields. Field-specific settings which allow different models to be applied to different fields is not currently supported but will be coming soon (and contributions are always welcome).

Currently, Marqo adds prefixes by default to e5 model queries. These are trained on data with prefixes, so adding those same prefixes to text chunks before embedding improves the quality of the embeddings. The default prefix for queries is "query: " and for documents, "passage: ". For more information, refer to the model card here

Although use case specific, a good starting point is the model flax-sentence-embeddings/all_datasets_v4_MiniLM-L6. It provides a good compromise between speed and relevancy. The model flax-sentence-embeddings/all_datasets_v4_mpnet-base provides the best relevancy (in general).

Images
The models that are used for vectorizing images come from CLIP. We support different implementations, including models from OpenAI, open clip, and our state-of-the-art models Marqo FashionCLIP.

Marqo FashionCLIP
Marqo-FashionCLIP & Marqo-FashionSigLIP are two new state-of-the-art multimodal models for search and recommendations in the fashion domain. Both models can produce embeddings for both text and images that can then be used in downstream search and recommendations applications. See the model release article on the Marqo blog for more details.

Marqo/marqo-fashionSigLIP
Marqo/marqo-fashionCLIP

index_settings = {
    "model": "Marqo/marqo-fashionCLIP",
    "treatUrlsAndPointersAsImages": True,
    "type": "unstructured",
}

mq.create_index("my", settings_dict=index_settings)
OpenAI
RN50
RN101
RN50x4
RN50x16
RN50x64
ViT-B/32
ViT-B/16
ViT-L/14
ViT-L/14@336px
Although use case specific, a good starting point is the model ViT-B/16. It provides a good compromise between speed and relevancy. The models open_clip/ViT-B-32/laion2b_s34b_b79k and ViT-L/14@336px provides the best relevancy (in general) but are typically slower.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-index", settings_dict=settings)
OpenAI-float16
Some OpenAI CLIP models can be implemented in float16, ONLY when cuda device is available. This can largely increase the speed with minor loss to accuracy. In our tests the inference latency is reduced by 50% (device dependent). Available models are:

fp16/ViT-L/14
fp16/ViT-B/32
fp16/ViT-B/16
You can load the model with:


settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-index", settings_dict=settings)
OpenCLIP
open_clip/RN101-quickgelu/openai
open_clip/RN101-quickgelu/yfcc15m
open_clip/RN101/openai
open_clip/RN101/yfcc15m
open_clip/RN50-quickgelu/cc12m
open_clip/RN50-quickgelu/openai
open_clip/RN50-quickgelu/yfcc15m
open_clip/RN50/cc12m
open_clip/RN50/openai
open_clip/RN50/yfcc15m
open_clip/RN50x16/openai
open_clip/RN50x4/openai
open_clip/RN50x64/openai

open_clip/ViT-B-16-plus-240/laion400m_e31

open_clip/ViT-B-16-plus-240/laion400m_e32
open_clip/ViT-B-16/laion2b_s34b_b88k
open_clip/ViT-B-16/laion400m_e31
open_clip/ViT-B-16/laion400m_e32
open_clip/ViT-B-16/openai
open_clip/ViT-B-16-SigLIP/webli
open_clip/ViT-B-16-SigLIP-256/webli
open_clip/ViT-B-16-SigLIP-384/webli
open_clip/ViT-B-16-SigLIP-512/webli
open_clip/ViT-B-16-quickgelu/metaclip_fullcc
open_clip/ViT-B-32-quickgelu/laion400m_e31
open_clip/ViT-B-32-quickgelu/laion400m_e32
open_clip/ViT-B-32-quickgelu/openai
open_clip/ViT-B-32/laion2b_e16
open_clip/ViT-B-32/laion2b_s34b_b79k
open_clip/ViT-B-32/laion400m_e31
open_clip/ViT-B-32/laion400m_e32
open_clip/ViT-B-32/openai
open_clip/ViT-B-32-256/datacomp_s34b_b86k

open_clip/ViT-H-14/laion2b_s32b_b79k

open_clip/ViT-H-14-quickgelu/dfn5b
open_clip/ViT-H-14-378-quickgelu/dfn5b

open_clip/ViT-L-14-336/openai

open_clip/ViT-L-14/laion2b_s32b_b82k
open_clip/ViT-L-14/laion400m_e31
open_clip/ViT-L-14/laion400m_e32
open_clip/ViT-L-14/openai
open_clip/ViT-L-14-quickgelu/dfn2b
open_clip/ViT-L-14-CLIPA-336/datacomp1b
open_clip/ViT-L-16-SigLIP-256/webli
open_clip/ViT-L-16-SigLIP-384/webli

open_clip/ViT-bigG-14/laion2b_s39b_b160k

open_clip/ViT-g-14/laion2b_s12b_b42k
open_clip/ViT-g-14/laion2b_s34b_b88k
open_clip/ViT-SO400M-14-SigLIP-384/webli

open_clip/coca_ViT-B-32/laion2b_s13b_b90k

open_clip/coca_ViT-B-32/mscoco_finetuned_laion2b_s13b_b90k
open_clip/coca_ViT-L-14/laion2b_s13b_b90k
open_clip/coca_ViT-L-14/mscoco_finetuned_laion2b_s13b_b90k

open_clip/convnext_base/laion400m_s13b_b51k

open_clip/convnext_base_w/laion2b_s13b_b82k
open_clip/convnext_base_w/laion2b_s13b_b82k_augreg
open_clip/convnext_base_w/laion_aesthetic_s13b_b82k
open_clip/convnext_base_w_320/laion_aesthetic_s13b_b82k
open_clip/convnext_base_w_320/laion_aesthetic_s13b_b82k_augreg
open_clip/convnext_large_d/laion2b_s26b_b102k_augreg
open_clip/convnext_large_d_320/laion2b_s29b_b131k_ft
open_clip/convnext_large_d_320/laion2b_s29b_b131k_ft_soup
open_clip/convnext_xxlarge/laion2b_s34b_b82k_augreg
open_clip/convnext_xxlarge/laion2b_s34b_b82k_augreg_rewind
open_clip/convnext_xxlarge/laion2b_s34b_b82k_augreg_soup

open_clip/roberta-ViT-B-32/laion2b_s12b_b32k

open_clip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k
open_clip/xlm-roberta-large-ViT-H-14/frozen_laion5b_s13b_b90k

open_clip/EVA02-L-14-336/merged2b_s6b_b61k

open_clip/EVA02-L-14/merged2b_s4b_b131k
open_clip/EVA02-B-16/merged2b_s8b_b131k
Like the OpenAI based models, the larger ViT based models typically perform better. For example, open_clip/ViT-H-14/laion2b_s32b_b79k is the best model for relevency (in general) and surpasses even the best models from OpenAI.

The names of the OpenCLIP models are in the format of "implementation source / model name / pretrained dataset". The detailed configurations of models can be found here.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "open_clip/ViT-H-14/laion2b_s32b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-index", settings_dict=settings)
Multilingual CLIP
Marqo supports multilingual CLIP models that are capable up to 200 languages. You can use the following models and achieve multimodal search in your preferred language:

visheratin/nllb-clip-base-siglip
visheratin/nllb-siglip-mrl-base
visheratin/nllb-clip-large-siglip
visheratin/nllb-siglip-mrl-large
These models can be specified at index creation time.

Note that multilingual clip models are very large models (approximately 6GB) therefore a cuda device is highly recommended.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "visheratin/nllb-siglip-mrl-base",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-index", settings_dict=settings)
Video and Audio Models
Marqo supports multimodal models for video, audio, image, and text documents using LanguageBind (see model card here). You can use the following modality combinations for your index

Model Name  Supported Modalities  Required Memory
LanguageBind/Video_V1.5_FT_Audio_FT_Image Video, Audio, Image, Text 8 GB
LanguageBind/Video_V1.5_FT_Audio_FT Video, Audio, Text  5 GB
LanguageBind/Video_V1.5_FT_Image  Video, Image, Text  5 GB
LanguageBind/Audio_FT_Image Audio, Image, Text  5 GB
LanguageBind/Audio_FT Audio, Text 2 GB
LanguageBind/Video_V1.5_FT  Video, Text 2 GB
For models with more than 4 GB required for memory, please set the environment variables MARQO_MAX_CPU_MODEL_MEMORY or MARQO_MAX_CUDA_MODEL_MEMORY to the appropriate value in GB, depending on your device.

For these models, it is recommended to set MARQO_MEDIA_DOWNLOAD_THREAD_COUNT to 5 initially, increasing depending on your machine. Preprocessing of the media files is done right after downloading in parallel and in chunks, so this is a CPU-heavy process.

Generic CLIP Models
You can use your fine-tuned clip models with custom weights in Marqo. Depending on the framework you are using (we currently support model frameworks from openai clip and open_clip), you can use set up the index as:

Open_CLIP

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "generic-clip-test-model-1",
    "modelProperties": {
        "name": "ViT-B-32-quickgelu",
        "dimensions": 512,
        "url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_avg-8a00ab3c.pt",
        "type": "open_clip",
    },
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-generic-model-index", settings_dict=settings)
Openai CLIP

settings = {
    "treatUrlsAndPointersAsImages": True,
    "model": "generic-clip-test-model-2",
    "modelProperties": {
        "name": "ViT-B/32",
        "dimensions": 512,
        "url": "https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt",
        "type": "clip",
    },
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-generic-model-index", settings_dict=settings)
It is very important to set "treat_urls_and_pointers_as_images": True to enable the multimodal search. The model field is required and acts as an identifying alias to the model specified through modelProperties.

In modelProperties, the field name is to identify the model type. dimensions specifies the dimension of the output. type shows the framework you are using. You should also provide you custom model (checkpoint) by field url. You will need to serve your model and access it via a url. For more detailed instructions, please check here.

Advanced usage If Marqo is not running on Docker, models may be stored locally and referenced using a local file pointer. By default, Marqo running within Docker will not be able to access these.

Users should conscious of the different fields model and name. model acts as an identifying alias in Marqo (for generic models, you can choose your own). name, in this case, is used to identify the CLIP architecture from OpenAI or OpenCLIP

A table of all the required fields is listed below

Required Keys for modelProperties
Field Type  Description
name  String  Name of model in library. If the model is specified by modelProperties.model_location, then this parameter refers to the model architecture, for example open_clip/ViT-B-32/laion2b_s34b_b79k
dimensions  Integer Dimensions of the model
url String  The url of the custom model
type  String, "clip" or "open_clip" The framework of the model
Optional fields provide further flexibilities of generic models. These fields only works for models from open_clip as this framework provides more flexibilities.

Optional Keys for modelProperties
Field Type  Default value Description
jit Bool  False Whether to load this model in JIT mode.
precision String  "fp32"  The precison of the model. Optional values: "fp32" or "fp16"
tokenizer String  "clip"  The name of the tokenizer. We support hugging face tokenizer.
mean  Tuple (0.48145466, 0.4578275, 0.40821073) The mean of the image for normalization
std Tuple (0.26862954, 0.26130258, 0.27577711)  The std of the image for normalization
model_location  Dictionary  ""  The location of the model if it is not easily reachable by URL (for example a model hosted on a private Hugging Face and AWS S3 repos). See here for examples.
Generic SBERT Models
You can also use models that are not supported by default.


settings = {
    "treatUrlsAndPointersAsImages": False,
    "model": "unique-model-alias",
    "modelProperties": {
        "name": "sentence-transformers/multi-qa-MiniLM-L6-cos-v1",
        "dimensions": 384,
        "tokens": 128,
        "type": "hf",
    },
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-generic-model-index", settings_dict=settings)
The model field is required and acts as an identifying alias to the model specified through modelProperties. If a default model name is used in the name field, modelProperties will override the default model settings.

Currently, models hosted on huggingface model hub are supported. These models need to output embeddings and conform to either the sbert api or huggingface api. More options for custom models will be added shortly, including inference endpoints.

Required Keys for modelProperties
Name  Type  Description
name  String  Name of model in library. This is required unless modelProperties.model_location is specified.
dimensions  Integer Dimensions of model
type  String  Type of model loader. Must be set to "hf" for generic SBERT models.
Optional Keys for modelProperties
Search Parameter  Type  Default value Description
tokens  Integer 128 Number of tokens
model_location  Dictionary  ""  The location of the model if it is not easily reachable by URL (for example a model hosted on a private Hugging Face and AWS S3 repos). See here for examples.
No Model
You may want to use marqo to store and search upon vectors that you have already generated. In this case, you can create your index with no model. To do this, set model to the string "no_model" and define model_properties with "type": "no_model" and "dimensions" set to your desired vector size.

Note that for a no model index, you will not be able to vectorise any documents or search queries. To add documents, use the custom_vector feature, and to search, use the context parameter with no q defined.


# Suppose you want to create an index with 384 dimensions
settings = {
    "treatUrlsAndPointersAsImages": False,
    "model": "no_model",
    "modelProperties": {
        "dimensions": 384,  # Set the dimensions of the vectors
        "type": "no_model",  # This is required
    },
}
response = mq.create_index("my-no-model-index", settings_dict=settings)
Required Keys for modelProperties
Name  Type  Description
dimensions  Integer Dimensions of the index
type  String  Type of model loader. Must be set to "no_model"
Other media types
At the moment only text and images are supported. Other media types and custom media types will be supported soon.

---

Reranking models
Reranking models are ones that are applied after an initial set of results are retrieved. The reranking model takes in the list of results and reranks them according to additional criteria or constraints.

Images
Image reranking works across fields that contain image pointers or references. They do not work across text fields.

OWL-ViT
OWL-ViT is a reranking model that allows localisation and can be used with both tensor and lexical search.

Description
OWL-ViT is a Vision Transformer for Open-World Localization. It is an object detector that can be used to localise objects within an image. OWL-ViT differs from other object detectors in that it is: 1) open vocabulary and can detect any class, and 2) performs conditional localisation by taking context into account when determining the location of the object. This permits its use as a reranking model by using the query as the context to condition the localisation on. When used as a reranker, the query and candidate images are fed into OWL-ViT, proposals for the context are generated and the final ranking is determined from the highest scoring proposal per image. In addition, the location (x1, y1, x2, y2) of the highest scoring proposal is returned as a highlight in the _highlights field of the response. The coordinates are on the scale of the original image and the coordinate system is the same that is used in PIL - the top-left corner is represented by (0,0).

Examples

# Create a structured index
import marqo

settings = {
    "type": "structured",
    "vectorNumericType": "float",
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
    "textPreprocessing": {
        "splitLength": 2,
        "splitOverlap": 0,
        "splitMethod": "sentence",
    },
    "imagePreprocessing": {"patchMethod": None},
    "allFields": [
        {"name": "Title", "type": "text", "features": ["lexical_search"]},
        {"name": "Description", "type": "text", "features": ["lexical_search"]},
        {"name": "image_location", "type": "image_pointer"},
    ],
    "tensorFields": ["image_location", "Description"],
    "annParameters": {
        "spaceType": "prenormalized-angular",
        "parameters": {"efConstruction": 512, "m": 16},
    },
}

mq = marqo.Client(url="http://localhost:8882")

mq.create_index("my-test-structured-index", settings_dict=settings)
If we add the following documents to the index:


mq.index("my-test-structured-index").add_documents(
    documents=[
        {
            "Title": "The Travels of Marco Polo",
            "Description": "A 13th-century travelogue describing Polo's travels",
            "image_location": "https://raw.githubusercontent.com/marqo-ai/marqo-api-tests/mainline/assets/ai_hippo_statue.png",
        },
        {
            "Title": "Extravehicular Mobility Unit (EMU)",
            "Description": "The EMU is a spacesuit that provides environmental protection",
            "image_location": "https://raw.githubusercontent.com/marqo-ai/marqo-api-tests/mainline/assets/ai_hippo_realistic.png",
            "_id": "article_591",
        },
    ]
)
then a search invoking OWL-ViT as a reranker is performed by passing through the model name to reranker. searchable_attributes should be specified explicitly with the image field to rerank over appearing first in the list. For example,


response = mq.index("my-test-structured-index").search(
    "space suit",
    searchable_attributes=["image_location"],
    reranker="owl/ViT-B/32",
)
Limitations
The current limitations of OWL-ViT are that it requires an image field to rerank over and currently only a text query can be used. Support for image based queries will be added shortly.

Available models
"owl/ViT-B/32"
"owl/ViT-B/16"
"owl/ViT-L/14"
Text
Coming soon!

---

Images
Marqo supports pre-processing of images in several ways. The pre-processing allows the image to be broken into sub images (or patches) that can be considered to represent regions of interest. These patches are indexed along with the original image by cropping the image to the proposed region. Each image can have multiple patches associated with it. The patches can be searched alongside the original image. This can allow for better search results as well as providing localisation (and is akin to highlighting in text based search). By default no pre-processing will be performed but it can be easily specified when setting up the index. The method needs to be specified at indexing time and cannot be changed. If another method is required then a new index should be created.

Heuristic based patching
Heuristic based patching relies on a simple heuristic scheme to provide the regions of interest.

Simple

settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "simple"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
The settings above will use a 'simple' chunking scheme by splitting the image into smaller patches. This means that at indexing time, not only will the original image be indexed and searchable, but sub-image patches are also generated and indexed.

The sub-image patches are generated by breaking the image into a 3x3 grid. This means that the original image now has child images which consist of the sub-images. At searching time, not only is the original image searched over but so are all the sub-images.

Overlap

settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "overlap"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
The settings above will use an extension of the 'simple' chunking scheme above to also include patches which overlap the grid.

Advanced methods
The advanced methods use learned models for proposing regions of the image to patch. Both supervised and unsupervised methods for region proposal are supported. These methods will continue to evolve and any suggested improvements or feature requests can be made as an issue or PR on GitHub.

Faster-rcnn
Setting using a patch method of frcnn will invoke PyTorch's pretrained faster-rcnn model.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "frcnn"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
Marqo-yolo
Setting using a patch method of marqo-yolo will invoke a pretrained yolox model that was trained on the LVIS dataset. The class agnostic scores ("objectness") are used for the nms.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "marqo-yolo"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
DINO v1
Setting using a patch method of dino-v1 will invoke a pretrained DINO vision transformer model (currently ViT small 16). The bounding boxes are determined from the (summed) attention maps using contours before being passed through nms. This method produces fewer boxes than the dino-v2 method described below.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "dino-v1"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
DINO v2
Setting using a patch method of dino-v2 will invoke a pretrained DINO vision transformer model (currently ViT small 16). The bounding boxes are determined from the individual attention maps using contours before being passed through nms.


settings = {
    "treatUrlsAndPointersAsImages": True,
    "imagePreprocessing": {"patchMethod": "dino-v2"},
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "normalizeEmbeddings": True,
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)

---

Text
Marqo supports pre-processing of text. Currently, the text pre-processing consists of (optionally) chunking pieces of text into shorter pieces.

Text chunking

settings = {
    "textPreprocessing": {
        "splitLength": 2,
        "splitOverlap": 0,
        "splitMethod": "sentence",
    },
}
response = mq.create_index("my-multimodal-index", settings_dict=settings)
The settings above will split text (in any field) into sentences (split_method) of length 2 (split_length) with no overlap (split_overlap) between consecutive chunks of text. For example, if we had a document given by the following python dictionary;


document = {
    "title": "This is a short title.",
    "description": "This field is for a description. In this example, it contains some text. And some more. And even more!",
    "other": 100,
}
Then each string field ("title" and "description") will be chunked according to the settings provided above. The "title" will remain unchanged "This is a short title." as it is only a single sentence. The "description" text will go from "This field is for a description. In this example, it contains some text. And some more. And even more! to being ["This field is for a description. In this example, it contains some text., "And some more. And even more!"] as the length is 2 sentences with 0 overlap between consecutive sentences. Other methods of splitting are also available -"character" which performs the chunking based on characters, "word" which performs it based on words, and "passage" which performs it based on passages of text (denoted by a \n\n). Currently the settings will be applied to all fields in the same way. Field specific settings will be added soon.

---

Troubleshooting
If you are having trouble running Marqo then see below for potential solutions. If these do not work then you can get help by raising an issue or by joining our Slack community. See Community and Support for further details.

RAM and VRAM
Depending on the settings used, Marqo can consume more memory (RAM or VRAM) than is available. Symptoms of exceeding system memory can include abrupt termination of Marqo with the message Killed. To reduce memory consumption, the following actions can be performed:

Do not load any models at startup. This will avoid pre-loading models and will reduce the memory footprint of Marqo. This can be achieved by adding the environment variable - -e MARQO_MODELS_TO_PRELOAD='[]' - to the Marqo docker run command. See Configuring preloaded models
Batch or request sizes are too large
Large batches of documents when indexing can degrade performance or reduce throughput, including when using Marqo Cloud. It is recommended to send documents for indexing in batches. The python client makes this easy to do by setting a client_batch_size parameter when indexing. Having a larger batch size reduces the communication overhead from sending more requests.

Marqo Cloud
It is recommended to not send very large batch sizes as timeout limits (for Marqo Cloud) can occur. The exact batch size will vary greatly on the document structure and models used. For typical scenarios client_batch_size=64 is a good starting point. Alternatively, choose batch sizes that keep indexing requests to < 10 seconds. The times can be observed in the log outputs of the python client.


response = mq.index("my-index").add_documents(
    documents, client_batch_size=64, tensor_fields=["myField"]
)
Documents are too large
Very large documents may cause issues during indexing due to request limit sizes. There are several options to remediate this:

Break large documents into multiple smaller documents.
Modify the MARQO_MAX_DOC_BYTES environment variable described in the section Configuring usage limits.
Running Marqo on older AMD64 processors
Marqo's built-in vector store is compiled to exploit advanced CPU instructions that may not be available on older AMD64 processors. If your processor does not support the required instructions, you may observe the following warning when starting Marqo:


Warning: Failed to configure local vector store. Marqo may not function correctly
followed by other errors due to the vector store not being available. To get around this, you can run the vector store externally using a Docker image built for older processors:

docker run --detach --name vespa -p 8080:8080 -p 19071:19071 -p 2181:2181 vespaengine/vespa-generic-intel-x86_64
Now deploy an initial application package to configure the vector store. An application package suitable for local runs is provided in the Marqo repository:

git clone https://github.com/marqo-ai/marqo.git
cd marqo
(cd scripts/vespa_local && zip -r - * | curl --header "Content-Type:application/zip" --data-binary @- http://localhost:19071/application/v2/tenant/default/prepareandactivate)
You can verify that the vector store has been set up correctly by visiting http://localhost:8080 in your browser. The vector store can take a few minutes to start responding after the initial configuration.
With your external vector store ready, you can now run Marqo configured to use it:


docker run --name marqo -p 8882:8882 --add-host host.docker.internal:host-gateway \
    -e "VESPA_CONFIG_URL=http://host.docker.internal:19071" \
    -e "VESPA_QUERY_URL=http://host.docker.internal:8080" \
    -e "VESPA_DOCUMENT_URL=http://host.docker.internal:8080" \
    marqoai/marqo
Image search results are poor
If results obtained when searching across an index of images are poor, it could be due to misconfigured index settings. You must make sure that URLS and pointers are being properly downloaded. Specifically, the following setting needs to be enabled on index creation:

treat_urls_and_pointers_as_images=True (if using Python client create_index parameter).

treatUrlsAndPointersAsImages: True (if using the REST API or settings_dict).

To check the settings of the index, see here. To see the available image models, refer to the "Image" section of the models reference in the documentation.

Long response times when adding documents to unstructured indexes
In Marqo 2.13.0, we introduced an improvement to the unstructured index that supports searchable attributes. This enhancement requires dynamic updates to the index settings, which are stored as a configuration file in Vespa. However, Vespa servers prior to version 8.396.18 contain a bug that can slow down the distribution of this configuration file, especially in clusters with multiple config servers. If you are hosting the Vespa server yourself, upgrading to version 8.396.18 or later will help mitigate this issue.

In addition, you can set the environment variable VESPA_FILE_DOWNLOAD_BACKOFF_INITIAL_TIME_MS to a low value, such as 100, when running Vespa clusters. This adjustment can significantly reduce the overall time required for configuration file distribution.

Other Issues
If your problem cannot be solved by these suggestions then you can get additional support from Slack or Github. See Community and Support for details.

---

Technical Best Practices
Reminder
Don't forget to check for open issues and upcoming enhancements on Marqo's GitHub repository for the latest updates on features and capabilities.

Resolving Connection Issues with Docker Containers
Problem:
When running the Marqo service on an M1 Mac, users may encounter issues where the Docker container is unable to connect to the host to add documents, particularly when attempting to access images with a message like "cannot resolve the host."

Solution:
If you're facing connectivity issues between your Docker container and the host, especially for image indexing with marqo:

Replace host.docker.internal with Localhost IP: Instead of using http://host.docker.internal:8222, use the localhost IP address (e.g., http://192.168.1.254:8222) to allow the container to access images on the host. This can often resolve the issue where the container cannot connect to the host.
Inspect Errors for Insights: If there are errors during document addition, inspect the error messages by indexing small batches and printing out the response. This will help you identify and troubleshoot specific issues with document indexing.
Consider Model Size: On M1 Macs, which lack GPU support for CUDA, opt for smaller models (like open_clip/ViT-B-32/laion2b_s34b_b79k) to avoid performance issues during the indexing of images.
Update Docker Run Command: Modify the Docker run command to include your gateway IP address, which can be found using docker network inspect bridge. This change can enhance the Docker container's ability to communicate with the host. Monitor Initial Batches: Be aware that the first batch might be slow due to model downloading times. Ensure that the model is pre-downloaded if possible to speed up the process.
Use Correct URLs: Verify the URLs you're using to access the images. They should be reachable from the container. Alternatives like http://0.0.0.0:8222/image.jpg or http://localhost:8222/image.jpg might work if the direct IP doesn't.
EC2 Instance Storage Management
Problem:
Users encounter a 507 Insufficient Storage error on a Linux EC2 instance despite having ample space.

Solution:

Verify the actual disk usage via the df -h command to understand how much space is truly available and what is being utilized.

Be aware that certain applications, like Docker, can consume significant disk space. Old Docker volumes, in particular, can accumulate and take up space.

Clean up or remove unnecessary data and volumes to free up space, especially when the root filesystem is reaching its capacity.

Image Search Issues in Marqo
Problem:
Difficulties in performing image searches using local paths or URLs, resulting in MarqoWebError messages.

Solution:

Ensure that Marqo has access to the images you're trying to use. If using a local path, Marqo may not recognize it due to access restrictions.

Host the image on an HTTP server and use the URL for the search query, as Marqo currently only supports image searching via URL.

For reverse image search issues, it's important to know that Marqo currently does not support image search via local paths in Docker. This functionality is on the roadmap for future releases, as indicated by the open GitHub issue. Meanwhile, continue to use image URLs for search queries.

Error when loading a custom model into marqo
Problem:
Loading a custom model from Hugging Face into Marqo can sometimes result in errors if the model isn't supported by the existing frameworks (open_clip or clip).

Solution:
If you encounter an error when loading a custom model, like ValueError: You have to specify pixel_values, it may mean that Marqo can't load the model as-is. A workaround could be to convert the model's .bin file to a .pt (PyTorch) file and attempt to load it again using open_clip.

Remember to update your settings with the new file path and ensure treatUrlsAndPointersAsImages is set to True for image handling. If the model still isn't supported, stay tuned for future updates where more models may be integrated based on user requests.

Choosing the Best Data Structure for Vector Search
Problem:
When setting up a data schema for products, you may wonder which structure is more effective for vector search: a single key-value (k-v) pair for multiple attributes or multiple k-v pairs for each attribute.

Solution:

Single k-v pairs with a string of comma-separated values (e.g., "Tags: blue, patterned, cotton, elegant") are more efficient for vector search.
Benefits:
Contextual Relevance: A single string provides more context, leading to better recall performance in search results.
Resource Efficiency: Only one tensor field and vector are generated, which conserves RAM and can speed up search times.
Technical Note: Remember, lists of strings are for non-tensor fields used only in filtering, not vector search.
Managing Document Deletion in Marqo
Problem:
You've split a large document into smaller sub-documents in Marqo, each with an incremental ID based on the title, and now you need to delete all sub-documents associated with a specific title.

Solution:
While Marqo allows for batch deletion by ID, this can be cumbersome when dealing with multiple sub-documents. Here are some tips to streamline the process:

Unique ID Generation: Instead of incremental IDs, consider creating a unique hash for each title. This makes tracking and deletion more straightforward, as every sub-document related to a title would share this unique identifier.
External Tracking: Keep an external record of document IDs. This offloads the complexity from Marqo and simplifies the deletion process, as you would have a ready list of IDs to remove.
Over Deletion Method: If you're aware of the maximum number of sub-documents, you could attempt to delete a range (e.g., title_0 to title_20). Marqo will skip non-existing IDs, so there's no harm in overshooting the actual count. However, this method is less efficient for large numbers of documents or a massive index.
Post-Processing: Rely on Marqo's internal chunking. Use search highlights to identify the relevant section of the text and then manually extract the required context, such as adding sentences before and after the highlighted section. This approach is beneficial when you're dealing with larger chunks of text and need only a single match per document.
Enhancing Product Categorization Accuracy with Marqo Search Parameters
Problem:
Inconsistent search results when mapping products to categories.

Solution:

When using Marqo for product mapping, you might find that increasing the limit parameter improves the search score. This is because Marqo’s limit is linked to the k parameter in vector search, which determines the number of nearest documents considered during the search process. A higher k means a better chance of finding the true closest match. If your initial search misses a more accurate document, increasing the limit may help you retrieve it. To enhance search stability and accuracy, try setting a higher limit, such as 100, and then filter out any unnecessary results afterward. This way, you ensure no potential matches are overlooked, leading to more precise categorization.

Optimizing Marqo Docker Images for Minimal Resource Usage
Problem:
Users with internet connectivity issues or limited system resources need a lightweight Marqo Docker image that consumes minimal memory.

Solution:

For those looking to deploy a text-encoding model with Marqo in a resource-constrained environment, we recommend the hf/e5-small-v2 model, which is quite lean at 134MB.

To build a light Marqo Docker image locally, please refer to the instructions provided in our GitHub repository, under the section 'Option C: Build and run the Marqo as a Docker'.

For existing Marqo Docker container instances that seem to be memory-intensive (e.g., around 2.5 GBs), it's possible to reduce memory usage by excluding unnecessary components. To achieve this, ensure that Marqo starts with preloaded models set to an empty list ([]). This configuration will prevent loading image models or any other models that are not essential for your specific use case, thereby conserving memory. Detailed guidance on this configuration can be found in the 'Configuring Preloaded Models' section of our advanced usage documentation for version 1.4.0.

---


Solution Best Practices
Pre-Filtering Accuracy in Vector Searches
Problem:
Users of our vector database sometimes face challenges with pre-filtering, particularly when trying to distinguish between closely related search terms. For example, a search for "Northern Africa" might erroneously pull up results for "Southern Africa" because traditional vector search methods can't effectively differentiate between the two.

Solution:

For those looking to refine their search capabilities, we suggest considering an external fuzzy filtering solution such as the rapidfuzz library to enhance the precision of search results. By creating a synonym list relevant to your domain, you can ensure that searches are not only comprehensive but also contextually accurate. See this link for more information

Selecting the Right AWS EC2 Instance for Marqo Docker Containers
Problem:
Running Marqo Docker containers on AWS EC2 instances may fail due to insufficient resources or compatibility issues with certain Ubuntu versions and cgroup configurations.

Solution:

Ensure your AWS EC2 instance is appropriately sized and your Ubuntu version is compatible with Marqo's requirements. Start with an instance that has at least 2 vCPUs and 8GB RAM, like the t2.large. Marqo can struggle on smaller instances, like the t2.micro, due to limited memory and processing power.

Selecting Appropriate Models and Optimal Weights for Multimodal Search with Marqo
Problem:
Choosing the right models and determining the optimal balance between text and image weights for multimodal search indexing is crucial for performance but can be complex and nuanced.

Solution:

Follow these guidelines to streamline the selection of models and weights for multimodal searches using Marqo:

Understand Your Data: The nature of your data should guide the model and weight choice.
For text-rich content, prioritize text weights. An xlm b-32 model with a balanced or slightly text-favored weight distribution (e.g., 50/50 or 60/40 image/text) often yields the best results.
For data where images carry more information, like stock images, increase the image weight. Use the default model with a higher image weight proportion (80/20, 90/10, or 95/5 image/text).
Experiment with Weights: Start with sensible weight distributions based on past empirical results.
For image-focused data, begin with a 90/10 image/text ratio.
For text-focused data, a 60/40 image/text ratio is a good starting point.
Iterate and Optimize: Perform manual inspections and tweak the model and weights accordingly.
Look for qualities like clarity, relevance, and completeness in your search results to judge the effectiveness of your current setup.
Be prepared to swap models and adjust weights multiple times to hone in on the most effective configuration.
Customizing Search Result Presentation
Problem:
The style of my results doesn’t look how I want it to or it doesn’t match my brand/platform.

Solution:

One of the best ways to tailor your results is with prompting. Similar to how you can prompt LLMs you can prompt your queries (and hide this from an end user).

For example in large datasets you may have thousands of relevant items, many of which might have terrible photos. To surface only the higher images you could prefix all searches with “a high quality aesthetic photo of”. Or likewise you could add a negative query term such as “low quality photo, blurry, jpeg artefacts” with a weight of -0.5 to push these sorts of results away.

If you want results to match a style you can quite explicitly ask for them in a query prompt such as, “a stock photo of”, “an oil painting of”, “a cyberpunk neon depiction of”, etc.

The same concepts apply to text search as well.

Optimizing Document Tokenization for Search and Summarization
Problem:
Finding the right balance in tokenizing lengthy documents (30-50 pages) for efficient search and summarization when using Marqo and a Language Model (LLM).

Solution:

For precise search results, consider tokenizing your documents into smaller chunks, such as 1-3 sentences, and then add those as separate documents to Marqo. This increases the specificity of the embeddings. However, when broader context is needed, larger segments can capture the essence of concepts spread over multiple sentences.

A practical approach is to tokenize one paragraph (4-6 sentences) per document. Utilize Marqo's textPreprocessing feature to determine the text amount for each vector, starting with a split length of 3 and an overlap of 1 to capture a wider context. For more exact matches, a split length of 2 or 1 could be more effective.

Experiment with different models like the E5, which are adept at embedding longer text pieces (up to 512 tokens). Remember to start queries with "query: " for these models.

When working with LLMs for summarization, consider only sending them the _highlights from Marqo to stay within the token processing limit and ensure relevance. You can also expand the context around the highlights when feeding it to the LLM to balance between search specificity and the need for context in summarization.