Description
We expose a field parameter called "copy_to". This parameter is supposed to copy one field to another. The main purpose is for searching text where you typically want to search over a couple fields. See https://opensearch.org/docs/latest/field-types/mapping-parameters/copy-to/.
For vector fields, this does not work. For one thing, we do not support multi-fields, so its unclear how this would work, but I suspect that it would result in some kind of multi-field effect. For another thing, in parsing, we process tokens and go over them. However, it seems like copyTo will just parse the context and then copy it to the copy to context, and then call the parser again. See:
- https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/DocumentParser.java#L527-L528
- https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/DocumentParser.java#L900-L951
This works for text fields which do not skip the text value (See: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/TextFieldMapper.java#L1038). However, it does not work for vectors fields which we call next token for.
The error message ends up looking something like:
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [copyto_vector] of type [knn_vector] in document with id '1'. Preview of field's value: 'null'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [copyto_vector] of type [knn_vector] in document with id '1'. Preview of field's value: 'null'","caused_by":{"type":"x_content_parse_exception","reason":"[1:76] Failed to parse list: expecting START_ARRAY but got END_ARRAY"}},"status":400}
# With a mapping like:
"mappings" : {
"properties" : {
"copyto_vector" : {
"type" : "knn_vector",
"dimension" : 16
},
"test_vector" : {
"type" : "knn_vector",
"dimension" : 16,
"copy_to" : [
"copyto_vector"
]
}
}
},
Description
We expose a field parameter called "copy_to". This parameter is supposed to copy one field to another. The main purpose is for searching text where you typically want to search over a couple fields. See https://opensearch.org/docs/latest/field-types/mapping-parameters/copy-to/.
For vector fields, this does not work. For one thing, we do not support multi-fields, so its unclear how this would work, but I suspect that it would result in some kind of multi-field effect. For another thing, in parsing, we process tokens and go over them. However, it seems like copyTo will just parse the context and then copy it to the copy to context, and then call the parser again. See:
This works for text fields which do not skip the text value (See: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/TextFieldMapper.java#L1038). However, it does not work for vectors fields which we call next token for.
The error message ends up looking something like: