[Feature Request] Relax `meta` block on field level

[urmichm]

Is your feature request related to a problem? Please describe

I'm always frustrated when I cannot add comments and explanation to my code :)
The index structure allows us to have _meta block in mappings.
Why not on (each) field level in properties?

Describe the solution you'd like

Allow to add a _meta or a description block to the fields

{
    "settings": {  ... },
    "mappings": {
        "properties": {
            "address_coordinates": {
                "type": "geo_point",
                "_meta": {
                       "description": "Geo Point in form lat,lan"
                }
            }, ........
        },
        "_meta": {
            "application": "My Beautiful Index",
            ...
        }
    }
}

Update

Many fields (not all) already support the meta block. However, this block is under heavy restrictions, which prohibit users from adding descriptive information.
Based on discussions, it was decided to relax the restrictions. However, since not all the fields support the meta block, AND refactoring from meta to _meta will result in Breaking Changes - the field will keep its name meta (without underscore)

[andrross]

Catch All Triage - 1 2 3

@jainankitk @msfroh What do you think of this idea?

[urmichm]

I am open to help and/or create a Pull Request of you find this feature useful

[msfroh]

This will inflate the size of the mappings and therefore the cluster state, which isn't great. On the other hand, it's up to users/administrators to decide how they want to use their cluster state bytes.

So, I'm okay with this. It's a strictly opt-in feature. @urmichm, I'm happy to review a PR.

My preferred approach is to store mappings in source control (as the source of truth) and comment on that. That is, the mappings in the Opensearch cluster are the output of a managed schema rollout.

[urmichm]

cc: @msfroh @jainankitk

There is a very similar existing field called meta, please see:

• MappedFieldType
• FieldMapper

However, this meta parameter allows only String-to-String values: metaParam: Map<String,String> in ParametrizedFieldMapper

I believe this meta block is not indexed and simply used for documentation, as it is stored and returned (correct me if i am wrong).

Therefore, I would like to ask:

• Why is this field called meta, rather than _meta as in Mapping, may we rename it for consistency?
• If this field is used for documentation, may we relax its type to be Map<String, Object> as in Mapping?

[urmichm]

TypeParsers adds restrictions, some of which are:

• [meta] can't have more than 5 entries Line 89
• [meta] keys can't be longer than 20 chars Line 95
• [meta] values can't be longer than 50 chars Line 104

Are these restriction under negotiations? Are them required due to other requirements?

[urmichm]

Following our discussion, I am leaving a few key points:

The meta block on its own brings lots of benefits, such as:

• it can provide essential support for the agents
• it adds field-level description, clarifying what each field means
• it can include a flag indicating whether a field is an array / multi-value (solr provides similar support to indicate the mullti-valued fields)
  • additionally, it could indicate if a field is nullable

Regarding the naming convention: _meta is preferable. However, the existing meta (without the underscore) would make switching to _meta a breaking change, so the migration should be gradual. We can start by supporting both versions and then progressively enforce the underscored form.

Existing restrictions (for example, the number of entries per meta block) could be made configurable.

Changing meta from Map<String, String> to Map<String, Object> to allow structures is still an open question, therefore, the discussion is open. In general, the consensus seems to be in favour of making this change.

cc: @epugh @msfroh @sandeshkr419

[urmichm]

During testing, I noticed that not all fields support meta
Could you explain why?

MapperTestCase has the supportMeta method, which is later overridden for some tests.

FlatObject with its FlatObjectFieldMapper does not support meta, is there a specific reason for this?

cc: @epugh @msfroh @sandeshkr419