Otel Input Packages should be time series

[graphaelli]

When using otel input packages data streams are not set to time series mode.

To Reproduce:

1. Install an otel input package, eg NGINX OpenTelemetry Input Package
2. Ingest data

POST metrics-nginxreceiver.otel-test/_doc
{
    "foo": "bar"
}

3. Note mode:

GET metrics-nginxreceiver.otel-test/_settings?filter_path=*.settings.index.mode

# returns:
{}

Expected:

POST metrics-notnginxreceiver.otel-test/_doc
{
    "foo": "bar"
}

GET metrics-notnginxreceiver.otel-test/_settings?filter_path=*.settings.index.mode

# returns:
{
  ".ds-metrics-notnginxreceiver.otel-test-2026.01.31-000001": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  }
}

Data streams created via these inputs should be indistinguishable from those created without the input packages - in this case if the user is already using the nginx receiver and then installs the package, the datastream would have started as tsds and reverted to standard, which would break any content using ES|QL TS, among other impacts.

[github-actions]

Claude finished @graphaelli's task in 12m 23s —— View job

---

Summary

OTel input packages (like nginx_otel_input) are missing time series data stream (TSDS) configuration. When Fleet creates index templates for these packages, they override Elasticsearch's default metrics-otel@template (which has index.mode: time_series) because Fleet templates have higher priority (150-200 vs 120 for the ES default).

Recommendation

Add elasticsearch.index_mode: time_series to OTel input packages that collect metrics. This setting is supported via package-spec and will be applied by Fleet when creating package policy index templates per kibana#145529.

Findings

Root Cause Analysis:

1. Elasticsearch's default OTel template (metrics-otel@template.yaml) includes index.mode: time_series and matches metrics-*.otel-* with priority 120

2. Fleet creates higher-priority templates (150-200) when input packages are installed. Per kibana#145529, input packages create index templates at package policy creation time

3. The input package spec supports elasticsearch.index_template settings (package-spec input manifest), but the current OTel input packages don't specify index_mode

4. Result: Fleet's package-specific templates take precedence over ES defaults, missing the TSDS configuration

Affected Packages (all type: input OTel packages with type: metrics policy templates):

• packages/nginx_input_otel/manifest.yml
• packages/httpcheck_otel/manifest.yml

Non-affected (logs packages, TSDS is excluded for logs per TSDS guidelines):

• packages/filelog_otel/manifest.yml

Detailed Action Plan

Step 1: Update nginx_input_otel manifest

Add the elasticsearch configuration to packages/nginx_input_otel/manifest.yml:

elasticsearch:
  index_template:
    settings:
      index.mode: time_series

This should be added at the package level (same level as owner:).

Step 2: Update httpcheck_otel manifest

Add the same configuration to packages/httpcheck_otel/manifest.yml:

elasticsearch:
  index_template:
    settings:
      index.mode: time_series

Step 3: Bump package versions

Update the version field in each manifest and add a changelog entry.

Step 4: Validate with elastic-package

cd packages/nginx_input_otel
elastic-package check
elastic-package build

Repeat for httpcheck_otel.

Step 5: Test the fix

1. Install the updated package
2. Create an integration policy
3. Ingest test data
4. Verify index mode:

GET metrics-nginxreceiver.otel-*/_settings?filter_path=*.settings.index.mode

Expected result should show "mode": "time_series".

Note: The filelog_otel package should NOT be modified - it's a logs package and TSDS is excluded for logs.

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
elastic/kibana	#145529 - Input only packages: Create index templates at policy creation	Core implementation that enables input packages to have elasticsearch settings
elastic/kibana	#146804 - Honor index_mode: time_series during installation	Fleet TSDS support implementation
elastic/package-spec	#311 - Support for TSDB in package spec	Package-spec TSDB support
elastic/elasticsearch	#112232 - time_series_dimension multi-value issue	Related OTel mappings issue with TSDB

Related Files

Repository	File	Relevance
elastic/integrations	packages/nginx_input_otel/manifest.yml	Primary file to modify for nginx OTel input
elastic/integrations	packages/httpcheck_otel/manifest.yml	Primary file to modify for HTTP check OTel input
elastic/elasticsearch	metrics-otel@template.yaml	ES default template with time_series mode
elastic/package-spec	input/manifest.spec.yml	Package spec for input packages

Related Documentation

Name	URL	Relevance
TSDS Guidelines	https://www.elastic.co/docs/extend/integrations/developer-tsds-guidelines	How to enable TSDS in integration packages
Fleet Advanced Indexing Features	https://www.elastic.co/guide/en/fleet/current/data-streams-advanced-features.html	How Fleet handles index templates
Time Series Index Settings	https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds-index-settings.html	ES TSDS configuration reference

---

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/ecosystem (Team:Ecosystem)

[github-actions]

Claude finished @graphaelli's task in 3m 22s —— View job

---

Recommendation

The package-spec for input packages (type: input) does not currently support elasticsearch.index_mode — it only supports elasticsearch.index_template. This is a spec limitation that needs to be addressed upstream before the fix can be applied to OTel input packages.

Action required: Open an issue in elastic/package-spec to add index_mode support to the input package manifest specification (similar to how data stream manifests support it).

Findings

Root Cause Analysis:

1. Elasticsearch's default OTel template (metrics-otel@template) includes index.mode: time_series and matches metrics-*.otel-* with priority 120

2. Fleet creates higher-priority templates (150-200) when input packages are installed, which override the ES defaults

3. Data stream manifests (used in integration packages) support elasticsearch.index_mode: "time_series" directly — see packages/zookeeper/data_stream/server/manifest.yml:16-17

4. Input package manifests only support elasticsearch.index_template per the package-spec input manifest:

   elasticsearch:
     description: Elasticsearch asset definitions
     type: object
     additionalProperties: false
     properties:
       index_template:
         $ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_template"

5. The elasticsearch_index_template definition does not include index.mode in its settings.index properties — only codec, mapping, and sort are supported

Affected Packages (all type: input OTel packages with type: metrics policy templates):

Package	Manifest
nginx_otel_input	packages/nginx_input_otel/manifest.yml
httpcheck_otel	packages/httpcheck_otel/manifest.yml

Not affected (logs package — TSDS is excluded for logs per TSDS guidelines):

• filelog_otel

Detailed Action Plan

Step 1: Open package-spec issue

Create an issue in elastic/package-spec requesting:

• Add index_mode property to the elasticsearch section in spec/input/manifest.spec.yml
• Should mirror the existing definition from data stream manifests at lines 609-616

Proposed change to spec/input/manifest.spec.yml:

elasticsearch:
  description: Elasticsearch asset definitions
  type: object
  additionalProperties: false
  properties:
    index_mode:
      description: |-
        Index mode to use. Index mode can be used to enable use case specific functionalities.
        This setting must be installed in the composable index template, not in the package component templates.
      type: string
      enum:
        - "time_series"
    index_template:
      $ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_template"

Step 2: Update OTel input packages (after package-spec is updated)

Once package-spec supports index_mode for input packages, add to each metrics input package manifest:

elasticsearch:
  index_mode: "time_series"

Step 3: Bump package versions and add changelog entries

For each affected package:

• Increment version in manifest.yml
• Add changelog entry describing TSDS enablement

Step 4: Validate

cd packages/nginx_input_otel
elastic-package check
elastic-package build

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
elastic/kibana	#145529 - Input only packages: Create index templates at policy creation	Core implementation that enables input packages to have elasticsearch settings
elastic/kibana	#146804 - Honor index_mode: time_series during installation	Fleet TSDS support implementation
elastic/package-spec	#311 - Support for TSDB in package spec	Original TSDB support addition (data streams only)
elastic/package-spec	#455 - Add support for elasticsearch index templates into input packages	Added elasticsearch.index_template to input packages

Related Files

Repository	File	Relevance
elastic/integrations	packages/nginx_input_otel/manifest.yml	Primary file to modify for nginx OTel input (after spec update)
elastic/integrations	packages/httpcheck_otel/manifest.yml	Primary file to modify for HTTP check OTel input (after spec update)
elastic/package-spec	spec/input/manifest.spec.yml	Input package spec that needs index_mode support
elastic/package-spec	spec/integration/data_stream/manifest.spec.yml	Reference implementation of index_mode for data streams

Related Documentation

Name	URL	Relevance
TSDS Guidelines	https://www.elastic.co/docs/extend/integrations/developer-tsds-guidelines	How to enable TSDS in integration packages
Package Spec Input Manifest	https://github.com/elastic/package-spec/blob/main/spec/input/manifest.spec.yml	Current input package specification
Time Series Index Settings	https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds-index-settings.html	ES TSDS configuration reference

---

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[github-actions]

Claude finished @graphaelli's task in 6m 15s —— View job

---

Recommendation

The package-spec for input packages (type: input) does not support elasticsearch.index_mode - it only supports elasticsearch.index_template. This is a spec limitation that needs to be addressed in elastic/package-spec before this fix can be applied to OTel input packages.

Required action: Open an issue in elastic/package-spec to add index_mode support to the input package manifest specification (mirroring data stream manifests).

Findings

Root Cause Analysis:

1. Elasticsearch's default OTel template (metrics-otel@template) includes index.mode: time_series and matches metrics-*.otel-* with priority 120

2. Fleet creates higher-priority templates (150-200) when input packages are installed, which override the ES defaults

3. Data stream manifests (in integration packages) support elasticsearch.index_mode: "time_series" directly — see packages/zookeeper/data_stream/server/manifest.yml:16-17

4. Input package manifests only support elasticsearch.index_template per the package-spec input manifest:

   elasticsearch:
     description: Elasticsearch asset definitions
     type: object
     additionalProperties: false
     properties:
       index_template:
         $ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_template"

5. The elasticsearch_index_template definition does not include index.mode in its settings.index properties — only codec, mapping, and sort are supported

Affected Packages (all type: input OTel packages with type: metrics policy templates):

Package	File
nginx_otel_input	packages/nginx_input_otel/manifest.yml
httpcheck_otel	packages/httpcheck_otel/manifest.yml

Not affected (logs package — TSDS is excluded for logs per TSDS guidelines):

• filelog_otel

Detailed Action Plan

Step 1: Open package-spec issue

Create an issue in elastic/package-spec requesting:

• Add index_mode property to the elasticsearch section in spec/input/manifest.spec.yml
• Should mirror the existing definition from data stream manifests at lines 609-616

Proposed change to spec/input/manifest.spec.yml:

elasticsearch:
  description: Elasticsearch asset definitions
  type: object
  additionalProperties: false
  properties:
    index_mode:
      description: |-
        Index mode to use. Index mode can be used to enable use case specific functionalities.
        This setting must be installed in the composable index template, not in the package component templates.
      type: string
      enum:
        - "time_series"
    index_template:
      $ref: "../integration/data_stream/manifest.spec.yml#/definitions/elasticsearch_index_template"

Step 2: Update OTel input packages (after package-spec is updated)

Once package-spec supports index_mode for input packages, add to each metrics input package manifest:

elasticsearch:
  index_mode: "time_series"

Step 3: Bump package versions and add changelog entries

For each affected package:

• Increment version in manifest.yml
• Add changelog entry describing TSDS enablement

Step 4: Validate

cd packages/nginx_input_otel
elastic-package check
elastic-package build

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
elastic/kibana	#145529 - Input only packages: Create index templates at policy creation	Core implementation that enables input packages to have elasticsearch settings
elastic/kibana	#146804 - Honor index_mode: time_series during installation	Fleet TSDS support implementation
elastic/kibana	#144530 - Add experimental toggle + support package spec field for TSDB	Fleet TSDB toggle support
elastic/package-spec	#311 - Support for TSDB in package spec	Original TSDB support addition (data streams only)

Related Files

Repository	File	Relevance
elastic/integrations	packages/nginx_input_otel/manifest.yml	Primary file to modify for nginx OTel input (after spec update)
elastic/integrations	packages/httpcheck_otel/manifest.yml	Primary file to modify for HTTP check OTel input (after spec update)
elastic/package-spec	spec/input/manifest.spec.yml	Input package spec that needs index_mode support
elastic/package-spec	spec/integration/data_stream/manifest.spec.yml	Reference implementation of index_mode for data streams
elastic/elasticsearch	metrics-otel@template.yaml	ES default template with time_series mode (priority 120)

Related Documentation

Name	URL	Relevance
TSDS Guidelines	https://www.elastic.co/docs/extend/integrations/developer-tsds-guidelines	How to enable TSDS in integration packages
Package Spec Input Manifest	https://github.com/elastic/package-spec/blob/main/spec/input/manifest.spec.yml	Current input package specification
Time Series Index Settings	https://www.elastic.co/docs/reference/elasticsearch/index-settings/time-series	ES TSDS configuration reference

---

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[jsoriano]

@claude I have some questions regarding the overall impact of the proposed change in package spec, also taking into account Fleet (the plugin in https://github.com/elastic/kibana/).

• Should this be allowed in input packages of all kinds and not only in the OTel ones? I tend to believe that this should be allowed in all input packages.
• Users can select the signal type in Fleet when creating a policy for input packages. Should Fleet control that this mode is only enabled on index templates for metrics data streams (and not to logs or traces)? Is it already doing it?
• If Fleet is controlling whether TSDB mode is enabled for metrics data streams, would it make sense for Fleet to enable TSDB by default for metrics data streams related to input packages?

[github-actions]

Claude finished @jsoriano's task in 4m 16s —— View job

---

Analysis of Fleet and Package-Spec Impact

Based on my investigation of the package-spec and Fleet (Kibana) codebases, here are detailed answers to your questions:

---

1. Should index_mode be allowed in all input packages, not just OTel ones?

Yes, this should be allowed in all input packages, not just OTel ones.

Reasoning:

• The current package-spec for input packages (spec/input/manifest.spec.yml) only supports elasticsearch.index_template, but not index_mode
• Data stream manifests (for integration packages) already support elasticsearch.index_mode: "time_series" (spec/integration/data_stream/manifest.spec.yml:609-617)
• There's no technical reason to restrict this to OTel inputs only — any input package that produces metrics data could benefit from TSDB mode
• Examples of non-OTel input packages that could use this include custom logs/metrics collectors

---

2. Should Fleet control that TSDB mode is only enabled for metrics data streams (not logs or traces)? Is it already doing this?

Fleet currently does NOT control this automatically. The index mode is determined by:

1. Package manifest setting - elasticsearch.index_mode in the data stream manifest
2. User-enabled experimental features - via experimental_data_stream_features with tsdb: true flag

Evidence from the Fleet codebase:

In template/install.ts:640-642:

const isIndexModeTimeSeries =
  dataStream.elasticsearch?.index_mode === 'time_series' ||
  !!experimentalDataStreamFeature?.features.tsdb;

Fleet applies TSDB settings based on these flags regardless of the data stream type (logs/metrics/traces). The only data-stream-type-aware logic is for selecting component templates:

In template/template.ts:148-157:

if (type === 'metrics') {
  if (isIndexModeTimeSeries) {
    return [STACK_COMPONENT_TEMPLATE_METRICS_TSDB_SETTINGS];
  }
  return [STACK_COMPONENT_TEMPLATE_METRICS_SETTINGS];
}

Recommendation: Fleet should add validation to prevent index_mode: time_series from being applied to logs data streams. Per the TSDS Guidelines, TSDB is excluded for logs. This could be implemented in Fleet when processing the package manifest.

---

3. Should Fleet enable TSDB by default for metrics data streams in input packages?

This is a policy decision with trade-offs. My analysis:

Arguments for enabling TSDB by default:

• Aligns with Elasticsearch's default metrics-otel@template which has index.mode: time_series at priority 120
• Provides consistency — users expect metrics to behave similarly whether from input packages or integration packages
• Reduces storage costs and improves query performance for metrics

Arguments against:

• Breaking change risk — existing input package policies would need migration
• Dimension cardinality issues — as noted in related issue elasticsearch#112232, TSDB with OTel data can have multi-value dimension problems
• Package authors should opt-in — allowing packages to declare index_mode gives flexibility

My recommendation:

Rather than Fleet enabling TSDB by default, the better approach is:

1. Update package-spec to allow elasticsearch.index_mode in input packages (applicable to all input types)
2. Package authors explicitly opt-in by adding index_mode: time_series to metrics input packages
3. Fleet validates that index_mode: time_series is only set on metrics (not logs/traces) data streams
4. Optionally: Fleet could warn users if a metrics input package doesn't have TSDB enabled, suggesting they could benefit from it

This approach gives package authors control while providing guardrails via Fleet validation.

---

Related Code References

File	Relevance
kibana: template/install.ts:640-642	Where Fleet determines if TSDB mode should be enabled
kibana: template/template.ts:148-157	Component template selection based on data stream type
kibana: common/types/models/epm.ts:465	Type definition for index_mode
package-spec: input/manifest.spec.yml:110-116	Current input package spec (missing index_mode)
package-spec: data_stream/manifest.spec.yml:609-617	Data stream spec with index_mode support

---

Why is Claude responding? | Type @claude to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[jsoriano]

Agree with Claude, we should add the flag to package-spec manifest, but we should also ensure that Fleet only sets index_mode: time_series on metrics data streams. Users can select the data stream type on input packages.