[Feature Request] Index transforms: Make "Pages per execution" configurable or increase it

[coding-jodl]

Is your feature request related to a problem? Please describe

I am using the Index Transforms jobs, especially with the Transforms APIs. I searched in the documentation but I couldn't find information about the max. size of "Pages per execution"/"page_size" but with trial and error it seems these are the max. values for page_size: 10,000 for "continuous: false" and 1,000 for "continuous: true".

Error if page_size is > 10,000 and "continuous" is set to false:
Failed to create transform: [illegal_argument_exception] Page size must be between 1 and 10,000

Error if page_size is > 1,000 and "continuous" is set to true:
Failed to create transform: [illegal_argument_exception] Page size must be between 1 and 1,000

I tried this with OpenSearch and OpenSearch-Dashboards v3.3.0 and also in v2.18.0 with the same results.

Describe the solution you'd like

I would like to configure the page_size for Index Transforms, e.g. to set it up to 20,000 or at least increase it to a higher max. number. I also don't understand why the max. value is just 1,000 if "continuous" is set to true but is 10,000 if it is false.

Related component

Plugins

Describe alternatives you've considered

No response

Additional context

No response

[cwperks]

Tranferring to ISM repo

[lawofcycles]

@coding-jodl @cwperks
I've investigated the current implementation and would like to share my findings and propose an approach.

Current Implementation

The page_size limits are hardcoded in Transform.kt.

index-management/src/main/kotlin/org/opensearch/indexmanagement/transform/model/Transform.kt

Lines 298 to 299 in e78c4c9

	const val MAXIMUM_PAGE_SIZE = 10_000
	const val MAXIMUM_PAGE_SIZE_CONTINUOUS = 1_000

The validation logic applies different limits based on the continuous flag.

index-management/src/main/kotlin/org/opensearch/indexmanagement/transform/model/Transform.kt

Lines 94 to 98 in e78c4c9

	if (continuous) {
	require(pageSize in MINIMUM_PAGE_SIZE..MAXIMUM_PAGE_SIZE_CONTINUOUS) { "Page size must be between 1 and 1,000" }
	} else {
	require(pageSize in MINIMUM_PAGE_SIZE..MAXIMUM_PAGE_SIZE) { "Page size must be between 1 and 10,000" }
	}

I believe the stricter limit (1,000) for continuous transforms exists because

1. Continuous transforms run repeatedly on a schedule, causing sustained load on the cluster
2. Higher page_size means more memory consumption per execution
3. The combination of frequent execution + high memory usage could destabilize the cluster

In contrast, one-shot transforms (continuous=false) only run once, so a temporary spike in resource usage is more acceptable.

Proposed Approach

Add settings to make these limits configurable:

val TRANSFORM_MAX_PAGE_SIZE = Setting.intSetting(
   "plugins.index_management.transform.max_page_size",
   10_000,  // default (backward compatible)
   1,
   Setting.Property.NodeScope,
   Setting.Property.Dynamic
)

val TRANSFORM_MAX_PAGE_SIZE_CONTINUOUS = Setting.intSetting(
   "plugins.index_management.transform.max_page_size_continuous",
   1_000,   // default (backward compatible)
   1,
   Setting.Property.NodeScope,
   Setting.Property.Dynamic
)

Points

• Maintains backward compatibility with current defaults
• Allows users to increase limits based on their hardware capacity
• Keeps the distinction between continuous and one-shot transforms

Looking forward to your feedback!