Add Elasticsearch 9.x support

mattmenefee · mattmenefee · commit 98955eb1a57c · 2026-03-17T19:23:33.000-04:00
Elasticsearch 9.x introduced breaking changes to the elasticsearch-ruby
client gem that prevent Chewy from working with ES 9 clusters:

1. The scroll API now requires scroll_id nested under a body: key
   ({scroll: '1m', body: {scroll_id: '...'}}) rather than as a
   top-level parameter. Without this, scroll requests raise an
   ArgumentError in elasticsearch-ruby 9.x.

2. API responses are now wrapped in Elasticsearch::API::Response objects
   instead of returning raw hashes/arrays. Code that calls .map or
   other Enumerable methods directly on responses (e.g. cat.indices)
   must now call .body first to access the underlying data.

3. The '_type' field was removed from search responses in ES 7+ and
   the elasticsearch-ruby 9.x client no longer includes it. All
   references to '_type' have been removed: mock response helpers,
   EVERFIELDS, wrapper accessors, and documentation examples.

4. The elasticsearch Ruby gem at version 9.x sends a compatible-with=9
   header that ES 8.x servers reject. CI matrix entries that test
   against ES 8.x must pin the gem to ~&gt; 8.14.

Changes:

- Widen elasticsearch gem dependency from &lt; 9.0 to &lt; 10.0
- Restructure perform_scroll to nest scroll_id under body:, which is
  compatible with both elasticsearch-ruby 8.x and 9.x
- Access .body on cat.indices response in the drop_indices test helper
- Remove '_type' from Minitest and RSpec mock response helpers,
  EVERFIELDS, Index::Wrapper accessors, and README examples
- Remove obsolete '_parent' from EVERFIELDS
- Add elasticsearch version matrix to CI (9.3.1 default with 8.15.0
  spot-checks) using dynamic service container images
- Pin elasticsearch gem to ~&gt; 8.14 in CI for ES 8.x matrix entries
- Add Ruby 3.4 / Rails 8.0 / ES 8.15.0 CI matrix entry
- Add scroll body format assertion in scrolling specs
- Expand migration guide with code examples, gem pinning guidance,
  corrected URLs, and more detailed upgrade steps
- Consolidate CHANGELOG entries into single item with sub-bullets
- Default docker-compose ES image to 9.3.1
- Bump version to 9.0.0
diff --git a/.github/workflows/ruby.yml b/.github/workflows/ruby.yml
@@ -27,7 +27,21 @@ jobs:
       matrix:
         ruby: [ '3.2', '3.3', '3.4', '4.0' ]
         gemfile: [rails.7.2.activerecord, rails.8.0.activerecord, rails.8.1.activerecord]
-    name: ${{ matrix.ruby }}-${{ matrix.gemfile }}
+        elasticsearch: ['9.3.1']
+        include:
+          - ruby: '3.2'
+            gemfile: rails.7.2.activerecord
+            elasticsearch: '8.15.0'
+            es_gem_constraint: '~> 8.14'
+          - ruby: '3.4'
+            gemfile: rails.8.0.activerecord
+            elasticsearch: '8.15.0'
+            es_gem_constraint: '~> 8.14'
+          - ruby: '4.0'
+            gemfile: rails.8.1.activerecord
+            elasticsearch: '8.15.0'
+            es_gem_constraint: '~> 8.14'
+    name: ${{ matrix.ruby }}-${{ matrix.gemfile }}-es${{ matrix.elasticsearch }}
 
     env:
       BUNDLE_GEMFILE: gemfiles/${{ matrix.gemfile }}.gemfile
@@ -45,7 +59,7 @@ jobs:
           --health-timeout 5s
           --health-retries 5
       elasticsearch:
-        image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
+        image: docker.elastic.co/elasticsearch/elasticsearch:${{ matrix.elasticsearch }}
         ports:
           - '9250:9200'
         env:
@@ -65,6 +79,9 @@ jobs:
         with:
           ruby-version: ${{ matrix.ruby }}
           bundler-cache: true
+      - name: Pin elasticsearch gem for ES 8.x
+        if: matrix.es_gem_constraint
+        run: echo "gem 'elasticsearch', '${{ matrix.es_gem_constraint }}'" >> $BUNDLE_GEMFILE
       - name: Tests
         run: bundle exec rspec
 
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,17 @@
 # Changelog
 
+## 9.0.0 (unreleased)
+
+### New Features
+
+### Changes
+
+* [#1009](https://github.com/toptal/chewy/pull/1009): **(Breaking)** Add Elasticsearch 9.x support while retaining ES 8.x compatibility. The `elasticsearch` gem dependency now allows `>= 8.14, < 10.0`. ([@mattmenefee][])
+  * The `search_query.chewy` notification payload for scroll requests now nests `scroll_id` under `body:` (i.e. `{scroll: '1m', body: {scroll_id: '...'}}` instead of `{scroll: '1m', scroll_id: '...'}`). Update any application code that subscribes to scroll notification payloads.
+  * `'_type' => '_doc'` has been removed from mock response helpers in `Chewy::Minitest::Helpers` and `Chewy::Rspec::Helpers` to match actual ES response format (ES 7+ no longer returns `_type`).
+
+### Bugs Fixed
+
 ## 8.0.2 (unreleased)
 
 ### Changes
diff --git a/README.md b/README.md
@@ -53,6 +53,7 @@ Chewy is compatible with MRI 3.2-4.0.
 
 | Chewy version | Elasticsearch version              |
 | ------------- | ---------------------------------- |
+| 9.0.x         | 8.x, 9.x                           |
 | 8.0.x         | 8.x                                |
 | 7.2.x         | 7.x                                |
 | 7.1.x         | 7.x                                |
@@ -220,7 +221,6 @@ end
     },
     "_data":{
       "_index":"users",
-      "_type":"_doc",
       "_id":"1",
       "_score":0.9808291,
       "_source":{
diff --git a/chewy.gemspec b/chewy.gemspec
@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ['lib']
 
   spec.add_dependency 'activesupport', '>= 7.2'
-  spec.add_dependency 'elasticsearch', '>= 8.14', '< 9.0'
+  spec.add_dependency 'elasticsearch', '>= 8.14', '< 10.0'
   spec.add_dependency 'elasticsearch-dsl'
   spec.metadata['rubygems_mfa_required'] = 'true'
 end
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -1,6 +1,6 @@
 services:
   elasticsearch_test:
-    image: "docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-8.15.0}"
+    image: "docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-9.3.1}"
     environment:
       - bootstrap.memory_lock=${ES_MEMORY_LOCK:-false}
       - "ES_JAVA_OPTS=-Xms${TEST_ES_HEAP_SIZE:-500m} -Xmx${TEST_ES_HEAP_SIZE:-500m}"
diff --git a/lib/chewy/index/wrapper.rb b/lib/chewy/index/wrapper.rb
@@ -39,7 +39,7 @@ def ==(other)
         end
       end
 
-      %w[_id _type _index].each do |name|
+      %w[_id _index].each do |name|
         define_method name do
           _data[name]
         end
diff --git a/lib/chewy/minitest/helpers.rb b/lib/chewy/minitest/helpers.rb
@@ -96,7 +96,6 @@ def mock_elasticsearch_response_sources(index, hits, &block)
             'hits' => hits.each_with_index.map do |hit, i|
               {
                 '_index' => index.index_name,
-                '_type' => '_doc',
                 '_id' => hit[:id] || (i + 1).to_s,
                 '_score' => 3.14,
                 '_source' => hit
diff --git a/lib/chewy/rspec/helpers.rb b/lib/chewy/rspec/helpers.rb
@@ -39,7 +39,6 @@ def mock_elasticsearch_response_sources(index, hits)
             'hits' => hits.each_with_index.map do |hit, i|
               {
                 '_index' => index.index_name,
-                '_type' => '_doc',
                 '_id' => (i + 1).to_s,
                 '_score' => 3.14,
                 '_source' => hit
diff --git a/lib/chewy/search/request.rb b/lib/chewy/search/request.rb
@@ -18,7 +18,7 @@ class Request
       include Scoping
       include Scrolling
       UNDEFINED = Class.new.freeze
-      EVERFIELDS = %w[_index _type _id _parent _routing].freeze
+      EVERFIELDS = %w[_index _id _routing].freeze
       DELEGATED_METHODS = %i[
         query filter post_filter knn order reorder docvalue_fields
         track_scores track_total_hits request_cache explain version profile
@@ -936,7 +936,7 @@ def find(*ids)
 
       # Returns and array of values for specified fields.
       # Uses `source` to restrict the list of returned fields.
-      # Fields `_id`, `_type`, `_routing` and `_index` are also supported.
+      # Fields `_id`, `_routing` and `_index` are also supported.
       #
       # @overload pluck(field)
       #   If single field is passed - it returns and array of values.
diff --git a/lib/chewy/search/scrolling.rb b/lib/chewy/search/scrolling.rb
@@ -128,9 +128,10 @@ def scroll_objects(**options, &block)
 
     private
 
-      def perform_scroll(body)
-        ActiveSupport::Notifications.instrument 'search_query.chewy', notification_payload(request: body) do
-          Chewy.client.scroll(body)
+      def perform_scroll(scroll:, scroll_id:)
+        request = {scroll: scroll, body: {scroll_id: scroll_id}}
+        ActiveSupport::Notifications.instrument 'search_query.chewy', notification_payload(request: request) do
+          Chewy.client.scroll(request)
         end
       end
     end
diff --git a/lib/chewy/version.rb b/lib/chewy/version.rb
@@ -1,3 +1,3 @@
 module Chewy
-  VERSION = '8.0.1'.freeze
+  VERSION = '9.0.0'.freeze
 end
diff --git a/migration_guide.md b/migration_guide.md
@@ -4,14 +4,57 @@ This document outlines the steps you need to take when migrating between major v
 Chewy and Elasticsearch. For simplicity's sake the guide will assume that you're using
 Chewy alongside a matching Elasticsearch version.
 
+## Chewy 8/Elasticsearch 8 to Chewy 9/Elasticsearch 9
+
+The Chewy 8 to 9 upgrade is relatively straightforward compared to previous major versions.
+The primary changes concern the `elasticsearch` Ruby gem version and the scroll notification
+payload format; no index mapping or query DSL changes are required.
+
+In order to upgrade Chewy 8/Elasticsearch 8 to Chewy 9/Elasticsearch 9 in the most seamless manner you have to:
+
+* Upgrade to the latest 8.x stable releases (Chewy 8.0.1, Elasticsearch 8.19.x). Your ES cluster must be on 8.19.x before upgrading to 9.x.
+* Study carefully [Breaking changes in 9.0](https://www.elastic.co/docs/release-notes/elasticsearch/breaking-changes), make sure your application conforms. Notable changes that may affect Chewy users:
+  * Stricter bulk API request parsing — malformed action lines that were previously tolerated are now rejected.
+  * The legacy `_knn_search` endpoint has been removed — use the `knn` clause in `_search` instead.
+  * Date histogram aggregations on boolean fields are no longer supported.
+  * Elasticsearch now returns HTTP 429 (instead of 5xx) for certain timeout conditions — update any retry logic that keys on 5xx status codes.
+* Update the `elasticsearch` gem — Chewy 9 requires `elasticsearch >= 8.14`.
+  * **Important:** The `elasticsearch` Ruby gem at version 9.x sends a `compatible-with=9` HTTP header that Elasticsearch 8.x servers reject with HTTP 400. Keep the gem pinned to `~> 8.14` (i.e. gem version 8.x) until after your Elasticsearch cluster has been upgraded to 9.x. You may optionally upgrade the gem to version 9.x after the cluster upgrade is complete.
+* Update scroll notification subscribers:
+  * The `search_query.chewy` notification payload for scroll requests now nests `scroll_id` under `body:`:
+    * Old format: `{scroll: '1m', scroll_id: '...'}`
+    * New format: `{scroll: '1m', body: {scroll_id: '...'}}`
+  * Update any application code that subscribes to scroll notification payloads:
+
+    ```ruby
+    # Before (Chewy < 9)
+    ActiveSupport::Notifications.subscribe('search_query.chewy') do |*, payload|
+      scroll_id = payload[:request][:scroll_id]
+    end
+
+    # After (Chewy 9+)
+    ActiveSupport::Notifications.subscribe('search_query.chewy') do |*, payload|
+      scroll_id = payload[:request].dig(:body, :scroll_id)
+    end
+    ```
+
+* Update test assertions for mock responses:
+  * If you use `mock_elasticsearch_response_sources` (in `Chewy::Minitest::Helpers` or `Chewy::Rspec::Helpers`), remove any assertions expecting `'_type' => '_doc'` in the mock hits it returns. The `_type` field has been removed from these helpers to match actual ES response format (ES 7+ no longer returns `_type`).
+  * If you use `mock_elasticsearch_response` with a hand-crafted raw response hash, ensure your raw response does not include `'_type'` for consistency with ES 8+/9 responses.
+* Run your test suite on Chewy 9 / Elasticsearch 9.
+* Run manual tests on Chewy 9 / Elasticsearch 9.
+* Deploy Chewy 9 (with the `elasticsearch` gem still pinned to `~> 8.14`).
+* Perform a [rolling upgrade](https://www.elastic.co/docs/deploy-manage/upgrade/deployment-or-cluster) of Elasticsearch to 9.x.
+* (Optional) Remove the `elasticsearch` gem version pin, allowing the gem to resolve to 9.x.
+
 ## Chewy 7/Elasticsearch 7 to Chewy 8/Elasticsearch 8
 
 In order to upgrade Chewy 7/Elasticsearch 7 to Chewy 8/Elasticsearch 8 in the most seamless manner you have to:
 
-* Upgrade to the latest 7.x stable release (Chewy 7.6.0, Elasticsearch 7.17)
-* Study carefully [Breaking changes in 8.0](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/breaking-changes-8.0.html), make sure your application conforms.
+* Upgrade to the latest 7.x stable release (Chewy 7.6.0, Elasticsearch 7.17).
+* Study carefully [Breaking changes in 8.0](https://www.elastic.co/guide/en/elasticsearch/reference/8.0/breaking-changes.html), make sure your application conforms.
 * Upgrade Ruby to >= 3.2 and Rails/ActiveSupport to >= 7.2 — Chewy 8 drops support for older versions.
-* Update the `elasticsearch` gem — Chewy 8 requires `>= 8.14, < 9.0`
+* Update the `elasticsearch` gem — Chewy 8 requires `elasticsearch >= 8.14`.
 * Update error class references:
   * The `elasticsearch-transport` gem was replaced by `elastic-transport`, moving all error classes from the `Elasticsearch::Transport` namespace to `Elastic::Transport`. Update any `rescue` clauses:
     * `Elasticsearch::Transport::Transport::Errors::NotFound` → `Elastic::Transport::Transport::Errors::NotFound`
diff --git a/spec/chewy/minitest/helpers_spec.rb b/spec/chewy/minitest/helpers_spec.rb
@@ -31,7 +31,6 @@ def assert_equal(expected, actual, message)
       [
         {
           '_index' => 'dummies',
-          '_type' => '_doc',
           '_id' => '2',
           '_score' => 3.14,
           '_source' => source
diff --git a/spec/chewy/rspec/helpers_spec.rb b/spec/chewy/rspec/helpers_spec.rb
@@ -13,7 +13,6 @@
     [
       {
         '_index' => 'cities',
-        '_type' => '_doc',
         '_id' => '1',
         '_score' => 3.14,
         '_source' => source
diff --git a/spec/chewy/runtime_spec.rb b/spec/chewy/runtime_spec.rb
@@ -4,6 +4,6 @@
   describe '.version' do
     specify { expect(described_class.version).to be_a(described_class::Version) }
     specify { expect(described_class.version).to be >= '8.0' }
-    specify { expect(described_class.version).to be < '9.0' }
+    specify { expect(described_class.version).to be < '10.0' }
   end
 end
diff --git a/spec/chewy/search/scrolling_spec.rb b/spec/chewy/search/scrolling_spec.rb
@@ -34,7 +34,11 @@
 
     describe '#scroll_batches' do
       context do
-        before { expect(Chewy.client).to receive(:scroll).twice.and_call_original }
+        before do
+          expect(Chewy.client).to receive(:scroll)
+            .with(hash_including(scroll: '1m', body: hash_including(:scroll_id)))
+            .twice.and_call_original
+        end
         specify do
           expect(request.scroll_batches(batch_size: 2).map do |batch|
             batch.map { |hit| hit['_source']['rating'] }
@@ -126,7 +130,7 @@
             hash_including(
               index: [CitiesIndex, CountriesIndex],
               indexes: [CitiesIndex, CountriesIndex],
-              request: {scroll: '1m', scroll_id: an_instance_of(String)}
+              request: {scroll: '1m', body: {scroll_id: an_instance_of(String)}}
             )
           ])
         end
diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
@@ -44,11 +44,12 @@
 # )
 
 # Low-level substitute for now-obsolete drop_indices.
-# Uses format: 'json' for version-portable parsing.
+# Uses format: 'json' for version-portable parsing and `.body` to unwrap
+# the Elasticsearch::API::Response (required for ES 9.x gem).
 # System indices (prefixed with '.') are excluded to avoid deleting
 # ES-internal indices like .security or .kibana.
 def drop_indices
-  indices = Chewy.client.cat.indices(format: 'json')
+  indices = Chewy.client.cat.indices(format: 'json').body
     .map { |entry| entry['index'] }
     .reject { |name| name.start_with?('.') }
   return if indices.blank?

Original file line number	Diff line number	Diff line change
`@@ -96,7 +96,6 @@ def mock_elasticsearch_response_sources(index, hits, &block)`
`96`	`96`	`'hits' => hits.each_with_index.map do \|hit, i\|`
`97`	`97`	`{`
`98`	`98`	`'_index' => index.index_name,`
`99`		`- '_type' => '_doc',`
`100`	`99`	`'_id' => hit[:id] \|\| (i + 1).to_s,`
`101`	`100`	`'_score' => 3.14,`
`102`	`101`	`'_source' => hit`
Original file line number	Diff line number	Diff line change
`@@ -39,7 +39,6 @@ def mock_elasticsearch_response_sources(index, hits)`
`39`	`39`	`'hits' => hits.each_with_index.map do \|hit, i\|`
`40`	`40`	`{`
`41`	`41`	`'_index' => index.index_name,`
`42`		`- '_type' => '_doc',`
`43`	`42`	`'_id' => (i + 1).to_s,`
`44`	`43`	`'_score' => 3.14,`
`45`	`44`	`'_source' => hit`