Add Elasticsearch 9.x support

Elasticsearch 9.x introduced breaking changes to the elasticsearch-ruby
client gem that prevent Chewy from working with ES 9 clusters:

1. The scroll API now requires scroll_id nested under a body: key
   ({scroll: '1m', body: {scroll_id: '...'}}) rather than as a
   top-level parameter. Without this, scroll requests raise an
   ArgumentError in elasticsearch-ruby 9.x.

2. API responses are now wrapped in Elasticsearch::API::Response objects
   instead of returning raw hashes/arrays. Code that calls .map or
   other Enumerable methods directly on responses (e.g. cat.indices)
   must now call .body first to access the underlying data.

3. The '_type' field was removed from search responses in ES 7+ and
   the elasticsearch-ruby 9.x client no longer includes it. Mock
   response helpers that included '_type' => '_doc' produced responses
   that diverged from actual ES behavior.

Changes:

- Widen elasticsearch gem dependency from < 9.0 to < 10.0
- Restructure perform_scroll to nest scroll_id under body:, which is
  compatible with both elasticsearch-ruby 8.x and 9.x
- Access .body on cat.indices response in the drop_indices test helper
- Remove '_type' => '_doc' from Minitest and RSpec mock response helpers
- Add elasticsearch version matrix to CI (9.3.1 default with 8.15.0
  spot-checks) using dynamic service container images
- Default docker-compose ES image to 9.3.1
- Bump version to 9.0.0

Author: mattmenefee

.github/workflows/ruby.yml

@@ -27,7 +27,15 @@ jobs:
       matrix:
         ruby: [ '3.2', '3.3', '3.4', '4.0' ]
         gemfile: [rails.7.2.activerecord, rails.8.0.activerecord, rails.8.1.activerecord]
-    name: ${{ matrix.ruby }}-${{ matrix.gemfile }}
+        elasticsearch: ['9.3.1']
+        include:
+          - ruby: '3.2'
+            gemfile: rails.7.2.activerecord
+            elasticsearch: '8.15.0'
+          - ruby: '4.0'
+            gemfile: rails.8.1.activerecord
+            elasticsearch: '8.15.0'
+    name: ${{ matrix.ruby }}-${{ matrix.gemfile }}-es${{ matrix.elasticsearch }}
 
     env:
       BUNDLE_GEMFILE: gemfiles/${{ matrix.gemfile }}.gemfile
@@ -45,7 +53,7 @@ jobs:
           --health-timeout 5s
           --health-retries 5
       elasticsearch:
-        image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
+        image: docker.elastic.co/elasticsearch/elasticsearch:${{ matrix.elasticsearch }}
         ports:
           - '9250:9200'
         env:

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 9.0.0 (unreleased)
+
+### New Features
+
+* [#1009](https://github.com/toptal/chewy/pull/1009): Add Elasticsearch 9.x support while retaining ES 8.x compatibility. The `elasticsearch` gem dependency now allows `>= 8.14, < 10.0`. ([@mattmenefee][])
+
+### Changes
+
+* [#1009](https://github.com/toptal/chewy/pull/1009): **(Breaking)** The `search_query.chewy` notification payload for scroll requests now nests `scroll_id` under `body:` (i.e. `{scroll: '1m', body: {scroll_id: '...'}}` instead of `{scroll: '1m', scroll_id: '...'}`). Update any application code that subscribes to scroll notification payloads. ([@mattmenefee][])
+* [#1009](https://github.com/toptal/chewy/pull/1009): Remove `'_type' => '_doc'` from mock response helpers in `Chewy::Minitest::Helpers` and `Chewy::Rspec::Helpers` to match actual ES response format (ES 7+ no longer returns `_type`). ([@mattmenefee][])
+
+### Bugs Fixed
+
 ## 8.0.2 (unreleased)
 
 ### Changes

chewy.gemspec

@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ['lib']
 
   spec.add_dependency 'activesupport', '>= 7.2'
-  spec.add_dependency 'elasticsearch', '>= 8.14', '< 9.0'
+  spec.add_dependency 'elasticsearch', '>= 8.14', '< 10.0'
   spec.add_dependency 'elasticsearch-dsl'
   spec.metadata['rubygems_mfa_required'] = 'true'
 end

docker-compose.yml

@@ -1,6 +1,6 @@
 services:
   elasticsearch_test:
-    image: "docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-8.15.0}"
+    image: "docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-9.3.1}"
     environment:
       - bootstrap.memory_lock=${ES_MEMORY_LOCK:-false}
       - "ES_JAVA_OPTS=-Xms${TEST_ES_HEAP_SIZE:-500m} -Xmx${TEST_ES_HEAP_SIZE:-500m}"

lib/chewy/minitest/helpers.rb

@@ -96,7 +96,6 @@ def mock_elasticsearch_response_sources(index, hits, &block)
             'hits' => hits.each_with_index.map do |hit, i|
               {
                 '_index' => index.index_name,
-                '_type' => '_doc',
                 '_id' => hit[:id] || (i + 1).to_s,
                 '_score' => 3.14,
                 '_source' => hit

lib/chewy/rspec/helpers.rb

@@ -39,7 +39,6 @@ def mock_elasticsearch_response_sources(index, hits)
             'hits' => hits.each_with_index.map do |hit, i|
               {
                 '_index' => index.index_name,
-                '_type' => '_doc',
                 '_id' => (i + 1).to_s,
                 '_score' => 3.14,
                 '_source' => hit

lib/chewy/search/scrolling.rb

@@ -128,9 +128,10 @@ def scroll_objects(**options, &block)
 
     private
 
-      def perform_scroll(body)
-        ActiveSupport::Notifications.instrument 'search_query.chewy', notification_payload(request: body) do
-          Chewy.client.scroll(body)
+      def perform_scroll(scroll:, scroll_id:)
+        request = {scroll: scroll, body: {scroll_id: scroll_id}}
+        ActiveSupport::Notifications.instrument 'search_query.chewy', notification_payload(request: request) do
+          Chewy.client.scroll(request)
         end
       end
     end

lib/chewy/version.rb

@@ -1,3 +1,3 @@
 module Chewy
-  VERSION = '8.0.1'.freeze
+  VERSION = '9.0.0'.freeze
 end

spec/chewy/search/scrolling_spec.rb

@@ -126,7 +126,7 @@
             hash_including(
               index: [CitiesIndex, CountriesIndex],
               indexes: [CitiesIndex, CountriesIndex],
-              request: {scroll: '1m', scroll_id: an_instance_of(String)}
+              request: {scroll: '1m', body: {scroll_id: an_instance_of(String)}}
             )
           ])
         end

spec/spec_helper.rb

@@ -46,7 +46,7 @@
 # Low-level substitute for now-obsolete drop_indices
 def drop_indices
   indices = Chewy.client.cat.indices(format: 'json')
-    .map { |entry| entry['index'] }
+    .body.map { |entry| entry['index'] }
     .reject { |name| name.start_with?('.') }
   return if indices.blank?