New chunking approach that never splits encoded chunks

[jsignell]

• Towards opening a zarr dataset taking so much time with dask #8902
  • it doesn't really close it, but it is a better alternative than chunks={} or chunks="auto". Since Remove special mapping of auto to {} in open_zarr #11010 got in xarra could eventually change the default on open_zarr to map it to "preserve" rather than to {} if dask is available.
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

Proposal

A new chunks option that is only allowed to use encoded chunks or multiples of them. No chunk splitting allowed.

Demo

Current behavior when chunks="auto"

ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3", chunks="auto")

This PR introduces a new option: chunks="preserve"

ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3", chunks="preserve")

Context

I originally set out to update the auto_chunks function in dask, but it felt like my goals were actually quite different. The goal of the dask auto_chunks function is to guarantee that the chunksize will be under a configurable limit while preserving the aspect ratio of previous_chunks (previous_chunks == encoding). This PR instead guarantees that encoded chunks are never split but it will multiply them by some factor to try to get the chunksize close to a targetsize. It doesn't try to preserve the aspect ratio of the chunks. Instead it goes after the dim where there is the greatest number of chunks and it tries to take those in bigger bites.

Also:

• I'm not quite sure how the interface should work and I am definitely not attached to the word "preserve".
• I originally put this in the DaskManager, but moved it to the base ChunkManagerEntrypoint class once I realized there was nothing dasky about it. I'm not sure if there is really supposed to be logic in methods on that class though.

[jsignell]

These tests are kind of slow.

[jsignell]

@dcherian is this something you would be able to review? I'd love to get more people trying it.

[dcherian]

IMO this should just be "auto". Are we really working around a dask bug?

[jsignell]

I thought about doing this work in dask, but I ended up deciding that this is a sufficiently different goal from dask auto. The goal of dask auto is to guarantee that the chunksize will be under a configurable limit while preserving the aspect ratio of previous_chunks. We don't really want either of those things.

But maybe you are just saying: this is what xarray should mean by "auto" in which case I definitely agree. I'm just not sure how to make the transition from the old version of "auto" to the new version. Maybe it would be easier to give it a new name ("preserve") and then change the default value in kwargs from chunks="auto" to chunks="preserve" at some point. If we just change what "auto" means then there is no way for people to get the dask auto behavior.