Various issues with chunks not being written with zarr-python v3

[TomNicholas]

What happened?

When writing to zarr_format=3, zarr-python v3 refuses to create chunks. It also is creating a file called store.zarr/arr/c rather than a directory called store.zarr/arr/c/.

What did you expect to happen?

Not that

Minimal Complete Verifiable Example

In [1]: import xarray as xr

In [2]: ds = xr.Dataset({"a": 1})

In [3]: ds.to_zarr("experiments/test.zarr", zarr_format=2, compute=True)
Out[3]: <xarray.backends.zarr.ZarrStore at 0x1087f8820>

In [4]: !ls experiments/test.zarr/a/
0

In [5]: !rm -r experiments/test.zarr/

In [6]: ds.to_zarr("experiments/test.zarr", zarr_format=2, compute=False)
Out[6]: Delayed('_finalize_store-c2915b5f-7ad6-4988-bab5-bd6d488520fd')

In [7]: !ls experiments/test.zarr/a/
0

In [8]: !rm -r experiments/test.zarr/

In [9]: ds.to_zarr("experiments/test.zarr", zarr_format=3, compute=True)
/Users/tom/Documents/Work/Code/xarray/.venv/lib/python3.13/site-packages/zarr/api/asynchronous.py:203: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Out[9]: <xarray.backends.zarr.ZarrStore at 0x108abf490>

In [10]: !ls experiments/test.zarr/a/c
experiments/test.zarr/a/c

In [11]: !ls experiments/test.zarr/a/c/
ls: experiments/test.zarr/a/c/: Not a directory

In [12]: !rm -r experiments/test.zarr/

In [13]: ds.to_zarr("experiments/test.zarr", zarr_format=3, compute=False)
/Users/tom/Documents/Work/Code/xarray/.venv/lib/python3.13/site-packages/zarr/api/asynchronous.py:203: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(
Out[13]: Delayed('_finalize_store-eaa2ebf4-9248-4b1b-9349-36b1532909bd')

In [14]: !ls experiments/test.zarr/a/c
experiments/test.zarr/a/c

In [15]: !ls experiments/test.zarr/a/c/
ls: experiments/test.zarr/a/c/: Not a directory

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

cc @jhamman @d-v-b

Environment

Details

Package Version Editable project location ----------------------------- ---------------------------------- ------------------------------------- alabaster 1.0.0 asttokens 3.0.0 attrs 25.3.0 babel 2.17.0 bokeh 3.7.0 bottleneck 1.4.2 cartopy 0.24.1 certifi 2025.1.31 cfgv 3.4.0 cftime 1.6.4.post1 charset-normalizer 3.4.1 click 8.1.8 cloudpickle 3.1.1 contourpy 1.3.1 coverage 7.6.12 crc32c 2.7.1 cycler 0.12.1 dask 2025.2.0 decorator 5.2.1 deprecated 1.2.18 distlib 0.3.9 distributed 2025.2.0 docutils 0.21.2 donfig 0.8.1.post1 execnet 2.1.1 executing 2.2.0 filelock 3.17.0 flox 0.10.0 fonttools 4.56.0 fsspec 2025.3.0 h5netcdf 1.6.1 h5py 3.13.0 hypothesis 6.129.1 identify 2.6.9 idna 3.10 imagesize 1.4.1 iniconfig 2.0.0 ipython 9.0.2 ipython-pygments-lexers 1.1.1 jedi 0.19.2 jinja2 3.1.6 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 kiwisolver 1.4.8 llvmlite 0.44.0 locket 1.0.0 lz4 4.4.3 markupsafe 3.0.2 matplotlib 3.10.1 matplotlib-inline 0.1.7 msgpack 1.1.0 mypy 1.15.0 mypy-extensions 1.0.0 narwhals 1.30.0 nc-time-axis 1.4.1 netcdf4 1.7.2 nodeenv 1.9.1 numba 0.61.0 numbagg 0.9.0 numcodecs 0.15.1 numpy 2.1.3 numpy-groupies 0.11.2 opt-einsum 3.4.0 packaging 24.2 pandas 2.2.3 pandas-stubs 2.2.3.250308 parso 0.8.4 partd 1.4.2 pexpect 4.9.0 pillow 11.1.0 platformdirs 4.3.6 pluggy 1.5.0 pooch 1.8.2 pre-commit 4.1.0 prompt-toolkit 3.0.50 psutil 7.0.0 ptyprocess 0.7.0 pure-eval 0.2.3 pyarrow 19.0.1 pygments 2.19.1 pyparsing 3.2.1 pyproj 3.7.1 pyshp 2.3.1 pytest 8.3.5 pytest-cov 6.0.0 pytest-env 1.1.5 pytest-mypy-plugins 3.2.0 pytest-timeout 2.3.1 pytest-xdist 3.6.1 python-dateutil 2.9.0.post0 pytz 2025.1 pyyaml 6.0.2 referencing 0.36.2 regex 2024.11.6 requests 2.32.3 roman-numerals-py 3.1.0 rpds-py 0.23.1 ruff 0.10.0 scipy 1.15.2 seaborn 0.13.2 setuptools 76.0.0 shapely 2.0.7 six 1.17.0 snowballstemmer 2.2.0 sortedcontainers 2.4.0 sparse 0.15.5 sphinx 8.2.3 sphinx-autosummary-accessors 2025.3.1 sphinxcontrib-applehelp 2.0.0 sphinxcontrib-devhelp 2.0.0 sphinxcontrib-htmlhelp 2.1.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 2.0.0 sphinxcontrib-serializinghtml 2.0.0 stack-data 0.6.3 tblib 3.0.0 tomlkit 0.13.2 toolz 1.0.0 tornado 6.4.2 traitlets 5.14.3 types-colorama 0.4.15.20240311 types-decorator 5.2.0.20250224 types-defusedxml 0.7.0.20240218 types-docutils 0.21.0.20241128 types-networkx 3.4.2.20250312 types-pexpect 4.9.0.20241208 types-psutil 7.0.0.20250218 types-pycurl 7.45.6.20250309 types-pygments 2.19.0.20250305 types-python-dateutil 2.9.0.20241206 types-pytz 2025.1.0.20250204 types-pyyaml 6.0.12.20241230 types-setuptools 76.0.0.20250313 typing-extensions 4.12.2 tzdata 2025.1 urllib3 2.3.0 virtualenv 20.29.3 wcwidth 0.2.13 wrapt 1.17.2 xarray 2025.1.3.dev87+g30f5bbaf.d20250318 /Users/tom/Documents/Work/Code/xarray xyzservices 2025.1.0 zarr 3.0.5 zict 3.0.0

[jhamman]

Scalar arrays behave a bit different in Zarr v3. This line from the spec is important:

Arrays may have 0 dimensions (when for example representing scalars), in which case the coordinate of a chunk is the empty tuple, and the chunk key will consist of the string c.

Zarr-Python seems to be doing the "right" thing here (as does Xarray):

In [12]: store = MemoryStore()

In [13]: dada = np.asarray(1)

In [14]: store = MemoryStore()

In [15]: data = np.asarray(1)

In [16]: zarr.create_array(store, data=data)
Out[16]: <Array memory://4756849984 shape=() dtype=int64>

In [17]: store._store_dict
Out[17]:
{'zarr.json': <zarr.core.buffer.cpu.Buffer at 0x1092c65d0>,
 'c': <zarr.core.buffer.cpu.Buffer at 0x11b66bc50>}

[TomNicholas]

Thanks @jhamman - that makes much more sense now. The behaviour I'm seeing with LocalStore seems consistent with that, so I'll close this.