Update Zarr to v3

[eroell]

Closes #170.

An optional dependency (vitessce-python) is still requiring zarr<3 though, zarr v3 is an open issue vitessce/vitessce-python#481

• adjusting for AnnData's use of consolidated metadata

[eroell]

No blocks from our side

[eroell]

@Zethson since you created the issue; was this specifically about using zarr>=3, or about removing the pins as done now?

[Zethson]

zarr>=3

I think we can go with this. Don't need to support the old format

[Zethson]

Hmm I had unpinned it before and ran into a test error back then. Surprised it works now but ok

[Zethson]

@eroell the tests used the cached environment. This shouldn't be the case right?

[Zethson]

Also, if you check the notebooks jobs:

• zarr==2.18.7

[eroell]

If vitessce is included as a documentation build dependency, the following error is raised

      File "/home/docs/.local/share/hatch/env/virtual/ehrdata/1hBfmIf-/docs/lib/python3.13/site-packages/upath/core.py", line 8, in <module>
        from pathlib import _PosixFlavour  # type: ignore
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ImportError: cannot import name '_PosixFlavour' from 'pathlib' (/home/docs/.asdf/installs/python/3.13.3/lib/python3.13/pathlib/__init__.py)

I think the new appearance of this error has to do with multiple more recent packages being used now that the zarr<3 pin is removed in favor of zarr>=3.
Within reasonable time, I could not backtrace its specific cause.

I became suspicious as well whether lamindb_setup's pin to "universal_pathlib==0.2.6" has something to do with it, but this seems to not be to blame.

Since vitessce-python relies on zarr<3, I suggest to remove this dependency from the docs build and skip intersphinx links until vitessce-python accepts zarr>=3.

[flying-sheep]

I became suspicious as well whether lamindb_setup's pin to "universal_pathlib==0.2.6" has something to do with it, but this seems to not be to blame.

That package provides the upath import, but starting from 0.2.0, it no longer tries to import _PosixFlavour, so this traceback implies that a version of universal_pathlib other than the one pinned by lamindb was installed.

---

also even if you remove the vitessce dep from the docs, there is absolutely no reason to remove the link from :meth:`~vitessce.config.VitessceConfig.widget`! Intersphinx doesn’t need the package installed, just an entry in theintersphinx_mapping.

[eroell]

That package provides the upath import, but fsspec/universal_pathlib#173, it no longer tries to import _PosixFlavour, so this traceback implies that a version of universal_pathlib other than the one pinned by lamindb was installed.

Indeed, I have the suspicion that along the dependency tree of vitessce a pin could exist? Since there were not actively maintained packages included too, I stopped digging at one point..

[eroell]

also even if you remove the vitessce dep from the docs, there is absolutely no reason to remove the link from :meth:~vitessce.config.VitessceConfig.widget! Intersphinx doesn’t need the package installed, just an entry in theintersphinx_mapping.

OK! I'll add this again then

[flying-sheep]

It’s really easy, I just added an lower bound to lamindb. Seems like uv thought it was a good idea to install some ancient lamindb version together with a universal_pathlib==0.1.0 that doesn’t support Python 3.12+

Also you let your docs degrade by not running it with -W. It was really easy to fix the breakage, just leave that flag in so you know when things break!.

• I added scanpy and fsspec to intersphinx_mapping (thank you LLMs for correctly guessing the URLs that I could then verify by simply clicking them)
• I added a qualnames_override for zarr.storage._common.StorePath → zarr.storage.StorePath
• I added nitpick_ignores for zarr.core.buffer.core.Buffer (which doesn’t seem to be exported) and Initial support for PEP 695 type aliases sphinx-doc/sphinx#13508
• I fixed the broken .. figure:: path

Most of these should have been fixed when they were introduced (except for the zarr stuff, which makes sense to be fixed in this PR)

[flying-sheep]

I actually don’t know if vitessce takes a StoreLike or only a real zarr.abc.store.Store.

If the latter, than the import should be changed.

[eroell]

It’s really easy, I just added a lowerupper bound to lamindb

🤯

[flying-sheep]

Ah sorry, lower bound! Don’t do upper bounds, this was just a typo.

[eroell]

Ah sorry, lower bound! Don’t do upper bounds, this was just a typo.

Yes I got it, the exploding head was regarding how quickly you fixed that

[flying-sheep]

Looks good except for that nasty implicit state change on import!

Let’s wait for the discussion in scverse/anndata#2193 also!

[eroell]

I changed a final thing here:

• add chunks argument to ehrdata.io.write_zarr, with
  • default “auto” (Same style as zarr.create_array); This option uses anndata.io.write_elem, and its defaults.
  • A second option “ehrdata_auto”, using a blueprint for custom chunking as suggested in the discussion on anndata.io.write_elem with an AnnData object raises error on chunk_shape when using chunks argument scverse/anndata#2193. I don’t claim this chunking is what is best for our usecases. Rather, the implementation is now embedded and available for trying, can be adapted in the future, and should act as blueprint should “auto” turn out to not be what is best for us. It'd be easier for contributors to alter this, than finding and adapting the blueprint mentioned in this discussion.