WIP: Support icechunk stores

[jbusecke]

I am working towards icechunk (for now likely just native) support in titler-multidim here.

Depends on #95 and developmentseed/titiler#1235

Tasks

• Updated zarr to v3
• Modified the test fixture scripts to write v2 and v3 zarr stores explicitly (I retriggered them and committed the new fixtures)
• Add scripts/fixtures for native icechunk
• Deploy and test on real-world native icechunk data
• Add scripts/fixtures for virtual icechunk
• Deploy and test on real-world virtual icechunk data

[jbusecke]

I noticed that some tests fail with

E           TypeError: cannot pickle 'module' object

in the XarrayReader

This seems to only happen with caching is enabled (and only for the netcdf store)! To show this I have added an additional parametrization to the app test fixture which runs all tests both with and without cache.

I did confirm that this error is introduced with the upgrade of either xarray or zarr here. When I just add the changes to conftest.py to main, all tests pass, so this was not a silent failure.

Ill move on here for now, but I would love to fix this issue here. @hrodmn do you have any intuition why the caching would not play nice with a new xarray or zarr version?

[jbusecke]

Ok making some progress here, but Ill need some input from folks.

I have refactored the test datasets for zarr v2 and v3. The reading in seems to work (since the tests to get the variables only do pass), but I have some questions about the expected responses.

when I run the test_get_info pytest I am not getting the expected response (left is returned and right is expected).

FAILED tests/test_app.py::test_get_info[without_cache-zarr_store_v3] - AssertionError: assert {'bounds': [-180.0, -90.0, 180.0, 90.0], 'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326', 'band_metadata': [['b1', {}]], 'band_descriptions': [['b1', '0']], 'dtype': 'float64', 'nodata_type': 'Nodata', 'name': 'CDD0', 'count': 1, 'width': 72, 'height': 36, 'dimensions': ['y', 'x'], 'attrs': {}} == {'bounds': [-180, -90.0, 180.0, 90.0], 'band_metadata': [], 'band_descriptions': [], 'dtype': 'float64', 'nodata_type': 'None', 'height': 36, 'count': 1, 'width': 72, 'attrs': {}, 'name': 'CDD0'}
  
  Common items:
  {'attrs': {},
   'bounds': [-180.0, -90.0, 180.0, 90.0],
   'count': 1,
   'dtype': 'float64',
   'height': 36,
   'name': 'CDD0',
   'width': 72}
  Differing items:
  {'nodata_type': 'Nodata'} != {'nodata_type': 'None'}
  {'band_descriptions': [['b1', '0']]} != {'band_descriptions': []}
  {'band_metadata': [['b1', {}]]} != {'band_metadata': []}
  Left contains 2 more items:
  {'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326', 'dimensions': ['y', 'x']}
  
  Full diff:
    {
        'attrs': {},
  -     'band_descriptions': [],
  ?                           --
  +     'band_descriptions': [
  +         [
  +             'b1',
  +             '0',
  +         ],
  +     ],
  -     'band_metadata': [],
  ?                       --
  +     'band_metadata': [
  +         [
  +             'b1',
  +             {},
  +         ],
  +     ],
        'bounds': [
  -         -180,
  +         -180.0,
  ?             ++
            -90.0,
            180.0,
            90.0,
        ],
        'count': 1,
  +     'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326',
  +     'dimensions': [
  +         'y',
  +         'x',
  +     ],
        'dtype': 'float64',
        'height': 36,
        'name': 'CDD0',
  -     'nodata_type': 'None',
  ?                       ^^
  +     'nodata_type': 'Nodata',
  ?                       ^^^^
        'width': 72,
    }

I am not exactly sure how these come about TBH.

Are 'band_metadata', nodata_type, dimensions... changes that have been made to main recently? Or is this something that comes from xarray?

Happy to just rewrite the expected output, but I first wanted to understand exactly what is going on.

All I could gather here was that these response fields are expected for e.g. the netcdf output.

[hrodmn]

The pickling problem is solved if we bump this up to 2025.10.1!

[jbusecke]

Ok I was able to pass the simple tests (including native icechunk) when running from a local modified version of titiler.xarray (now exists as PR here).

Just a note to myself how to link uv to a local file.

In pyproject.toml:

[tool.uv.sources]
titiler-xarray = { path = "../titiler/src/titiler/xarray", editable = true }

[jbusecke]

Almost have all tests passing, but I am not sure how to interpret these:

Is there any way to enable the full error trace to be displayed?

[jbusecke]

Ah thanks to @hrodmn I was able to look at the response detail and get:

{"detail":"Bounds and transform are inconsistent"}

currently stuck here. I suspect that this is somehow related to upstream changes ( for example here: #96 (comment)) or the way I have regenerated some of the datasets (less likely since this is happening for every test dataset and I only replaced the zarr/icechunk ones).

[jbusecke]

@hrodmn and I had the suspicion that this is a dependency issue, and I traced down changes by checking out a fresh branch based on main, doing rm uv.lock; uv sync and then git diff uv.lock.

It turns out the errors are introduced by rio-tiler (7.9.x caused the issues!) so I have for now pinned it to 7.8.1.

[jbusecke]

Moving this over to #101