Prototype gcp lithops for combined variables

[jbusecke]

I refactored developmentseed#2 to use gcp functions with lithops.

In the course of trying to get around some stubborn writing issues between gcs and icechunk I did refactor some of the logic.

• The process_all_files wrapper function now enables you to define local_storage and local_execution to test everything locally.
• I only use concurrent execution for building the virtual datasets and then save to icechunk in serial (this still fails on the GCS bucket, see below).
• I had to go through quite a bunch of steps to get the service account on the new project working. I have noted them in CLOUD_SETUP.md.

Handoff point

• I was not able to find out why on earth I cannot reliably write virtual datasets to icechunk on GCS. I kept getting Server returned non-2xx status code: 429\n \x1b[31m│\x1b[0m Too Many Requests: <?xml version=\'1.0\' encoding=\'UTF-8\'?><Error><Code>SlowDown</Code><Message>The object exceeded the rate limit for object mutation operations (create, update, and delete). sort of errors (see below for full error).
• I did run the script successfully (majority of datasets wrote sucessful) running fully local, but whenever I try to write to the bucket it fails for most datasets.
• To unblock this to the users I have for now manually copied my local store to gs://ismip6-icechunk/manual-transfer/. notebooks/open_icechunk_combined_vars.ipynb shows how to open that store.
• I am pretty lost how to diagnose the core issue here to be honest. The bucket observability metrics never show writes over 20/sec, so I assume this must be happening because a single file is hammered repeatedly?

Full Error

 # Full error trace: [0m session error: object store error Generic GCS error: Error performing PUT https://storage.googleapis.com/ismip6%2Dicechunk/\n  \x1b[31m│\x1b[0m combined%2Dvariables%2D2025%2D12%2D19%2Dv2%2Frefs%2Fbranch%2Emain%2Fref%2Ejson in 3.898954958s, after 9 retries, max_retries: 9, retry_timeout: 300s  - Server returned non-2xx status code: 429\n  \x1b[31m│\x1b[0m Too Many Requests: <?xml version=\'1.0\' encoding=\'UTF-8\'?><Error><Code>SlowDown</Code><Message>The object exceeded the rate limit for object mutation operations (create, update, and delete).\n  \x1b[31m│\x1b[0m Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429.</Message><Details>The object ismip6-icechunk/combined-variables-2025-12-19-v2/refs/branch.main/ref.json\n  \x1b[31m│\x1b[0m exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429.</Details></Error>\n  \x1b[31m│\x1b[0m \n  \x1b[31m│\x1b[0m context:\n  \x1b[31m│\x1b[0m    0: icechunk::storage::object_store::write_ref\n  \x1b[31m│\x1b[0m            with ref_key="branch.main/ref.json" previous_version=VersionInfo { etag: Some(ETag("\\"bca5b65bdadff9b7a58b11f9586fe1c9\\"")), generation: Some(Generation("1766201379785026")) }\n  \x1b[31m│\x1b[0m              at icechunk/src/storage/object_store.rs:528\n  \x1b[31m│\x1b[0m    1: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    2: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    3: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    4: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    5: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    6: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    7: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    8: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m    9: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   10: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   11: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   12: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   13: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   14: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   15: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   16: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   17: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   18: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   19: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   20: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   21: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   22: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   23: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   24: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   25: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   26: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   27: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   28: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   29: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   30: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   31: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   32: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   33: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   34: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   35: icechunk::refs::update_branch\n  \x1b[31m│\x1b[0m            with name="main" new_snapshot=B26JW2A14PX1D4Q5YS5G current_snapshot=Some(0T8K5249691YH9KF2WMG)\n  \x1b[31m│\x1b[0m              at icechunk/src/refs.rs:173\n  \x1b[31m│\x1b[0m   36: icechunk::session::_commit\n  \x1b[31m│\x1b[0m            with Added ULB_fETISh_32km/ctrl_proj_std rewrite_manifests=false\n  \x1b[31m│\x1b[0m              at icechunk/src/session.rs:1042\n  \x1b[31m│\x1b[0m   37: icechunk::session::commit\n  \x1b[31m│\x1b[0m            with Added ULB_fETISh_32km/ctrl_proj_std\n  \x1b[31m│\x1b[0m              at icechunk/src/session.rs:970\n  \x1b[31m│\x1b[0m \n'

[jbusecke]

cc @abarciauskas-bgse @martham93

[jbusecke]

Additional info from an earthmover slack chat: This object-store PR merged but apparently not yet published (I misread this during my research the other day).

So the best course of action here might be to wait for the release, and see if the issue goes away on its own.