Skip to content

fix pr time coordinate handling#52

Merged
SBeairsto merged 1 commit intomainfrom
prec_test
Jan 30, 2026
Merged

fix pr time coordinate handling#52
SBeairsto merged 1 commit intomainfrom
prec_test

Conversation

@SBeairsto
Copy link
Copy Markdown
Collaborator

Summary

This PR fixes issues with converting forecast data that has separate forecast_initial_time and forecast_hour dimensions into a flattened time coordinate, and resolves a related bug in the zarr-to-torch conversion pipeline.

Changes

1. Fixed time flattening in metadata.py

Problem: When converting forecast data with forecast_initial_time and forecast_hour dimensions to valid time coordinates, the code was:

  • Converting hours to nanoseconds instead of hours (timedelta64[ns]timedelta64[h])
  • Creating a MultiIndex that couldn't be properly overwritten
  • Had a duplicate line for coordinate assignment

Solution:

init, hour = xr.broadcast(ds.forecast_initial_time, ds.forecast_hour)
valid_time = init + hour.astype("timedelta64[h]")  # Changed from [ns] to [h]
ds = ds.stack(time=("forecast_initial_time", "forecast_hour"))
ds = ds.reset_index("time", drop=True)  # Reset MultiIndex before assignment
ds["time"] = valid_time.stack(time=("forecast_initial_time", "forecast_hour")).values
return ds

This properly converts forecast hours (e.g., 1-12) to hour offsets and creates valid datetime coordinates (e.g., 1979-01-01T07:00:00 instead of 1979-01-01T06:00:00.000000001).

2. Fixed iterator validation in zarr_to_torch.py

Problem: progress_starmap was receiving a zip iterator which couldn't be validated even with the total parameter specified, causing:

ValueError: If the tasks are an iterator, the total parameter must be specified` 

Solution:

pool_tuple = list(zip(  # Convert iterator to list
    indices,
    partial_paths,
    ds[var.name].transpose(*[dim for dim in chunks.keys() if dim in ds[var.name].dims]),
))

progress_starmap(
    parallel_loop, pool_tuple, total=len(pool_tuple), n_cpu=24, chunk_size=1
)

Testing

  • Verified time coordinates are correctly computed for forecast data with hour offsets
  • Confirmed zarr-to-torch conversion completes without errors
  • Validated temporal alignment between datasets using correlation analysis (no offset detected)

Files Modified

  • metadata.py: Fixed time coordinate flattening logic
  • tools/zarr_to_torch.py: Converted zip iterator to list for progress_starmap

@SBeairsto SBeairsto merged commit 82afc15 into main Jan 30, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant