Dramatically different download speeds between versions

[irm-codebase]

Version Checks (indicate both or one)

• I have confirmed this bug exists on the lastest release of Atlite.

• I have confirmed this bug exists on the current master branch of Atlite.

Issue Description

It seems like older atlite versions achieved faster downloads... somehow?

I've been comparing download speeds between atlite=0.2.1 and atilite=0.2.13, and the former consistently beats out the latter when it comes to download speeds (tested around half a dozen times).

The difference is quite dramatic: minutes to hours.

Reproducible Example

# with atlite 0.2.1
import atlite

cutout = atlite.Cutout(
    path="tmp/cutout.nc",
    module="era5",
    xs=slice(-15.8, 37),
    ys=slice(30, 75),
    time=slice("2015-01", "2016-12")
)
cutout.prepare(["runoff"])

#################################
# with atlite 0.2.13
import atlite
cutout = atlite.Cutout(
    path="tmp/cutout.nc",
    module=["era5"],
    x=slice(-15.8, 37),
    y=slice(30, 75),
    time=slice("2015-01", "2016-12")
)
cutout.prepare(["runoff"])

Expected Behavior

Download speeds should generally improve between versions, or remain unchanged.

Installed Versions

Details

old: 0.2.1
new: 0.2.13

[irm-codebase]

Example with the old version. Completed in seconds.

[irm-codebase]

Meanwhile, a 0.2.13 run has not finished in one and a half hours...
The 0.2.1 version was even started later!

[irm-codebase]

More testing: I identified v0.2.10 to be the last "fast" version (although I feel like 0.2.1 was even faster but I cannot say for sure).

CDS logs tell me that the main difference is that v0.2.13 requests annual data by one month at a time, while v0.2.10 requests data all at once.

My guess is that the new approach is penalized by the CDS scheduler, putting jobs back in the queue instead of just executing them.

[irm-codebase]

One last test with different features (["height", "wind", "influx", "temperature", "runoff"]) shows a similar problem: requests are several requests are made instead of one, resulting more queue time.

I am not familiar enough with CDS to know if avoiding this one is possible, though.

[FabianHofmann]

great initiative @irm-codebase! so can we narrow it down to the way we are doing the feature requests with the CDS package?

[irm-codebase]

@FabianHofmann thanks!
That is my guess, yeah. I am not familiar enough with CDS, but my gut tells me the issue is somewhere in the requests.

Also, I keep seeing a message to "move to CDS-Beta", so I'd also check if this behavior also applies there.

[FabianHofmann]

yes, the beta version is an important upcoming step which we have to take in near future. If you are really motivated, you could also have a look at this and the (hopefully) performance improvements.

also note one more thing. CDS keeps files that were recently downloaded "warm" for fast re-downloading. So, whenever you already downloaded a feature recently, it would be much faster.

[fneum]

For reference, this is the PR where the monthly chunking was introduced (#236) and I think this is the one where requests were split by features (#86).

The CDS is undergoing a significant migration at the moment, which has resulted in throttled requests. The new CDS-BETA is out for a couple of weeks and the latest master of atlite can handle the new infrastructure (requires a changed API key). (#361, #364). But I have not tested the performance of the new infrastructure yet.

Will also do some investigation in the following days.

[irm-codebase]

Thanks for giving this priority!

I did test with the same dataset every time. A successful download (with older versions) did not seem to help the newer one, but I did not test this thoroughly.

[fneum]

I benchmarked a similar setup in the new CDS infrastructure (https://cds-beta.climate.copernicus.eu/). The old one will be shut down in September. The major aspect that changed between the atlite versions mentioned is that we switched from annual to monthly requests, so I varied these using #372. I varied the x-y coordinates so that CDS would not use cached artifacts, but kept the extent (5°x5°) identical.

import atlite

# annual request

cutout = atlite.Cutout(
    path="cutout-monthly.nc",
    module="era5",
    xs=slice(-10, -5),
    ys=slice(35, 40),
    time=slice("2015-01", "2015-12")
)
cutout.prepare(["wind"], monthly_requests=False)

# completed in 35 minutes

# sequential monthly request

cutout = atlite.Cutout(
    path="cutout-annual.nc",
    module="era5",
    xs=slice(0, 5),
    ys=slice(40, 45),
    time=slice("2015-01", "2015-12")
)
cutout.prepare(["wind"], monthly_requests=True)

# completed in 49 minutes

It indeed seems that reverting to annual requests was faster in the test cases and may make sense for cutouts with a smaller geographical extent; though the exact trade-off is unclear.

One thing I noticed is that the monthly requests are made sequentially rather than in parallel. I don't think this used to be like that in the past and could add to the queuing times.

Will test next if we can do parallel requests and if this achieves a speed-up.

Logs

Here are the request logs from https://cds-beta.climate.copernicus.eu/requests?tab=all (the last item is the annual request):

[fneum]

Using dask.delayed and dask.compute, I investigated if concurrent requests of the time chunks might achieve a speedup.

In era5.get_data:

atlite/atlite/datasets/era5.py

Lines 451 to 453 in d04aff0

	datasets = map(
	retrieve_once, retrieval_times(coords, monthly_requests=monthly_requests)
	)

I changed:

-    datasets = map(
-        retrieve_once, retrieval_times(coords, monthly_requests=monthly_requests)
-    )
+    time_chunks = retrieval_times(coords, monthly_requests=monthly_requests)
+    delayed_datasets = [delayed(retrieve_once)(chunk) for chunk in time_chunks]
+    datasets = compute(*delayed_datasets)

xref: 1866fc3

I tested similar to the cases above:

# concurrent monthly request

cutout = atlite.Cutout(
    path="cutout-monthly-parallel.nc",
    module="era5",
    xs=slice(-5, 0),
    ys=slice(45, 50),
    time=slice("2015-01", "2015-12")
)
cutout.prepare(["wind"], monthly_requests=True)

# completed in 23 minutes

With 23 minutes queueing time, this is even faster. However, it should be noted that the overall queue could have had different lengths and, therefore, the numbers can only be indicative.

The concurrent time chunk requests should be optional though, as too many parallel requests may get you throttled (https://confluence.ecmwf.int/display/CEMS/CDS+-+Best+Practices):

Indeed the CDS system penalises users that submit too many requests, decreasing the priority of their requests. In short: Too many parallel requests will eventually result in a slower overall download time. For this reason, we suggest limiting to a maximum of 10 parallel requests.

This might especially happen if you 5 different features, which already uses dask.delayed to post requests for different features in parallel. Downloading a year will land you at 60 (5 x 12) parallel requests, which is above the recommendation.

Logs

[fneum]

Final follow-up (and with that, I'm quite happy with the solutions proposed in #372).

I tried a bulk concurrent submission with 50 simultaneous requests for different features:

cutout = atlite.Cutout(
    path="cutout-annual-parallel-features.nc",
    module="era5",
    xs=slice(0, 5),
    ys=slice(45, 50),
    time=slice("2015-01", "2015-12")
)
cutout.prepare(
    ["height", "wind", "influx", "temperature", "runoff"],
    monthly_requests=True,
    concurrent_requests=True
)

This one finished in 26 minutes, so it took a comparable time to downloading a single feature concurrently.