Reading netcdf files is slow if there are unlimited dimensions

[tkarna]

Reading arrays from NetCDF files is slow if one dimension is unlimited.

An example: reading an array of shape (5000, 50) takes ~7 s if the first dimension is unlimited. If both dimensions are fixed, it takes ~0.02 s. This is a major bottleneck if many (100s) of such files need to be processed. Time dimension is often declared unlimited in files generated by circulation models.

Test case:

import iris
import time

f = 'example_dataset.nc'
var = 'sea_water_practical_salinity'

tic = time.process_time()
cube = iris.load_cube(f, var)
cube.data
duration = time.process_time() - tic
print('Duration {:.3f} s'.format(duration))

The input NetCDF file can be generated with:

import iris
import numpy
import datetime

ntime = 5000
nz = 50
dt = 600.
time = numpy.arange(ntime, dtype=float)*dt
date_zero = datetime.datetime(2000, 1, 1)
date_epoch = datetime.datetime.utcfromtimestamp(0)
time_epoch = time + (date_zero - date_epoch).total_seconds()
z = numpy.linspace(0, 10, nz)
values = 5*numpy.sin(time/(14*24*3600.))
values = numpy.tile(values, (nz, 1)).T

time_dim = iris.coords.DimCoord(time_epoch, standard_name='time',
                                units='seconds since 1970-01-01 00:00:00-00')
z_dim = iris.coords.DimCoord(z, standard_name='depth', units='m')
cube = iris.cube.Cube(values)
cube.standard_name = 'sea_water_practical_salinity'
cube.units = '1'
cube.add_dim_coord(time_dim, 0)
cube.add_dim_coord(z_dim, 1)
iris.fileformats.netcdf.save(cube, 'example_dataset.nc',
                             unlimited_dimensions=['time'])

Profiling suggest that in the unlimited case, each time slice is being read separately, i.e. NetCDFDataProxy.__getitem__ is being called 5000 times.

Tested with: iris version 2.2.0, Anaconda3 2019.03

[bjlittle]

Thanks for taking the time to report this @tkarna.

I'll see if I can replicate your timings on my side. What version of iris are you using? I'm assuming 2.2.1

[tkarna]

I'm using version 2.2.0 installed with conda.

[tkarna]

For what it's worth I see the same performance with version 2.2.1 and the master branch as well.

[pp-mo]

Could even be a candidate for user-controlled chunking ? see #3333

[bjlittle]

@tkarna I can recreate the behaviour on my side, so thanks for the repeatable example.

That said, after digging a little, it appears that dask is blocking in a sleep...

I'll dig a little further...

For the non unlimited case, the netcdf variable sea_water_practical_salinity has chunking of contiguous i.e. no chunking. However, for the unlimited case, the netcdf variable sea_water_practical_salinity has a default chunking of [1, 50].... hence why it is being read 5000 times. This explains why the load time is chronically slow - note that dask prefers much larger chunk sizes, otherwise the overhead of using dask outweighs the processing of small chunks of data payload, see https://docs.dask.org/en/latest/array-chunks.html#specifying-chunk-shapes

That said, I simply saved the netcdf file with the netcdf chunksizes set specifically to [5000, 50] e.g.

import iris
import numpy
import datetime

ntime = 5000
nz = 50
dt = 600.
time = numpy.arange(ntime, dtype=float)*dt
date_zero = datetime.datetime(2000, 1, 1)
date_epoch = datetime.datetime.utcfromtimestamp(0)
time_epoch = time + (date_zero - date_epoch).total_seconds()
z = numpy.linspace(0, 10, nz)
values = 5*numpy.sin(time/(14*24*3600.))
values = numpy.tile(values, (nz, 1)).T

time_dim = iris.coords.DimCoord(time_epoch, standard_name='time',
                                units='seconds since 1970-01-01 00:00:00-00')
z_dim = iris.coords.DimCoord(z, standard_name='depth', units='m')
cube = iris.cube.Cube(values)
cube.standard_name = 'sea_water_practical_salinity'
cube.units = '1'
cube.add_dim_coord(time_dim, 0)
cube.add_dim_coord(z_dim, 1)
iris.fileformats.netcdf.save(cube, 'example_dataset_unlimited_chunks.nc',
                             unlimited_dimensions=['time'], chunksizes=[5000, 50])

This resulted in a load time of ~0.057s

So either:

• set the chunksizes explicitly to iris.fileformats.netcdf.save
• or align the dask reading chunks with the default netcdf variable chunksizes

Iris will always set the chunksize on loading within dask to align with the chunksize in netcdf, but obviously this can be sub-optimal to dask (and iris) for small netcdf chunksizes

[bjlittle]

@tkarna As an extension, I customised a development version of iris to hardwire the chunks to be [5000, 50] on loading a cube i.e. the custom chunks = [5000, 50] overrides the netcdf chunking specified in the file for the variable sea_water_practical_salinity of [1, 50].

This custom chunks aligns with the underlying [1, 50] default chunking of the netcdf variable with UNLIMITED 1st dimension. This results in a load and touching of the data of ~0.15s.

I get the same good loading/touching performance if I set the custom chunks to -1 or auto.

This suggests to me that it would be useful to give control to the user to specify the chunks and override any chunking that is specified in the netcdf file at the iris API level.

@tkarna Does this help?

[bjlittle]

Ping @pp-mo and @TomekTrzeciak

[tkarna]

Thanks @bjlittle for looking into this. I confirm that the performance hit is caused by chunking. Setting chunks=None in netcdf._get_cf_var_data fixes the issue.

Modifying the input file chunking is not an option for me, as I get these files from an external model and I do not wish to convert them. Iris should be able to read any netCDF file with reasonable performance.

Having the chunk options in the iris load_cube API would be useful. This would fix my issue.

[tkarna]

Another option, perhaps, would be to define better default chunking scheme. For example, if the file chunk only contains < N values, one would revert to chunks=None as (I'd imagine) reading in tiny chunks is never a good idea. This would also be easier for the user as one wouldn't need to tweak with the chunk options.

[bjlittle]

@tkarna I think that there should be a happy compromise between:

1. iris honouring the chunking specified in the netcdf file
2. iris overriding the chunking specified in the netcdf file that is considered too small and will result in a performance hit
3. allowing the user to override iris by specifying the chunks at the API level

iris already performs step 1. above, as we know. Extending iris to support steps 2. and 3. should cover most other use cases, and give enough wiggle room for users to circumvent most chunking issues. Like you said, you're a third-party to the data and can't change the native netcdf chunking of variables in the file, so iris needs to help you easily work around that.

[tkarna]

@bjlittle Yes, that sounds very good indeed.