read() returns incomplete file? (not clear on documentation)

[fernandobrito]

Hello,

This is my first time using the library, so please apologize me if I'm missing something obvious.

I'm trying to read a .csv.gz file (quite small, less than 100kb) following the documentation (https://gcsfs.readthedocs.io/en/latest/#examples), by doing:

fs = gcsfs.GCSFileSystem(project=project_id, token=service_key_path)
uri = bucket_name + '/' + filename
with fs.open(uri, 'rb') as f:
    content = f.read()

Here, I expect content to have the entire file, but it has only 168 rows (out of 946 in total). Reading the same file with google-cloud-storage works fine.

storage_client = storage.Client(credentials=credentials, project=project_id)
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(filename)
content = blob.download_as_string()

What am I missing?
Thanks

[martindurant]

That's certainly not what's supposed to happen, and not what happens in other people's experience. I cannot think of what might be happening, so any debugging and poking around the API you might do would be appreciated. Can you make the file for which this is happening public (or other similar file with random data) ?

(cc @TomAugspurger )

[fernandobrito]

@martindurant Hey, thanks for the quick response! I had time to do some investigation this morning, and the issue that I described happens when I upload files to GCS bucket using gsutil cp -Z (for compression: https://cloud.google.com/storage/docs/gsutil/commands/cp#options_1)

Somehow, gcsfs is able to decompress the file correctly, as I can see its content, but it stops before the end of the file.

How to reproduce:

Run the script below (public bucket) and then change to filename = dummy.csv.
It's the same file in my computer, but one was uploaded with gsutil cp -Z and the other one without -Z.

from google.cloud import storage
import gcsfs

bucket_name = 'fernando-playground-temp'
filename = 'dummy.compressed.csv'

storage_client = storage.Client.create_anonymous_client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(filename)

content = blob.download_as_string()
print(len(content))
print(content[0:20])

print('--------')

fs = gcsfs.GCSFileSystem(token='anon')
uri = bucket_name + '/' + filename
with fs.open(uri, 'rb') as f:
    content = f.read()
    print(len(content))
    print(content[0:20])

Output for compressed:

34134
b';{installed_at};{cre'
--------
2597
b';{installed_at};{cre'

Output for non-compressed:

34134
b';{installed_at};{cre'
--------
34134
b';{installed_at};{cre'

[TomAugspurger]

Note that the length of the result from GCSFile.read() matches the size

In [52]: len(content)
Out[52]: 2597

In [53]: f.details['size']
Out[53]: 2597

Which is the size reported by GCS

(Pdb) pp page['items'][0]
{'bucket': 'fernando-playground-temp',
 'cacheControl': 'no-transform',
 'contentEncoding': 'gzip',
 'contentLanguage': 'en',
 'contentType': 'text/csv',
 'crc32c': 'QppSQg==',
 'etag': 'CLzD3belmecCEAE=',
 'generation': '1579767739933116',
 'id': 'fernando-playground-temp/dummy.compressed.csv/1579767739933116',
 'kind': 'storage#object',
 'md5Hash': 'ihvA+gonzoj8w418KmzrLQ==',
 'mediaLink': 'https://www.googleapis.com/download/storage/v1/b/fernando-playground-temp/o/dummy.compressed.csv?generation=1579767739933116&alt=media',
 'metageneration': '1',
 'name': 'dummy.compressed.csv',
 'selfLink': 'https://www.googleapis.com/storage/v1/b/fernando-playground-temp/o/dummy.compressed.csv',
 'size': '2597',
 'storageClass': 'STANDARD',
 'timeCreated': '2020-01-23T08:22:19.932Z',
 'timeStorageClassUpdated': '2020-01-23T08:22:19.932Z',
 'updated': '2020-01-23T08:22:19.932Z'}

Perhaps we can't trust that size when the contentEncoding is compressed (gzip in this case?)

[martindurant]

If you gzip the file yourself and upload without the content encoding set, do you get the right amount of bytes back when you then read it?

[fernandobrito]

@martindurant yes. I uploaded dummy.compressed.local.csv.gz to the bucket and the right amount of bytes came back.

[TomAugspurger]

@fernandobrito are you able to look into this any more? I'm not sure exactly where the fix should be, but I think we're treating the compressed size as the total uncompressed size somewhere. It might be in gcsfs or in fsspec.

[martindurant]

So it appears that GCS is returning the size-on-disc, but the download is giving the total uncompressed bytes. I am assuming that with gzip encoding, you cannot do seek (bytes range requests), so this is starting to look a lot like the unknown HTTP case.
It is annoying, because if the file were not compressed on disc, the transfer would probably be gzip compressed anyway, the same bytes, but the reported size would be the same.

[martindurant]

So, do we need to explicitly say that ContentEncoding=(compression) , i.e., auto-compression at rest, is not supported by gcsfs? The download itself is always compressed (gzip or deflate) regardless, but such files cannot be random-accessed.