reduce release memory footprint

[jannikmi]

Hitting the project size limit of 10 GB.
Had to file an issue to request for more project space: pypi/support#7038
Had to delete all pyPI releases up to version 3.4.2 (last version supporting python 2.7) to free up project space

How large is each release?

600 MB in Total

The current footprint of one distribution is around 51 MB.
11 distributions per release: 1 source, 10 wheels

• how can we reduce the release size?
• do we really need (10) wheels?
• is it fine to build wheels only for 1 python version?
• is it fine to build wheels only for 1 linux version?
• are there any other ideas?

[jannikmi]

@theirix what's your take on this?

[theirix]

1. New wheel size

I downloaded the latest 7.0.0 release wheel for Linux (timezonefinder-7.0.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl) The size of a native binary is 32K, which is approximately the size of the timezonefinder.py file (22K).
99.7% of the package size is geo data - 62 Mb. Compressed wheel size is 51Mb, uncompressed is 62Mb - zip saves 14%.

2. Old wheel size

For older (6.5.9) one-geo-data-file wheels, the size of the geo data is 59 Mb of compressed wheel size 49 Mb. So the old package is slightly smaller.

3. Geodata size

Even flatbuffer data can be compressed. A simple tar with lzma (tar cf - data | xz --best | wc -c) gives 22 Mb instead of 62 Mb. Of course, wheel's zip compression won't help now, and the wheel stays the same.

Hence, the savings with this approach are 60%.

The low-hanging fruit - use npz instead of npy (but it's only 40Kb). The largest file is the flatbuffer file. Simply unpack with the built-in lzma compression while reading,

4. Do we really need (10) wheels?

That's a good question. I would provide several of the latest versions for all Linux/Python types. Older versions could be stripped from PyPi with a GHA cron job. Still, it's worth attaching wheels to GitHub releases in case people want to get them. It's entirely reasonable to download them from GitHub releases, even programmatically.

5. musl vs glibc Linux flabours

Generally, musl is quite helpful for lightweight usage. For example, I use it in containers and AWS images

[jannikmi]

Note: #319 asks for additional ARM wheels.

[jannikmi]

Geodata size

Even flatbuffer data can be compressed. A simple tar with lzma (tar cf - data | xz --best | wc -c) gives 22 Mb instead of 62 Mb.

Almost 1/3 the size! That sounds promising, but I wonder if it safe to assume that the client systems have tar installed for unpacking? zipfile states that

Decryption is extremely slow as it is implemented in native Python rather than C.

Perhaps there are other more performant alternatives, which can be used!?

it's worth attaching wheels to GitHub releases in case people want to get them. It's entirely reasonable to download them from GitHub releases, even programmatically.

Good idea. Would it be reasonable to attach all the wheels in the github release and upload only the sdist to pyPI?

[theirix]

Geodata size
Even flatbuffer data can be compressed. A simple tar with lzma (tar cf - data | xz --best | wc -c) gives 22 Mb instead of 62 Mb.

Almost 1/3 the size! That sounds promising, but I wonder if it safe to assume that the client systems have tar installed for unpacking? zipfile states that

In our case, it's enough just to apply compression to one fbs file. The rest is much smaller. So, we don't need tar (it was an example)

Decryption is extremely slow as it is implemented in native Python rather than C.

Perhaps there are other more performant alternatives, which can be used!?

From me experience, zstd is faster.

it's worth attaching wheels to GitHub releases in case people want to get them. It's entirely reasonable to download them from GitHub releases, even programmatically.

Good idea. Would it be reasonable to attach all the wheels in the github release and upload only the sdist to pyPI?

Let me first check the ARM ticket to clarify is the platform-indeendent sdist is usable at all.

In the worst case, we can make a pure timezonefinder package and the binary timezonefinder-binary wheel package

[jannikmi]

For future reference:

• pyzstd, 500+ stars github
• python-zstd 180 stars

[jannikmi]

Update: the current releases have a size of 3x ~28 MB = ~ 80 MB