Incompatibility Issue Between EasyCompressor Versions 1.4.0 and 2.0.2 Resulting in Corrupted Pickle (LZ4)

[nathan5580]

Description:
After updating the EasyCompressor library from version 1.4.0 to 2.0.2 in our project, we encountered an issue related to LZ4 compression where the compressed data appears to be corrupted, yielding the error: "Pickle is corrupted: Version 7 is not recognized". This issue has disrupted the functionality related to file data retrieval in our application, specifically when attempting to decompress data that was compressed with the previous version of EasyCompressor.

Error Message:

Pickle is corrupted: Version 7 is not recognized
stacktrace:
   at K4os.Compression.LZ4.LZ4Pickler.DecodeHeader(ReadOnlySpan`1 source)
   at K4os.Compression.LZ4.LZ4Pickler.Unpickle(ReadOnlySpan`1 source)
   at EasyCompressor.LZ4Compressor.BaseDecompress(Byte[] compressedBytes) in /_/src/EasyCompressor.LZ4/LZ4Compressor.cs:line 200
   at EasyCompressor.BaseCompressor.Decompress(Byte[] compressedBytes) in /_/src/EasyCompressor/Compressors/Base/BaseCompressor.cs:line 80
   at Flux.API.Domain.Services.FilesService.GetFileData(Int32 userId, Int32 fileId) in D:\Projects\ISPB_BACK\Flows\Flux.API\Domain\Services\FilesService.cs:line 77

Steps to Reproduce:

1. Compress data using EasyCompressor version 1.4.0 with LZ4 compression.
2. Update EasyCompressor to version 2.0.2.
3. Attempt to decompress the previously compressed data using the updated version.

Expected Behavior:
The data compressed with version 1.4.0 should be decompressed successfully with version 2.0.2 without any errors or data corruption.

Actual Behavior:
The decompression process fails, and the application throws an error indicating that the pickle version is not recognized.

Questions:

• Is there a compatibility issue between the LZ4 compression used in versions 1.4.0 and 2.0.2 of EasyCompressor?
• How can we enable or use a legacy mode for LZ4 compression to ensure compatibility and successful decompression of data compressed with earlier versions of EasyCompressor?

Environment:

• EasyCompressor version: 2.0.2 (upgraded from 1.4.0)
• Runtime: (.Net 8.)
• OS: (W10)

We would appreciate any guidance or suggestions on how to address this issue to ensure smooth data handling across different versions of EasyCompressor. Thank you for your support.

[mjebrahimi]

Hi @nathan5580

Thank you for your comprehensive report.

Reason

According to the changelog for v2.0.2, the default compression mode of LZ4 has been changed (to optimize the performance and memory allocation)

• Three new LZ4 binary compression modes added:

  • LZ4BinaryCompressionMode.Optimal (new Default mode)
    Default compression mode. (NOT compatible with other modes neither LegacyCompatible nor StreamCompatible modes)
    But it's fast and most efficient in memory allocation. (Best Performance overall - Fast_GCEfficient)
    It applies only to binary Compress/Decompress and does not affect Stream/Stream[Async] methods.

  • LZ4BinaryCompressionMode.LegacyCompatible (legacy Default mode)
    Legacy compatibility with old/legacy versions. (NOT compatible with other modes neither StreamCompatible nor Optimal modes)
    It's the fastest mode (a bit faster than Optimal) but less efficient in memory allocation. (Fast_GCInefficient)
    It prepends 4 bytes to the beginning of the array to define the original array length.
    It applies only to binary Compress/Decompress and does not affect Stream/Stream[Async] methods.

  • LZ4BinaryCompressionMode.StreamCompatible
    StreamCompatible which is compatible with Stream's output. (NOT compatible with other modes neither LegacyCompatible nor Optimal modes)
    It's slower than other modes but moderate in memory allocation. (Slow_GCModerated)

Benchmark

You can see the comparison benchmark between these 3 modes (HTML and Image)

Solution

In order to mitigate this breaking change, you just need to set the compression mode to LZ4BinaryCompressionMode.LegacyCompatible

//Configure Services
//Use and overload that accepts the LZ4BinaryCompressionMode enum value
services.AddLZ4Compressor(LZ4BinaryCompressionMode.LegacyCompatible)

//Or if you are instantiating LZ4Compressor directly
//Use a constructor overload that accepts the LZ4BinaryCompressionMode enum value
var compressor = new LZ4Compressor(LZ4BinaryCompressionMode.LegacyCompatible);

[nathan5580]

Hi @mjebrahimi ,

Thank you very much for the detailed and informative response.

It turns out I was attempting the update in the wrong project, where the specified configurations do not apply. After re-evaluating and applying your suggested solution to the correct project, I can confirm that setting the compression mode to LZ4BinaryCompressionMode.LegacyCompatible resolved the issue perfectly, and everything is working as expected now.

I appreciate your swift support and the clarity of your explanation, which guided me to the right solution.

Thanks again for your help!