Possible concurrency issue during deflate on libflate 0.1.25 #41
Description
I'm using zip-rs with rayon to zip many files concurrently (I think 16x? Whatever rayon ends up using on this machine) This works fine with compression level store, and with bzip2. (Bzip2 is also way, way faster, 150+MB/s instead of 10MB/s, but that is.... another issue.)
All of the actual zipping is happening inside each loop iteration that is parallelized: open the source and destination files, io::copy, close the file when done.
When not using par_iter it completes successfully (but takes 14 hours to do so on a small test dataset of 283GB.)
With deflate (which uses this library,) I run into the below stack trace eventually, on win64.
It takes about an hour or so to run into the issue and doesn't seem to be on the same file every time, so reducing this test case seems difficult.
I've also gotten an illegal instruction once or twice which makes me think there is undefined behavior somewhere in this lib.
Thread '<unnamed>' panicked at 'index out of bounds: the len is 15 but the index is 15', /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\src\libcore\slice\mod.rs:2695:10
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
thread '<unnamed>' panicked at 'index out of bounds: the len is 15 but the index is 15', /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\src\libcore\slice\mod.rs:2695:10
stack backtrace:
0: 0x7ff62b6a7aad - std::sys::windows::backtrace::set_frames
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\sys\windows\backtrace\mod.rs:95
1: 0x7ff62b6a7aad - std::sys::windows::backtrace::unwind_backtrace
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\sys\windows\backtrace\mod.rs:82
2: 0x7ff62b6a7aad - std::sys_common::backtrace::_print
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\sys_common\backtrace.rs:71
3: 0x7ff62b6aacad - std::sys_common::backtrace::print
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\sys_common\backtrace.rs:59
4: 0x7ff62b6aacad - std::panicking::default_hook::{{closure}}
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:197
5: 0x7ff62b6aa9aa - std::panicking::default_hook
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:211
6: 0x7ff62b6ab53f - std::panicking::rust_panic_with_hook
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:474
7: 0x7ff62b6ab073 - std::panicking::continue_panic_fmt
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:381
8: 0x7ff62b6aaf58 - std::panicking::rust_begin_panic
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:308
9: 0x7ff62b6b8b1b - core::panicking::panic_fmt
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libcore\panicking.rs:85
10: 0x7ff62b6b8ad9 - core::panicking::panic_bounds_check
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libcore\panicking.rs:61
11: 0x7ff62b54505c - <core::iter::adapters::Rev<I> as core::iter::traits::iterator::Iterator>::try_fold::h4df698d468c8a2ab
12: 0x7ff62b537e30 - <libflate::deflate::symbol::DynamicHuffmanCodec as libflate::deflate::symbol::HuffmanCodec>::save::h0b55c10f5385fd10
13: 0x7ff62b54023a - libflate::deflate::encode::Encoder<W>::new::h14c0d67a7d5ae201
14: 0x7ff62b540664 - libflate::deflate::encode::Encoder<W,E>::finish::h5e76f2387f1275f7
15: 0x7ff62b53ca1e - zip::write::ZipWriter<W>::finalize::h7a22fb7f667ff968
16: 0x7ff62b53bbc4 - zip::write::ZipWriter<W>::start_file::hae527b1388e644b5
17: 0x7ff62b53be81 - zip::write::ZipWriter<W>::finalize::h7a22fb7f667ff968
18: 0x7ff62b53da4e - <zip::write::ZipWriter<W> as core::ops::drop::Drop>::drop::h457d134e7a1b2002
19: 0x7ff62b52c73f - core::ptr::real_drop_in_place::h8b27e2444c24fda3
20: 0x7ff62b52ec4a - mame_coalesce::rom::zip::write_zip::hc9fe388a39a2a3b3
21: 0x7fff553e102f - <unknown>
22: 0x7fff553e3595 - is_exception_typeof
23: 0x7fff553ebb23 - _C_specific_handler
24: 0x7fff553e2fec - is_exception_typeof
25: 0x7fff553ebfe0 - _CxxFrameHandler3
26: 0x7fff6aec47fe - _chkstk
27: 0x7fff6ae2600b - RtlUnwindEx
28: 0x7fff553ebe48 - _C_specific_handler
29: 0x7fff553e2688 - is_exception_typeof
30: 0x7fff553e29e2 - is_exception_typeof
31: 0x7fff553e30f2 - is_exception_typeof
32: 0x7fff553ebfe0 - _CxxFrameHandler3
33: 0x7fff6aec477e - _chkstk
34: 0x7fff6ae24bee - RtlWalkFrameChain
35: 0x7fff6ae289e5 - RtlRaiseException
36: 0x7fff67d99128 - RaiseException
37: 0x7fff553e486c - CxxThrowException
38: 0x7ff62b6b0777 - panic_unwind::imp::panic
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libpanic_unwind\seh.rs:281
39: 0x7ff62b6b0777 - panic_unwind::__rust_start_panic
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libpanic_unwind\lib.rs:101
40: 0x7ff62b6ab717 - std::panicking::rust_panic
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:523
41: 0x7ff62b6ab5ed - std::panicking::rust_panic_with_hook
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:494
42: 0x7ff62b6ab073 - std::panicking::continue_panic_fmt
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:381
43: 0x7ff62b6aaf58 - std::panicking::rust_begin_panic
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libstd\panicking.rs:308
44: 0x7ff62b6b8b1b - core::panicking::panic_fmt
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libcore\panicking.rs:85
45: 0x7ff62b6b8ad9 - core::panicking::panic_bounds_check
at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c\/src\libcore\panicking.rs:61
46: 0x7ff62b54505c - <core::iter::adapters::Rev<I> as core::iter::traits::iterator::Iterator>::try_fold::h4df698d468c8a2ab
47: 0x7ff62b537e30 - <libflate::deflate::symbol::DynamicHuffmanCodec as libflate::deflate::symbol::HuffmanCodec>::save::h0b55c10f5385fd10
48: 0x7ff62b54023a - libflate::deflate::encode::Encoder<W>::new::h14c0d67a7d5ae201
49: 0x7ff62b53fadd - <libflate::deflate::encode::Encoder<W,E> as std::io::Write>::write::h11ec20597a5b3e80
50: 0x7ff62b53ab33 - std::io::Write::write_all::h9a21696f30eac3c4
51: 0x7ff62b538631 - std::io::util::copy::h7fe9620ed861a5de
52: 0x7ff62b52e3a2 - mame_coalesce::rom::zip::write_zip::hc9fe388a39a2a3b3