Bug during compression

[jaime-olivares]

Hi there, I have been using your library and ported it to C# (see https://github.com/jaime-olivares/bzip2). After processing thousand of files, we found some few that throw an error. We have tried that in Java and the C# ported library as well. Please let me know if I can email you the Java snippet and the conflicting sample file.

[jaime-olivares]

Any chance to see this? The library will not be reliable until this case is solved. My hunch is that the issue arises whenever there are too much repeated and consecutive values.

[drone1400]

Perhaps I am committing an act of necromancy by posting here, but I have forked jaime-olivares's implementation ( https://github.com/drone1400/bzip2 ) because I liked how fast it was compared to others I had tried before.

So far I have not been able to reproduce this old bug. Granted, I have not tested it on thousands of real files, although I did run some thousands of randomized data tests and randomized data with repeated consecutive values of random lengths.

If anyone comes across this fabled compression bug I'd be interested in hearing about it.

[jaime-olivares]

Hi @drone1400,
Unfortunately, I don't have the conflicting file anymore. I can barely remember the error occurring for files with a long trail of zeroes (256? 1K, 10K?) don't remember.

[drone1400]

@jaime-olivares Hello! I have just tried testing this on a file with well over 10k of trailing 0s, and there was no issue. I also tried adding some non zero data at the end after the zeros, and it was still fine.

Then I tried extending a file with about 100MB worth of zero bytes, my thought was that since a block has up to 900000 bytes, then it should fit 180000 RLE sequences of 255 zero bytes (or whatever other same byte value), which is about ~45MB worth of repeated byte values. Figured I'd try double that just in case?...

I also tried running some tests with a random byte value repeated for 100 million times, no issues there either.

For now, until I can find some example that replicates the issue, I can only assume that the bug somehow mysteriously vanished somehow?...

One thing I must note, I have been running these tests under .NET6. However, I doubt this would be something that only popped up in .NET Framework since you mentioned the bug manifested in both Java and C#, so it sounds like it was an algorithm issue not a runtime issue / mysterious garbage collector issue or whatever else...