Understanding Deflate

https://jjrscott.com/to-deflate-or-not/

Great to see the community building powerful tools on Blosc2!

Check out compress-image: a new C++/Python library for working with compressed images directly in memory.

It allows you to keep lots of images in RAM while minimizing I/O and memory footprint.

Kudos to Emil Dohne for this fantastic work!

Project here: https://github.com/EmilDohne/compressed-image

Xz format inadequate for general use

https://www.nongnu.org/lzip/xz_inadequate.html

#PeaZip 10.6.1 has been released (#7Z / #7ZIP / #ARC / #PEA / #RAR / #TAR / #ZIP / #ZIPX / #zstd / #Zstandard / #FileArchiver / #DataCompression) https://peazip.github.io/

#7Zip 25.01 has been released (#SevenZip / #7z / #LZMA / #LZMA2 / #ZIP / #RAR / #FileArchiver / #DataCompression) https://7-zip.org/

Blosc2 is incredibly powerful, but its flexibility comes with a challenge: navigating thousands of possible compression parameter combinations (codecs, levels, filters) to find the best one for your data.

Stop guessing and start optimizing. Save valuable time and unlock the full potential of your data compression pipeline with Btune

https://ironarray.io/btune

Compress Better

#WinRAR 7.12 has been released (#RAR / #ZIP / #ZIPX / #7Zip / #7z / #GZip / #zstd / #Zstandard / #FileArchiver / #DataCompression / #DarkMode) https://rarlab.com/

#Blosc2 now runs directly in your browser! Leveraging the power of #WASM, #Pyodide, and #JupyterLite, you can harness efficient, adaptable compression through the web's universal interface. Experience the future of large-scale data processing without leaving your browser window.

Compress Better, Compute Bigger, Share Faster

#7Zip 25.00 has been released (#SevenZip / #7z / #LZMA / #LZMA2 / #ZIP / #RAR / #FileArchiver / #DataCompression) https://7-zip.org/

#PeaZip 10.5.0 has been released (#7Z / #7ZIP / #ARC / #PEA / #RAR / #TAR / #ZIP / #ZIPX / #zstd / #Zstandard / #FileArchiver / #DataCompression) https://peazip.github.io/

C-Blosc2 2.19.0 is out!
We’ve added b2nd_expand_dims(), making it easy to add new dimensions to your b2nd arrays—perfect for evolving your data structures on the fly.
Big thanks to @lshaw8317 for the contribution!

Check out the release notes: https://github.com/Blosc/c-blosc2/blob/main/RELEASE_NOTES.md

#C-Blosc2 #DataCompression #OpenSource

C-Blosc2 2.18.0 is out now!

What's new:

* Introducing b2nd_concatenate() - now you can easily join b2nd arrays together!

* Fixed mmap files to flush modified pages only in write mode (thanks Jan Sellner!)

Get the full details: https://github.com/Blosc/c-blosc2/blob/main/RELEASE_NOTES.md

wow, if I'm reading the graphs right, 'zstd' long-range matching mode blows 'lrzip' on default settings out of the water in terms of performance

https://github.com/facebook/zstd/releases/tag/v1.3.2

(and that's for the 2017 release, who knows how well it performs nearly 8 years later!)

#Python-Blosc2 is hitting 1 million weekly downloads on PyPI! https://pypacktrends.com/?packages=blosc2&packages=blosc&time_range=2years

Users are rapidly adopting #Blosc2, which now accounts for over 95% of downloads compared to Blosc1. This success is thanks to our amazing users and community contributors. We're dedicated to making Python-Blosc2 even better.

Our motto: Compress Better, Compute Bigger!

#libzip 1.11.4 has been released (#LZMA / #ZIP / #Zip64 / #zstd / #Zstandard / #bzip2 / #AES / #PKWARE / #FileArchiver / #DataCompression) https://libzip.org/

SIMD blog series: @folkertdev shows examples of using SIMD in the zlib-rs project.

Part 2 explains what to do when the compiler is not capable of using the SIMD capabilities of modern CPUs effectively. We end up with a basic, but very effective, example of a custom SIMD implementation beating the compiler.

https://tweedegolf.nl/en/blog/155/simd-in-zlib-rs-part-2-compare256

@trifectatech

New #blog post: Unix Data Compression Shootout

https://rldane.space/unix-data-compression-shootout.html

881 words

cc: my wonderful #chorus: @joel @dm @sotolf @thedoctor @pixx @twizzay @orbitalmartian @adamsdesk @krafter @roguefoam @clayton @giantspacesquid

(I will happily add/remove you from the chorus upon request! :)

#100DaysToOffload #32

So @rl_dane introduced #bzip3 to me to use instead of #bzip2. Let's turn some bz2 files into bz3 to see the difference.

First example: 90k opus files

hey snips wake word dataset. It has ~90k opus files and a tar file of 3.1GB. bzip2 produces the same 3.1GB which is as expected. bzip3 created 3.0GB but used tons of computation power. Not worth the 100MB

Second example: Windows 7 virtual box VM image

Windows7.vdi it's Windows 7 VM image for the "special" days. I think I have to get rid of it. But while it is still there, let's see how each will perform. It is 16GB uncompressed. bzip2 -9 is 7.0GB. bzip3 is 6.3GB but at the expense of like 3x CPU time. Deleting all of them anyway. Down with Windows.

Third example: Pure XML text file

Pure XML file. It's Persian and English characters. Uncompressed is 1.7GB. bzip2 -9 is 276M while bzip3 is 260MB

Final example: Creating a simple bomb

So I did this:

dd if=/dev/zero of=./justzero bs=2G count=6

So now I have a 16GB with only zero bytes. bzip2 -9 is 672KB. bzip3 is 46KB.

Conclusion

Thank you @rl_dane

Real nice thing!

Python-Blosc2 3.3.3 is out!

This release brings bug fixes & optimizations, including improved string lazy expression chaining and a C-Blosc2 update fixing Windows mmap issues (thanks @JanSellner!).

More info: https://github.com/Blosc/python-blosc2/releases

Get the latest: pip install blosc2 --update

Backbit Impawsible Ep. 19 - Optimizing C64 Video with 128 Character Frames
#C64 #BackBitImpawsible #PETSCII #8BitCoding #RetroDev #Commodore64 #ChiptuneGraphics #CharacterSetHacks #8bitVideo #Commodore64 #DataCompression
https://theoasisbbs.com/backbit-impawsible-ep-19-optimizing-c64-video-with-128-character-frames/?feed_id=3293&_unique_id=68249453b62b3