Anything that helps reduce the environmental impacts of LLMs is a good thing.
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).https://github.com/microsoft/BitNet #BitNet
The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices.
GitHubGitHub - microsoft/BitNet: Official inference framework for 1-bit LLMsOfficial inference framework for 1-bit LLMs. Contribute to microsoft/BitNet development by creating an account on GitHub.
Behold, the groundbreaking 587th iteration of 
. Remember, it's not a real tech report without a job ad and some grateful acknowledgments for funding! 
Just discovered BitNet b1.58, a new kind of Large Language Model (LLM). Instead of using standard 16-bit numbers, its weights are simplified to just -1, 0, or 1 (around 1.58 bits)! 
Built on 
Compatible with existing 1-bit models from 
