We’re excited to introduce the release of Intel® Gaudi® software version 1.16.0, bringing forth numerous enhancements and updates for an improved GenAI development experience. We added preview support for vLLM, a fast and easy-to-use library for LLM inference and serving. Performance optimizations will be added in subsequent releases. For more information see the Intel Gaudi vLLM .
Added preview support for Mixture of Experts (MoE) BF16 pretraining using Megatron-DeepSpeed. This release includes a recipe for pretraining Mixtral 4x7B and Mixtral 8x7B. For more information see Megatron-DeepSpeed models.
We have improved the performance of various LLM models, including LLaMA 2 7B/13B/70B FP8 for pre-training, and LLaMA 2 7B/70B FP8 and Mistral 7B for inference. For more information, check out the Intel Gaudi model performance page.
In this release, we’ve also upgraded the validated versions of several libraries, including PyTorch 2.2.2, DeepSpeed 0.14.0, PyTorch Lightning 2.2.4, Megatron-DeepSpeed PR #372, and Ray 2.20. Lastly a reminder that support for Debian 10.10 will be deprecated following 1.16 release. In subsequent releases, Gaudi Lazy mode execution will be deprecated, and PyTorch Eager mode and torch.compile will be the default execution mode.
You can find more information on the Gaudi Software 1.16.0 release on the Intel Gaudi release notes page.
