Ggml v1.0.0

v1.0.0

Choose a tag to compare

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

No results found

Overview

ggml 1.0.0 prunes the deprecated public API and the legacy DIAG_MASK ops, adds configurable YaRN truncation and FP8 support, and brings broad performance work across backends: few-row quantized matmuls and MoE fusion on CUDA/Metal, a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels, expanded Hexagon op coverage, and scheduler improvements that avoid uploading unused inputs.

API changes

• Pruned deprecated API and removed the legacy DIAG_MASK_INF/DIAG_MASK_ZERO ops, bumping the RPC protocol to reject incompatible peers (llama/29801, llama/31946)

• Added ggml_rope_set_truncate/ggml_rope_get_truncate to control YaRN correction-bound rounding (llama/30206, llama/30802)

• Implemented the backward pass for TRI (llama/30712)

• Added MUL_MAT src1 precision tests and gating for ggml_prec_set_acc (llama/31896, llama/31851)

• Bumped the RPC protocol to v9 with faster loading and graceful failure on unreachable endpoints or failed handshakes (llama/29413, llama/30415)

Core changes

• Scheduler skips uploading inputs that are not read and prefetches only inputs produced by earlier splits (llama/31419, llama/32100)

• MoE cache can compute a share of the cache misses on the CPU and copy the results back asynchronously (llama/30966)

• Meta backend raises the compute view limit for buffers with few tensors (llama/31447)

Backend changes

CUDA

• Added few-row MMQ for batch sizes 2-24 and MMVF for BF16 few-row matmuls (llama/31533, llama/31959)

• Added OCP FP8 support and fused NVFP4 gate/up matmuls with GLU in MMQ (llama/28413, llama/31296)

• Extended FlashAttention with an sm70 head-size-256 config and MMA head sizes 40/72 (llama/27997, llama/30285)

• Improved graph handling: skip compat checks on reuse, re-instantiate on cluster-dim changes, move graph state (llama/32004, llama/31284, llama/32001)

• MoE: skip empty MMQ stream-k tiles, linearize the MoE layout, and sanitize post-bias NaNs (llama/32067, llama/31047, llama/31005)

• Added a debug buffer pool and halved the VMM pool reservation with reserve retries (llama/31919, llama/30542)

• HIP: use a fixed BLAS workspace and add an MFMA lightning indexer for CDNA2 (llama/31838, llama/29050)

• Added a d_state=64 SSM_SCAN fast path and made radix Top-K deterministic for ties (llama/31818, llama/31084)

Metal

• Fused MoE routing and kept fusable node groups inside a single command buffer (llama/31615, llama/31688)

• Made radix Top-K deterministic for ties, added mul_mm for 32<=K<64, and switched TRI to select (llama/31084, llama/30982, llama/30711)

• Fixed GROUP_NORM for ne03>1 and out-of-bounds accesses in the FA output and SOLVE_TRI kernels (llama/31852, llama/31662, llama/32103)

Vulkan

• Added an Intel FlashAttention prefill kernel (llama/29357)

• Added batched tensor copies, a test-fusion interface, and ReBAR detection (llama/31418, llama/30769, llama/31658)

• Enabled dequantized KV with CM2, YaRN truncation, RDNA3/4 large-tile split-K, and large coopmat matmul on newer AMD drivers (llama/31526, llama/30802, llama/30280, llama/31587)

• Added a tiled-transpose concat shader and FWHT kernels for block widths above 512 (llama/30591, llama/29772)

• Fixed MoltenVK out-of-bounds accesses, GROUP_NORM workgroup limits, and FA scratch sizing (llama/31890, llama/31930, llama/30487)

• Capped scalar FlashAttention at 8 rows for large heads on Qualcomm (llama/30829)

SYCL

• Added a device-side MoE scheduler (llama/30156)

• Made the oneDNN FA cache thread-safe and un-gated fused SDPA for Alchemist at head_dim=64 (llama/31138, llama/31574)

• Cached the device id and reject buffers that would exceed VRAM (llama/31126, llama/32013, llama/31285)

• Added exact rounding for CPY to Q4_1/Q5_1/Q8_0, strided CLAMP, and f16 SWIGLU_OAI (llama/31465, llama/31784, llama/30857)

• Fixed AOT for allocations above 4GB (llama/31873)

OpenCL

• Added decomposed Adreno FlashAttention and improved decode performance (llama/32046, llama/31633)

• Added q6_k tiled, q8_0, and f32/q5_k multi-column GEMV kernels and f32 MUL_MAT_ID (llama/31758, llama/30947, llama/31783)

• Added arange, non-contiguous sigmoid, repeat for f16/i32/i16/bf16, and DSV4 HC pre/post kernels (llama/31829, llama/31687, llama/31782, llama/31623)

• Enabled multi-column GEMV by default for Adreno X2 (llama/31694)

WebGPU

• Added batched tensor API support and returned maxBufferSize from get_max_size (llama/31473, llama/31348)

• Added GROUP_NORM, MEAN, COUNT_EQUAL, and PAD_REFLECT_1D kernels (llama/31490, llama/31591, llama/31578, llama/31877)

• Supported the YaRN truncation config (llama/30802)

Hexagon

• Added Q5_0/Q5_1 support and optimized conv_2d while adding conv_3d (llama/32094, llama/31814)

• Fused post-norm residual add into RMS_NORM_MUL and added set_tensor_n (llama/31822, llama/31744)

• Added automatic libhtp loading and fixed 64-bit FastRPC handle truncation (llama/31781, llama/30952)

• Fixed the host build with clang-cl on Windows ARM64 (llama/32011)

CPU

• Switched MUL_MAT to F32 src1 and accumulate FlashAttention V in F32 (llama/31854, llama/29810)

• Added SpacemiT GATED_DELTA_NET RVV and Q4_1 32x256 HP kernels and enabled IME2 MXFP4 repack (llama/31897, llama/31929, llama/31776)

• Enabled the 4x4 simd_gemm tile on ARM NEON and added bf16/q8_0 VXE to llamafile (llama/32128, llama/31866)

CANN

• Fail early when a buffer type is unsupported (llama/31270)

ZDNN

• Accept non-zdnn tensors, invalidate compute tensors after each op, and fix release backend lib detection (llama/31824, llama/31905)

OpenVINO

• Supported the YaRN truncation config (llama/30802)

More info

• Releases and versioning of ggml-org projects

Changelog since v0.29.0

417a024 ggml : bump version to 1.0.0 (#1690) 771b3d9 requirements : fix numpy for python 3.14 (#1691) f154959 sync : llama.cpp ec04faf fix(hexagon): make host build work with clang-cl on Windows ARM64 (llama/32011) b45738e examples : adapt to removal of ggml_diag_mask_inf 1514439 sync : llama.cpp 5f29555 ggml : prune (llama/29801) 076861c hexagon: add Q5_0 and Q5_1 support (llama/32094) 63ec52d Title: CUDA: few-row MMQ for BS 2-24 (llama/31533) c6fe796 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (llama/31897) 4ae6e72 vulkan : Intel FA kernel for prefill. (llama/29357) b22071c ggml-cpu: use the 4x4 simd_gemm tile on ARM NEON (llama/32128) ba8b013 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (llama/26048) 05f605a ggml-backend : prefetch only inputs computed by earlier splits (llama/32100) b8a5b87 ggml : dispatch SpacemiT Q4_1 32x256 through the HP kernel (llama/31929) 7a8cef5 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (llama/31138) 6dfea1a sycl : fix AOT for more than 4GB (llama/31873) 012ff44 sycl : use cached device-id in set_main_device (llama/32013) 2f9d47b CUDA: add cuda debug buffer pool (llama/31919) 522d257 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (llama/30947) 0f9dcb7 CUDA: skip empty MoE tiles in MMQ stream-k fixup (llama/32067) 6bed9a8 opencl: add decomposed FA for Adreno (llama/32046) a357e7b metal : prevent out-of-bounds reads in SOLVE_TRI kernel (llama/32103) c36ccd8 vulkan: support batched tensor copy API (llama/31418) 1bffc59 backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (llama/31284) 8153155 ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (llama/31838) ad7d200 cpu: accumulate flash attention V in F32 (llama/29810) 7f4a2db webgpu: add PAD_REFLECT_1D support (llama/31877) 0f51454 vulkan: allow dequantized KV optimization with CM2 (llama/31526) eb4de49 vulkan: allocate the device fault vendor binary buffer (llama/32008) 6b7f47a cuda : use MMVF for BF16 matrices of few rows at small batch sizes (llama/31959) 77d01f1 cuda : add sm70 FlashAttention config case for DKQ=256, DV=256, ncols=64 (llama/27997) bd37ba0 cuda : fattn mma head sizes 40 72 (llama/30285) 161ca15 CUDA: Add OCP FP8 support (llama/28413) 06faf80 sycl : fixed sycl CLAMP to actually support strided view (llama/31784) a986181 cuda : remove unused ids and type_src0 parameters (llama/32018) 55104b2 llama : Add FP8 KV-cache support (llama/28392) 0753641 ggml : add MUL_MAT src1 precision tests and gating (llama/31896) 1cd16bd cuda : remove unused quantize_cuda_t typedef (llama/31884) 4a18cdd cuda : halve VMM pool reservation size and retry on cuMemAddressReserve failure (llama/30542) 1c9ac9d CUDA: skip graph compat check when graph is being reused (llama/32004) 999c725 ggml : fix FreeBSD Vulkan build by setting _XOPEN_SOURCE=700 (llama/31701) 47bc1d8 CUDA: move graph and concurrent stream state out of common.cuh (llama/32001) df70e12 cann: fail early when checking if buffer type is supported (llama/31270) e447eb5 sycl: Add device-side MoE scheduler (llama/30156) 9a1c582 vulkan: add tiled-transpose concat shader for Qwen models (llama/30591) 8b3c8cf ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (llama/30415) f860f1d vulkan: Divide GROUP_NORM workgroup to 2D to avoid maxComputeWorkGroupCount limit (llama/31930) eb5571a vulkan: Fix out of bounds reads/writes on MoltenVK (llama/31890) 5a9a264 vulkan: cap scalar flash attention at 8 rows for large heads on Qualcomm (llama/30829) 5610f41 sycl: un-gate oneDNN fused SDPA for Alchemist at head_dim=64 (llama/31574) 487cadf hexagon: fix 64-bit FastRPC handle truncation on close (llama/30952) be37c9c ggml-zdnn: accept non-zdnn tensors, invalidate compute tensors after every op (llama/31824) 294d5fc ggml : implement the backward pass for TRI (llama/30712) e18c2c6 ggml-backend: don't upload inputs that are not read (llama/31419) e84d36b metal : prevent out-of-bounds writes to the Metal FA output buffer (llama/31662) 0dd184f metal : make ggml_tri select instead of scaling by the comparison (llama/30711) b6d482c opencl : include climits (llama/31935) 62de780 RPC: Improve loading times (llama/29413) a7276d0 ggml : support YaRN truncation config on openvino, sycl, vulkan, webgpu, rpc (llama/30802) da89459 sycl: cache device-id (llama/31126) fb57085 tests: bound the test_rope tolerance by the f32 angle resolution (llama/31874) e43d847 opencl: add f32 MUL_MAT_ID (llama/31783) d1c1e4b ggml-zdnn: fix release binary not detecting backend lib (llama/31905) 7e7b6e9 mtmd: refactoring: remove clip.h, consolidate code into mtmd (llama/31865) 5a0e536 cpu: enable SpacemiT IME2 MXFP4 dense repack (llama/31776) 7f4b25f llamafile: impl bf16 and q8_0 vxe (llama/31866) f22fc6a ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (llama/31084) 47b8b4f ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (llama/31818) b524960 webgpu : add COUNT_EQUAL support (llama/31578) 75a4f5a ggml : switch to F32 src1 in the CPU backend (llama/31854) 6cddab2 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (llama/31851) 7353275 opencl: improve q8_0 and MoE small-n verification for spec (llama/31831) 8f1404e hexagon: support for automatic loading of libhtp without legacy env vars (llama/31781) 91c102a sycl: exact rounding in CPY to Q4_1, Q5_1 and Q8_0 (llama/31465) 8cf32bc sycl: refuse device buffers that would exceed VRAM (llama/31285) d969b07 hexagon: n_hvx and n_threads cleanup (llama/31857) 5746ebd llama_moe_cache: run missed experts on CPU, add tuning script (llama/30966) a49f174 sycl: add f16 support for SWIGLU_OAI (llama/30857) c6a0780 metal : fix GROUP_NORM for ne03 > 1 and non-contiguous inputs (llama/31852) 1e5c3e0 webgpu : Implement group norm kernel (llama/31490) 37ecdca vulkan: Add 'readonly' to decodeBuf declarations. (llama/31833) c08538b CUDA: fix shared memory bank conflicts in the lightning indexer (llama/31803) 07b8362 opencl: add arange (llama/31829) f55a504 meta : raise the compute view limit for buffers with few tensors (llama/31447) 2870133 hexagon: fuse post-norm residual add into RMS_NORM_MUL (llama/31822) a135ec6 hexagon: conv_2d optimizations and support for conv_3d (llama/31814) e3d3633 tests : qualify isnan and signbit with std:: (llama/31791) a7d3330 tests : override hardware_concurrency in CI (llama/31797) ab7ca9c opencl: support repeat for f16, i32, i16, bf16 (llama/31782) 9290392 webgpu: add batched tensor API (get_tensor_n and set_tensor_n) (llama/31473) f594fed vulkan: detect ReBAR support, avoid BAR allocations if unavailable (llama/31658) 38e4be2 CUDA: fix Maxwell build (llama/31787) 8bd6b61 opencl: add q6_k multi column tiled GEMV for speculative verification and refactoring (llama/31758) d3ce63d hexagon: Implement set_tensor_n and improve repacking/tiling robustness (llama/31744) 9b1880f ggml-cuda: sanitize post-bias NaNs in MoE selection (llama/31005) 7e607b8 vulkan: Implement test-fusion interface (llama/30769) 2188d13 CUDA: use highest compiled arch in get_mmvq_mmid_max_batch (llama/31679) 114ce5e vulkan: RDNA3/4 Large-Tile Split-K Optimization (llama/30280) e074347 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (llama/31615) f0f5b42 CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a) (llama/29050) fa8c617 CUDA: linearize MoE data layout for MMQ/MMF (llama/31047) 1690d2e vulkan: FWHT kernels for block widths above 512 (llama/29772) 40278c4 vulkan : check combined K+V f16 scratch size in FA dequant path (llama/30487) 2dccfdb sync : llama.cpp 25b845e ggml : support truncate=false for YaRN (llama/30206) accf2a1 sync : llama.cpp b2ce926 metal : use mul_mm for 32 <= K < 64 (llama/30982) 1a5f477 webgpu : add MEAN support (llama/31591) 2ab6d6c vulkan: Re-enable large coopmat matmul configuration on AMD proprietary driver 26.10.1+ (llama/31587) 19df6cf tests: fix CI flakes of TOPK_MOE caused by near ties (llama/31426) a0df99d metal: keep fusable node groups inside one command buffer (llama/31688) 3a996db opencl: enable f32 and q5_k multi column GEMV by default for Adreno X2 (llama/31694) d75184e opencl: support non-contig sigmoid (llama/31687) c634501 opencl: add DSV4_HC_PRE and DSV4_HC_POST (llama/31623) 0816137 Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (llama/31296) 4ac0dae opencl: improve decode perf with FA (llama/31633) 4f681f2 webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (llama/31348) cd6d2b7 docs : sync ggml_type enum in gguf.md (#1685)