Newshot

Google发布TPU Sync:面向TPU的开源KV缓存传输方案

AI summary

Google Cloud发布了TPU Sync,这是一个用于在Cloud TPU上迁移KV缓存和模型权重的开源C++库。该库将TPU HBM、主机DRAM、共享存储和数据中心网络视为统一的内存层级,旨在解决模型状态迁移这一日益突出的瓶颈。Google称,在235B至480B参数的MoE模型上,KV传输延迟降低超过90%。

This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.

Published by MarkTechPost
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗