Newshot

Cerebras Disaggregated Inference

Original text(via Hacker News)

We plan to deploy disaggregated inference broadly across hundreds of megawatts in the Cerebras Inference Cloud and in partner deployments by the end of 2027. Our aim is to increase the ultrafast inference capacity available for our customers.

Cerebras disaggregated inference for infrastructure partners

The Cerebras disaggregated inference solution was also built for our partners to adopt. Infrastructure partners can deploy disaggregated inference clusters based on Cerebras-provided reference architecture, co-locating their chosen prefill accelerators and Cerebras systems within a unified network. For software integration, partners can either integrate with the Cerebras end-to-end disaggregated software stack or a decode-only SDK.

The end-to-end disaggregated software stack is best suited for partners optimizing time-to-market. It consists of:

  • A disaggregated inference framework to route requests between prefill and decode pools and manage replica capacity and health.
  • A prefill engine to process prompts using open-source inference engines like vLLM or SGLang.
  • A Cerebras decode engine for response generation on Cerebras systems.
  • KV Portal software, an interface to transfer intermediate model state, including key-value caches, between prefill and decode engines over RDMA.

The decode-only SDK is best suited for partners optimizing for control. With the decode-only SDK, partners can bring their own inference framework and prefill engine to the Cerebras decode engine and KV Portal software. In addition, the decode-only SDK provides development aids such as a mock Cerebras decode replica and a reference integration to an open-source disaggregated inference framework and prefill engine.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Hacker News
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗