Newshot
Account

No GPU required: enthusiast runs 2.8 trillion-parameter Kimi K3 AI model on discontinued Optane memory at 1.5 tokens per second

Source: IT之家 Published: AI 中文 AIKimi
AI summary

A Reddit user (u/IndividualSurvival) ran Moonshot's 2.8 trillion-parameter Kimi K3 model without any graphics card, using discontinued Intel Optane Persistent Memory 200 series with a 40-core 3rd Gen Intel Xeon Scalable processor in App Direct mode and llama.cpp for inference. With 1-bit quantization, initial generation was about 0.85 tokens per second, rising to 1.50 tokens per second after disabling DAX so frequently used data could benefit from DRAM caching; a separate 3-bit quantization test reached only 0.75 tokens per second, taking roughly 18.5 hours to generate 50,000 tokens.

Translation(model translation, not the source's own text)

No GPU required: Enthusiast runs 2.8-trillion-parameter Kimi K3 AI model on discontinued Optane memory, outputs 1.5 tokens per second

IT Home, November 13: Reddit user u/IndividualSurvival posted on November 2 in the Reddit community, sharing a home-lab attempt that, without a graphics card, successfully ran the 2.8-trillion-parameter Moonshot AI Kimi K3 model using discontinued Intel Optane persistent memory.

IT Home note: “Optane” (Intel Optane) is a memory and storage product technology based on 3D XPoint, a non-volatile storage technology jointly developed by Intel and Micron. The user used the Optane Persistent Memory 200 series, whose form factor and installation method are similar to server memory modules; each module is available in 128GB, 256GB, and 512GB capacities and is designed for the third-generation Xeon Scalable processor platform.

In terms of configuration, the user used a 40-core third-generation Intel Xeon Scalable processor (the original article did not specify exact specifications), as well as Optane Persistent Memory 200 series (the original article did not specify exact capacity), and tested in App Direct mode (DRAM and Optane are presented to software as two separate data spaces), and used llama.cpp software to perform inference.

However, the project is still only a test project, and token output speed is not smooth. According to the report, after using 1-bit quantization and compressing model data, the initial generation speed was about 0.85 tokens per second; after disabling Direct Access (DAX) and letting frequently used data benefit from DRAM caching, the speed increased to 1.50 tokens per second. Another 3-bit quantization test only reached 0.75 tokens per second, requiring about 18.5 hours to generate 50,000 tokens.

Related articles

  • A key step toward AI taking over phones: Google Gemini automation app list revealed, currently 134 apps in total
  • Legendary architect behind Apple's M1 chip discusses AI power crisis: building supercomputers with the “frugal” philosophy of making iPhone chips
  • Axelera AI unveils data-center-class AIPU chip Titania architecture, production to start in 2027Q3
  • No subsidies, no added costs for others: Microsoft promises to pay for its own AI expansion electricity bills
  • After 黄仁勋 resigned, flowers and candles appeared outside 英伟达 headquarters
  • The “brakeman” of 黄仁勋's most aggressive AI bet: 英伟达 interim CEO 克雷斯 once pushed to cut the $250 billion OpenAI gamble

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by IT之家
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗