Newshot

Lawyer’s $9,450 private AI server with 12 used Nvidia V100s now runs every DeepSeek V4.1 Flash expert

Summary

A lawyer has upped the ante by upgrading his private AI server from eight to 12 used Nvidia V100s, which can run every DeepSeek V4.1 Flash expert at 4-bit NVFP4. Performance can reach up to 184 tokens per second for code, and long documents start answering in about half the time as before. He says it can outrun published DGX Spark stacks at decoding a single request, for half the price of three units, based on the $9,450 he paid. Lawyer with too many GPUs: DeepSeek-V4.1-Flash on 12x V100, all 15,360 experts in NVFP4 on the cards. 157-184 tok/s on one code stream, faster than any published 4x DGX Spark stack and double 3 Sparks, for half the price of 3 Sparks. 128K file read in 28 s. from r/LocalLLM Some of the best movie franchises come in threes, and Nick Mermiges seems determined to match that, taking his original setup — which he previously upgraded with more memory and other fixes — and expanding it for fuller precision, after a previous upgrade got him most of the way there. The n…

Original text(via Tom's Hardware)
PCIe switch cards with red and blue status lights and braided cables in an open-frame server
PCIe switch cards with red and blue status lights and braided cables in an open-frame server

A lawyer has upped the ante by upgrading his private AI server from eight to 12 used Nvidia V100s, which can run every DeepSeek V4.1 Flash expert at 4-bit NVFP4. Performance can reach up to 184 tokens per second for code, and long documents start answering in about half the time as before. He says it can outrun published DGX Spark stacks at decoding a single request, for half the price of three units, based on the $9,450 he paid.

图表:from r/LocalLLM

Lawyer with too many GPUs: DeepSeek-V4.1-Flash on 12x V100, all 15,360 experts in NVFP4 on the cards. 157-184 tok/s on one code stream, faster than any published 4x DGX Spark stack and double 3 Sparks, for half the price of 3 Sparks. 128K file read in 28 s. from r/LocalLLM

Some of the best movie franchises come in threes, and Nick Mermiges seems determined to match that, taking his original setup — which he previously upgraded with more memory and other fixes — and expanding it for fuller precision, after a previous upgrade got him most of the way there. The newest upgrade, this time, involves a third board of V100s, for a total of 12 with four per board. There may be a method to his madness.

The original eight 32GB cards could manage 254 GiB, but the experts for the chosen model, with Nvidia’s NVFP4 release, need 278.9 GiB. His previous workaround was to have every expert on the cards at 2-bit precision, with the most-used, 5,949 or about 38.7%, swapped to NVFP4 until the cards were full. Now, all 15,360 experts can live on the cards in their full glory, as he intended, with “none compressed further,” according to his GitHub repo. In a longer write-up on his site, he says the expert values “equal the ones in DeepSeek’s own release,” although other parts of the model use compact forms.

A keen observer will notice a limitation of the V100s, which he spells out: the V100 has no FP4 or FP8 tensor cores. Every NVFP4 number has to be unpacked in software, first, and into half precision. The compact experts still pay off “because the cards read fewer bytes, not because they compute faster.” This helps explain why a card from 2018 can run a format from the Blackwell era.

The model’s layers are, again, divided among the boards, this time with 0–13, 14–27, and 28–39, and each board’s section is split across its own four cards. The cards on a board share Nvidia’s fast link, or NVLink, and the hand-offs between boards cross over the host’s PCIe through two separate PCIe switches. The first two boards share a PEX88096 switch, so their hand-off stays inside it, but the third board is behind a separate PLX 8749, so the second hand-off also has to cross the host’s PCIe bridges, which adds latency.

All this trouble is worth it, though, as with different settings — each card’s power cap raised from 150 W to 175 W and its clock cap lowered from 1,522 MHz to 1,452 MHz — there are noticeable improvements to performance as well as precision.

His measurements show the first code token, with 256K context, at 58 seconds, almost twice as fast as the 108.4 seconds after his previous upgrade. Prompt reading is at “about 4,500 tokens a second” past 32K, which is more than twice the roughly 2,200 tokens a second of his first build, though the higher power cap likely accounts for some of that. Code decode, meanwhile, is 23% faster at 1K, but shows no real gains at 32K and up, so the real gain is a reduced wait for the first token.

Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.

Bar chart comparing the wait for the first token on a cold read and on an already cached document, for code and legal prompts from 1K to 256K tokens, on a log scale
Bar chart comparing the wait for the first token on a cold read and on an already cached document, for code and legal prompts from 1K to 256K tokens, on a log scale

There was some eagerness to get more out of the system by pushing each card to 200 W for a day, but “the host tripped once under four long chats at once,” implying the power draw was perhaps too much. He also worked to optimize the setup as much as possible, but four optimizations — a conversation store that saves to disk, a leaner hand-off between sections, tighter expert packing, and exact four-card-sum arithmetic — had to be turned off due to missing their targets.

Another common technique is to run multiple requests at once, usually netting a speed improvement. In this case, three requests at 128K got about 360 tokens per second in total on code and 213 on legal. That said, “this is not a batched-decode engine; more requests than sections gain little.”

However, that might be next on the docket, as he plans to study batched decoding, “the one thing the Sparks have that I want,” he wrote on Reddit. Beyond that, there are plans to turn on the conversation store so a matter’s documents can survive a restart. While he acknowledges that the numbers do not establish “answer quality in use,” the setup has been his default DeepSeek machine since Monday, and he says the model is the first one “that works like Opus 4.6” for him.

Google Preferred Source
Google Preferred Source

Follow Tom’s Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.

Shane Downing
Shane Downing

Shane Downing is a Contributing Writer for Tom’s Hardware, covering consumer storage, PC hardware, and AI.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Tom's Hardware
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗