Enterprises racing to build AI factories and push edge inference into the field are learning that buying graphics processing units is the easy part. Power, cooling and network throughput now decide whether those costly systems produce intelligence or sit idle.
Those physical limits shaped much of the high-performance computing and AI conversation at SC26, where commercial AI infrastructure increasingly shares the floor with scientific computing. Rack densities that once topped out at about 10 kilowatts are now on a path toward a megawatt, a step change that forces enterprises to look both above and below the traditional network, storage and compute layers, according to Jack Hogan (pictured), vice president of advanced growth technologies at SHI International Corp.
“When we look at the full stack, we recognize that customers are getting tripped up. They can go get the GPUs, but they can’t power them. They can’t cool them,” Hogan said. “These are not cheap systems either. Procuring the infrastructure is just the start.”
Hogan spoke with theCUBE Research’s John Furrier at SC26, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed edge inference and how power, networking, model choice and business outcomes are reshaping the way enterprises build and run AI infrastructure. (* Disclosure below.)
Edge inference and the megawatt as AI’s new yardstick
The economics of AI infrastructure are shifting along with the hardware. Enterprises once sized deployments by the rack unit or the rack, but entire data centers now operate as a single system with shared memory, and every GPU must be kept fed to justify its cost. That change has altered how SHI measures success, Hogan explained.
“Now we’re talking about row and now data center scale. … We’re measuring that now in megawatts,” Hogan said. “So the megawatt now starts to become a measurement that people start to grasp and understand. How many flops can you get through that megawatt? How fast can you process your tokens?”
SHI’s role spans the full life cycle of these systems. Drawing on 37 years in the industry, the company builds customized racks in its integration facilities around the globe and runs pre-site assessments so systems can be powered on as soon as they arrive. In a recent New Jersey project, SHI is hosting and operating the customer’s system in its own data center. Location matters at the edge, too, where workloads such as video input generate data that must be processed in place, Hogan noted.
“You can’t backhaul that. The networks are not necessarily the right way to move that,” Hogan said. “We’re creating the scenario of both the mega hyperscaler as well as the edge inference and be able to tie those two things together.”
Busy GPUs are not necessarily productive ones. SHI starts every engagement by asking customers what they are trying to achieve, then works backward to the right model, whether that means frontier intelligence or a small language model running on an edge device. Silicon follows the same logic, with GPUs, newer CPUs, inference-specific chips and neural processing units validated in the company’s AI and Cyber Lab.
“It’s not about the cost of the token or reducing the number of tokens or even finding cheaper tokens. It’s about optimizing and ensuring that those tokens are turning into business value,” Hogan said.
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of SC26:
(* Disclosure: SHI sponsored this segment of theCUBE. Neither SHI nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.