Introducing New Hyperscale NAS Class to Take away Storage Bottlenecks and Present the Excessive-Efficiency Knowledge Pipeline Wanted for GPU Computing

Hammerspace, the corporate orchestrating the Subsequent Knowledge Cycle, unveiled the high-performance NAS structure wanted to handle the necessities of broad-based enterprise AI, machine studying and deep studying (AI/ML/DL) initiatives and the widespread rise of GPU computing each on-premises and within the cloud. This new class of storage structure – Hyperscale NAS – is constructed on the tenants required for giant language mannequin (LLM) coaching and supplies the velocity to effectively energy GPU clusters of any measurement for GenAI, rendering and enterprise high-performance computing.

“Most computing websites are confronted with broad workload traits needing a storage answer with enterprise options, distance/edge, classical HPC, interactive, and AI/ML/knowledge analytics capabilities all at giant scale. There’s a quickly rising want for a distributed and parallel knowledge storage structure that covers this broad area in order that websites don’t face the inefficiencies of supporting many alternative options,” mentioned Gary Grider, Excessive Efficiency Computing Division Chief at Los Alamos Nationwide Laboratory. “With the latest and near-future developments to the NFS customary, the open supply implementation and acceptance into Linux, NFS has the options that allow a storage structure to service this rising selection and scale of workloads effectively. We at LANL are happy to see an business companion contribute to Linux/Web requirements that deal with broad wants and scales.”

Legacy NAS Architectures Will By no means Meet the Calls for of AI Coaching at Scale

Constructing and coaching efficient AI fashions require large efficiency to feed the GPU clusters that course of the info. The efficiency necessities are different, requiring a mixture of streaming giant information, read-intensive purposes and random read-write workloads for checkpointing and scratch area. Conventional scale-out NAS architectures – even all-flash programs – can’t meet these purposes’ efficiency or scale necessities. Delivering constant efficiency at this scale has beforehand solely been potential with HPC parallel file programs, that are complicated to deploy and handle and don’t meet enterprise necessities.

A Hyperscale NAS structure supplies the perfect structure for coaching efficient fashions, dashing time-to-market and time-to-insight, and finally deriving enterprise worth from knowledge.

“Enterprises pursuing AI initiatives will encounter challenges with their present IT infrastructure by way of the tradeoffs between velocity, scale, safety and ease,” mentioned David Flynn, Hammerspace Founder and CEO. “These organizations require the efficiency and cost-effective scale of HPC parallel file programs and should meet enterprise necessities for ease of use and knowledge safety. Hyperscale NAS is a essentially totally different NAS structure that permits organizations to make use of the perfect of HPC know-how with out compromising enterprise requirements.”  

Hyperscale NAS is Confirmed because the Quickest File System for AI Mannequin Coaching at Scale

The Hyperscale NAS structure has now been confirmed to be the quickest file system on the planet for enterprise and web-scale AI coaching. It’s in manufacturing with programs constructed on roughly 1,000 storage nodes, feeding as much as 30,000 GPUs at an combination efficiency of 80 Terabits/sec over customary ethernet and TCP/IP.

Hyperscale NAS is Wanted for Enterprise GPU Computing at Any Scale

Hyperscale NAS is adapting huge tech methods for enterprise use. Identical to Amazon Net Companies (AWS) developed S3 for large-scale, environment friendly storage, changing into a mannequin for object storage in firms, Hyperscale NAS is doing the identical. It’s the system used for coaching giant language fashions (LLMs) and is now being utilized in companies for computing with GPUs and coaching generative AI fashions. This strategy is spreading, bringing superior tech firm strategies to firms of all sizes.

The Hammerspace Hyperscale NAS structure is right for each hyperscalers and enterprises because it doesn’t require proprietary shopper software program, effectively scales to satisfy the calls for of any variety of GPUs throughout coaching and inference, makes use of present Ethernet or InfiniBand networks, present commodity or third-party storage infrastructure, and has a whole set of information companies to satisfy compliance, safety and knowledge governance necessities.

Hammerspace Hyperscale NAS is Licensed as NVIDIA GPUDirect Storage

The Hammerspace Hyperscale NAS structure has accomplished the GPUDirect Storage Assist validation course of from NVIDIA. This certification permits organizations to leverage Hammerspace software program to unify unstructured knowledge and speed up knowledge pipelines with NVIDIA’s GPUDirect^® household of applied sciences. By deploying Hammerspace in entrance of present storage programs, any storage system can now be offered as GPUDirect Storage through Hammerspace to supply excessive throughput and low latency efficiency to maintain NVIDIA GPUs absolutely utilized.

Trade Commentaries

“As enterprises and authorities organizations more and more harness the ability of AI, the significance of effectively managing file knowledge throughout heterogeneous infrastructure has by no means been extra crucial. Our partnership with Hammerspace represents a pivotal step ahead, enabling Infinidat’s prospects to seamlessly incorporate file-based workloads into their trusted InfiniBox enterprise storage environments,” acknowledged Eric Herzog, Chief Advertising Officer at Infinidat. “The introduction of the Hyperscale NAS answer into the Hammerspace World Knowledge Atmosphere is a testomony to their dedication to assembly our prospects’ evolving wants. This answer not solely enhances Infinidat’s unmatched efficiency, superior knowledge safety, and cyber storage resilience but additionally integrates a world namespace, making certain our prospects have entry to a complete, high-performance knowledge administration platform. Collectively, Infinidat and Hammerspace are setting a brand new customary for enterprise storage options tailor-made to the calls for of the trendy knowledge panorama.”

“We now have historically separated Scale-out File Programs, generally often called parallel file programs, from NAS exactly because of the nature of their efficiency for very giant HPC/ AI environments. As we enter into this subsequent era of AI, new applied sciences, significantly in knowledge infrastructure, are wanted,” mentioned Camberley Bates, VP and Follow Lead at The Futurum Group. “Hammerspace isn’t solely bringing distinct knowledge administration however is now enhancing in place NAS programs to handle this very large-scale atmosphere that will probably be commonplace for all organizations.”

“Many storage programs contain a number of layers of communication and knowledge switch. By embedding NFS straight into an Ethernet-attached SSD array, many of those layers are bypassed, leading to considerably decrease latency,” mentioned Thomas Isakovich, CEO at Nimbus Knowledge. “We’re excited to work with Hammerspace as we companion to proceed to ship beforehand unmatched low latency and knowledge path velocity to high-performance purposes.”