Azure NetApp Files: Revolutionizing silicon design for high-performance computing

High-performance computing (HPC) workloads place significant demands on cloud infrastructure, requiring robust and scalable resources to handle complex and intensive computational tasks. These workloads often necessitate high levels of parallel processing power, typically provided by clusters of central processing unit (CPU) or graphics processing unit (GPU)-based virtual machines. Additionally, HPC applications demand substantial data storage and fast access speeds, which exceed the capabilities of traditional cloud file systems. Specialized storage solutions are required to meet the low latency and high throughput input/output (I/O) needs.

Microsoft Azure NetApp Files is designed to deliver low latency, high performance, and enterprise-grade data management at scale. Unique capabilities of Azure NetApp Files make it suitable for several high-performance computing workloads such as Electronic Design Automation (EDA), Seismic Processing, Reservoir Simulations, and Risk Modeling. This blog highlights Azure NetApp Files’ differentiated capabilities for EDA workloads and Microsoft’s silicon design journey. 

Infrastructure requirements of EDA workloads

EDA workloads have intensive computational and data processing requirements to address complex tasks in simulation, physical design, and verification. Each design stage involves multiple simulations to enhance accuracy, improve reliability, and detect design defects early, reducing debugging and redesigning costs. Silicon development engineers can use additional simulations to test different design scenarios and optimize the chip’s Power, Performance, and Area (PPA).

EDA workloads are classified into two primary types—Frontend and Backend, each with distinct requirements for the underlying storage and compute infrastructure. Frontend workloads focus on logic design and functional aspects of chip design and consist of thousands of short-duration parallel jobs with an I/O pattern characterized by frequent random reads and writes across millions of small files. Backend workloads focus on translating logic design to physical design for manufacturing and consists of hundreds of jobs involving sequential read/write of fewer larger files.

The choice of a storage solution to meet this unique mix of frontend and backend workload patterns is non-trivial. The SPEC consortium has established the SPEC SFS benchmark to help with benchmarking the various storage solutions in the industry. For EDA workloads, the EDA_BLENDED benchmark provides the characteristic patterns of the frontend and backend workloads. The I/O operations composition is described in the following table.

Azure NetApp Files supports regular volumes which are ideal for workloads like databases and general-purpose file systems. EDA workloads work on large volumes of data and require very high throughput; this requires multiple regular volumes. The introduction of large volumes to support higher quantities of data is advantageous for EDA workloads, as it simplifies data management and delivers superior performance compared to multiple regular volumes. 

Below is the output from the performance testing of the SPEC SFS EDA_BLENDED benchmark which demonstrates that Azure NetApp Files can deliver ~10 GiB/s throughput with less than 2 ms latency using large volumes.

Electronic Design Automation at Microsoft

Microsoft is committed to enabling AI on every workload and experience for devices of today and tomorrow. It begins with the design and manufacturing of silicon. Microsoft is surpassing scientific boundaries at an unprecedented pace for running EDA workflows, pushing the limits of Moore’s Law by adopting Azure for our own chip design needs.

Using the best practices model to optimize Azure for chip design between customers, partners, and suppliers has been crucial to the development of some of Microsoft’s first fully custom cloud silicon chips: 

• The Azure Maia 100 AI Accelerator, optimized for AI tasks and generative AI. 
• The Azure Cobalt 100 CPU, an Arm-based processor tailored to run general purpose compute workloads on Microsoft Azure.
• The Azure Integrated Hardware Security Module; Microsoft’s newest in-house security chip designed to harden key management.
• The Azure Boost DPU, the company’s first in-house data processing unit designed for data-centric workloads with high efficiency and low power. 

The chips developed by the Azure cloud hardware team are deployed in Azure servers delivering best-in-class compute capabilities for HPC workloads and further accelerate the pace of innovation, reliability, and operational efficiency used to develop Azure’s production systems. By adopting Azure for EDA, the Azure cloud hardware team enjoys these benefits: 

• Rapid access to scalable on-demand cutting edge processors.
• Dynamic pairing of each EDA tool to a specific CPU architecture. 
• Leveraging Microsoft’s innovations in AI-driven technologies for semiconductor workflows. 

How Azure NetApp Files accelerates semiconductor development innovation 

• Superior performance: Azure NetApp Files can deliver up to 652,260 IOPS with less than 2 milliseconds of latency, while reaching 826,000 IOPS at the performance edge (~7 milliseconds of latency).
• High scalability: As EDA projects advance, data generated can grow exponentially. Azure NetApp Files provides large-capacity, high performance single namespaces with large volumes up to 2PiB, seamlessly scaling to support compute clusters even up to 50,000 cores. 
• Operational simplicity: Azure NetApp Files is designed for simplicity, with convenient user experience via the Azure Portal or via automation API. 
• Cost efficiency: Azure NetApp Files offers cool access to transparently move cool data blocks to managed Azure storage tier for reduced cost, and then automatically back to the hot tier on access. Additionally, Azure NetApp Files reserved capacity provides significant cost savings compared to pay-as-you-go pricing, further reducing the high costs associated with enterprise-grade storage solutions. 
• Security and reliability: Azure NetApp Files provides enterprise-grade data management, control-plane, and data-plane security features, ensuring that critical EDA data is protected and available with key management and encryption for data at rest and for data in transit. 

The graphic below shows a production EDA cluster deployed in Azure by the Azure cloud hardware team where Azure NetApp Files serves clients with over 50,000 cores per cluster.

Azure NetApp Files provides the scalable performance and reliability that we need to facilitate seamless integration with Azure for a diverse set of Electronic Design Automation tools used in silicon development.

—Mike Lemus, Director, Silicon Development Compute Solutions at Microsoft.

In today’s fast-paced world of semiconductor design, Azure NetApp Files offers agility, performance, security, and stability—the keys to delivering silicon innovation for our Azure cloud.

—Silvian Goldenberg, Partner and General Manager for Design Methodology and Silicon Infrastructure at Microsoft.

Learn more about Azure NetApp Files

Azure NetApp Files has proven to be the storage solution of choice for the most demanding EDA workloads. By providing low latency, high throughput, and scalable performance, Azure NetApp Files supports the dynamic and complex nature of EDA tasks, ensuring rapid access to cutting-edge processors and seamless integration with Azure’s HPC solution stack.

Check out Azure Well-Architected Framework perspective on Azure NetApp Files for detailed information and guidance.

For further information related to Azure NetApp Files, check out the Azure NetApp Files documentation here.

The post Azure NetApp Files: Revolutionizing silicon design for high-performance computing appeared first on Microsoft Azure Blog.