Apptainer, a Container System for High-Performance Computing

Apptainer, the container system for high-performance computing (HPC) formerly known as Singularity, has joined the Linux Foundation with at least one stated goal, according to Apptainer founder and project lead Gregory Kurtzer, summed up like this: “cross-pollination.”

“In HPC, we’ve built upon decades of research in scalable computing architectures and models. I don’t think this translates well to microservice orchestration well yet, but we are seeing more companies who are not traditionally HPC consumers needing HPC and HPC-like capabilities,” Kurtzer wrote in an email interview. “It is the goal of the project that this move into the Linux Foundation will foster cross-pollination between these two sectors of the ecosystem in such a way to yield better and more capable systems.”

Many of the traditional container runtimes and orchestration tools prioritize providing scalability, while Apptainer focuses instead on performance. Comparing Kubernetes with Apptainer, Kurtzer noted that the scaling factors for Kubernetes differed greatly from HPC and performance-intensive computing.

“While cloud can scale up to many thousands of bare metal hosts, HPC must be able to scale up to many tens of thousands of cores and processes all of which may run in lock-step coordination with each other in parallel. Think of interprocess communication occurring many times a second across tens of thousands of processes running on thousands of computers,” he wrote. “Kubernetes can do loosely coupled (or rather non-coupled) instance parallelization, but it is not good at any type of tightly coupled parallelization or even the rich policy requirements of general performance-intensive computing against consumable resources (e.g. memory, cores, GPUs, etc..).”

At the same time, each of these systems finds at its core a common component – a containerized workload that gives developers the same benefits of universal operation and portability that containers were originally adopted for. Those benefits are enjoyed by developers creating applications needing end-user scalability as well as more performance-intensive use cases like AI and machine learning. As time has gone on, these developers have increasingly become one and the same and one of the reasons given for the project’s move to the Linux Foundation is its recent growth, which Kurtzer attributes to a shared interest.

The Singularity

“Over the years, Singularity has become the massively dominant container system in HPC and recently we’ve seen a lot more interest from non-traditional HPC environments looking to do HPC-like workloads also being interested in Singularity,” Kurtzer explained.

While there have been efforts to bring more performance-intensive functionality into systems like Kubernetes, there remain some core challenges. Kurtzer explains that more standard enterprise applications running on Kubernetes allow for an oversubscription of resources because those services and microservices “are almost always sleeping in a select loop (or equiv) and run forever (from the perspective of the service itself, it never ends).” This makes it easy to couple hundreds of microservices in a single instance, whereas in HPC, a performance intensive application doesn’t sleep and “it is designed to efficiently run the hardware as fast as it can possibly go until it bottlenecks on a particular component (network, storage, PCI, memory, CPU, GPU, etc..).”

“So enterprise and cloud-focused capabilities have been distinct from HPC, and vise versa. We are seeing now that the uptake and interest in cross-pollination is increasing considerably. For example, I’ve now spoken to several very large hyperscale companies about building giant HPC systems for their data-analytics and AI models,” wrote Kurtzer. “Moving Singularity into the Linux Foundation and renaming it Apptainer provides a fantastic substrate for cross-pollination between various LF projects like OpenHPC and CNCF/OCI. We’ve already started considering things like how to take our security and trust models and couple it with projects like SigStore and/or Notary2.”

When we first looked into the topic of containers for HPC, one of the issues noted was that systems like Singularity (now Apptainer) were not OCI-compliant, but this has since been remedied and means that organizations can easily bring an OCI-compliant container from one system to another. By joining the Linux Foundation, Apptainer hopes to further co-mingle the capabilities of enterprises using containers for microservice application architectures as well as for performance-intensive applications.

“At this stage of the project’s maturity, coupled with the increasing desire for HPC and enterprise/cloud crossover, the Singularity project’s stakeholders all felt that the move to the Linux Foundation would help the project mature and bring these capabilities into other areas,” wrote Kurtzer. “There hasn’t been much sharing of capabilities between enterprise, cloud, hyperscale with HPC. HPC has been an isolated island and it’s time for that to change. Every sector of the ecosystem has value to bring to the table and Singularity, being the dominant container system in HPC, is ready for the next stage in its development and maturity. Joining the Linux Foundation will help everyone!”