Apple Plans to Run Most of Its ‘Compute Management’ on Kubernetes

Consumer hardware giant Apple is ramping up full production use of Kubernetes to provide “compute management” for its developers and engineers, according to a talk given today by Apple Software Engineer Alena Prokharchyk, speaking at KubeCon + CloudNativeCon North America, being held virtually this year by the Cloud Native Computing Foundation.

Over time, Apple will run the majority of its workloads on Kubernetes clusters, she said.

For the CNCF, which is encouraging greater input from enterprise end-users on its technologies at this year’s conference, Apple’s use — and willingness of this typically secretive company to publicly disclose its K8s operations — is a strong validation point for wider enterprise usage of its open source orchestration platform.

In developing an internal app development platform on Kubernetes, the Apple platform team was challenged to offer “Cluster as a Service” and “Namespace as a Service” access to give teams the ability to choose tools to run inside their own namespaces @Lemonjet #kubeconNA pic.twitter.com/4gCxuDNGww

— Libby Clark (@LibbyMClark) November 18, 2020

Prior to using Kubernetes, Apple had been using the Apache Mesos resource management platform customized into a framework called Jarvis, which offered both stateless and stateful services to developers, though the container orchestrator had become a bottleneck. The company also wanted to find a platform that would offer greater utilization of hardware resources.

Of course, the natural choice was Kubernetes, said Prokharchyk, who is also a member of the CNCF Technical Oversite Committee. The plug-in nature of K8s meant that choosing storage, runtime networking operations were “no longer life-and-death decisions,” Prokharchyk said. “The choice could always be reevaluated without refactoring the entire system.” Another important feature for developers was the ability to extend the Kubernetes API through controllers and custom resource definitions, and Apple appreciated the openness and transparency of the community around Kubernetes.

One of the chief goals of the implementation is to meet the demands of the diverse user base within the organization (though some groups were already using K8s for individual projects at the start of the rollout). Apple was aware that there would be a K8s learning curve for the company’s platform developers, as well as a reevaluation of the company’s engineering practices and how they would fit into this new cloud native paradigm. “We realized the impact of the decision,” Prokharchyk said.

👁 Image

Part of the work the company is undertaking is configuring Kubernetes to make it as easy to use for developers as possible. This involves understanding the requirements of different groups of users. One group of Java/Python/Go developers, for instance, liked using containers to test their applications. Application Site Reliability Engineers (SREs) look to building advanced deployment workflows for their teams. Hardware engineers want to expedite their methods of validating devices. Machine learning and batch workload engineers want to run large jobs across thousands of pods.

“All of these users want to adopt cloud native tools for better debugging, logging, monitoring and tracing of their apps,” Prokharchyk said. “Our responsibility as platform developers is to provide a scalable orchestration layer with secure resource isolation and reliable scheduling.”

It is the work of the platform group to offer high-level interfaces for developers, Prokharchyk said. Now Apple provides these K8s resources as two kinds of services: namespace-as-a-service and cluster-as-a-service. With namespace-as-a-service — the primary way these services will probably be consumed at Apple — users don’t have to worry about infrastructure management. “They can focus on application deployment and development.”

For multitenant usage, resource management was separated out as its own abstraction, with an emphasis on balancing workloads across a hybrid environment. “The system would do dynamic balancing between clusters, priorities, usage and limits, to find the best placement for the user application,” she said. Within the namespace service, users are free to choose their preferred cloud native tools, offered supported by CRDs defined by the platform team.

Currently, Apple engineers are looking to contribute more to Kubernetes development, hoping to get needed features accepted upstream. For instance, they are looking for greater support of microVMs, which could offer multilayer isolation for multitenant workloads. They are also exploring the use of “virtual clusters,” of self-service clusters where the user gets to define the control and data plane for their own use. Multicluster workload management is also an area of investigation.

“We want to reduce the platform complexity for developers and operators. At the same time we want to maintain high security standards and resource utilization,” she said.