WebAssembly Needs Schedulers, and Kubernetes Doesn’t Quite Fit the Bill

Cloud native computing no longer only means working with containers.

One good example is the architecture of Cosmonic WebAssembly Platform as a Service. It is as cloud native as you can be, but with only a minimal reliance on containers. Bring your WebAssembly to Cosmonic, and the platform offers an easy way to run the code and even to connect to other services.

When building out the infrastructure to run such a potentially highly-scalable platform, Cosmonic didn’t go with the seemingly standard choice these days, namely Kubernetes, but rather went with the HashiCorp Nomad open source orchestrator.

The advantage of Nomad over Kubernetes is that offers more flexibility when dealing with non-container runtimes, the platform service has found.

“Applications are not always containers,” said Taylor Thomas, engineering director at Cosmonic, speaking at the HashiCorp‘s annual user conference, HashiConf 2022, held last week in Los Angeles.

Reducing Boilerplate

The current cloud native approach for running even a minimal application can be a chore to developers. Even a few hundred lines of some simple business logic must be compiled, and then packaged into a container, then scheduled on Kubernetes.

The idea behind Cosmonic is the eliminate all this boilerplate code a developer has to write.

Cosmonic is basically the “enterprise goo” that sits around WASMCloud, an open source Cloud Native Computing Foundation platform, currently a sandbox project, for WebAssembly. Cosmonic is one of the principal maintainers for WASMCloud.

Offering a space for customer application runtimes, the Cosmonic cloud is built to be horizontally and vertically scalable.

So, each time a developer needs an http server, or a Redis database, for instance, the developer just writes a contract in the code, for say a server or a key-value store, and the platform provides the service automatically.

“These capabilities allow us to abstract away that code that they don’t need to manage anymore. It gets rid of that boilerplate. And it makes this choice a runtime decision which people running platforms, like us, then get to handle,” Thomas said.

The services, which can be considered composable actors, are all “stateless and reactive. ” No connection strings or clients are needed. The actors are written in WASM so they have a low footprint and are fast. There are other bits of backend needed to make this happen, such as a key-value management store and the containerized telemetry.

The Cosmonic service runs on Nomad, HashiCorp’s scheduler and orchestrator. Nomad runs as the job scheduler, not the runtime. For the control plane, Nomad works with the HashiCorp Consul identity networking software and the HashiCorp Vault secrets manager.

For multitenant security, the system uses the Firecracker microVMs. These very lightweight, secure and fast virtual machines “allow us to run customer hosts very tightly, and very securely all next to each other on the same actual servers,” Taylor said.

“Nomad is the perfect scheduler for this use case, because it allows for mixed environments. And it is, it’s able to scale to any kind of like structure or thing you want to set it up with,” Thomas said.

This setup would not be possible to run under Kubernetes, Thomas said.

With Nomad, “you only have to pick the specific things you need, instead of trying to swallow the world like you do with Kubernetes,” he said.

Better yet, it is not limited to just containers. It works with WASM for Cosmonic, but it can also easily support Java, QEMU, and even fork exec. It also works easily with adjacent standards such as the Container Storage Interface (CSI) and the Container Networking Interface,

Thomas worked on the Krustlet project, an effort to extend Kubernetes to support non-container workloads — including WASM. His takeaway?

“Let me tell you, Kubernetes does not like things that are not containers. It was a pain in the butt,” he said, explaining that it took “thousands and thousands of lines of code,” to bridge the two technologies.

You do not have this problem with Nomad, he went on to assert. It is very extensible, by way of HashiCorp’s own Task Driver. Task Driver is the runtime component that executes the workloads. It is also a framework that can be used to bring in new types of workloads. Cosmonic built its own Firecracker Task Driver.