Cloud Control Planes for All: Implement Internal Platforms with Crossplane

Over the last two decades, the rise of cloud computing has enabled the creation of countless new businesses and revolutionized how existing organizations operate and scale. The value offered by cloud-service providers during this transition cannot be overstated, but there have also been tremendous advantages for the cloud-service providers themselves.

The efficiencies of scale that are distributed to customers are experienced to an even larger degree internally, allowing for these platforms to grow and expand at a rapid rate. With Crossplane, an open source framework for building cloud native control planes, customers can start to realize those efficiencies in their own internal platforms.

The Value of Foundation Services

Most cloud-service providers started out with a few core offerings in storage and compute. Amazon Web Services (AWS) famously began with just S3 before expanding into compute with EC2. These services have and continue to provide immense value to customers who previously would have had to invest a lot of time and money in both managing physical infrastructure and writing software. However, they also serve as a foundation for AWS to rapidly expand the catalog of services they offer.

The reason cloud-service providers are able to launch new services at an increasingly rapid rate is that the new services can be built on the capabilities of the existing infrastructure.

For example, AWS Lambda, a popular serverless platform, was launched at scale because it is built on top of the battle-tested compute offered by EC2. Furthermore, these new services are able to integrate with others offered by the cloud provider, such as databases and storage, and a common identity and access management (IAM) framework could be extended to support the new functionality. The result is a flywheel effect where, in aggregate, the cost of launching the next service diminishes over time.

While customers have benefited from this capability in the form of having new services made available to them, they have typically been unable to develop the same competency when building their own internal platforms. The impact is twofold:

• Customers are subject to the decisions of cloud providers when it comes to what new services are launched. Often these decisions are heavily influenced by the provider’s largest customers.
• Customers must take on the burden of onboarding and “productionizing” new services, which typically includes extensive testing, documentation and compliance review.

It is rare that an organization of reasonable size exposes raw cloud infrastructure to developers. Instead, they implement some layer of abstraction over the API in order to ensure that only infrastructure that meets internal policies can be provisioned. These policies may enforce certain security, reliability or cost requirements. For example, developers may only have the ability to provision some subset of EC2-instance types in order to avoid incurring high costs.

Unfortunately, every time a new cloud service is introduced into an organization, these policies have to be mapped to that service’s API. While the cloud provider may be offering new products by building on top of its own foundational services, every new service they expose to customers is a net new API to be onboarded in an organization.

This linear trend in overhead of onboarding the next service stands in stark contrast to the logarithmic curve that the cloud providers experience when launching the service.

Ideally, doing the hard work of onboarding a service in an organization would be amortized over the subsequent services it enables. More and more organizations are moving to building internal platforms, rather than just consuming cloud-service primitives. This pattern creates a unique opportunity for these organizations to build their own foundational services, which can subsequently be used to build higher-level services in the future.

However, operating a platform is complex, and cloud providers have done a lot of work to build control planes that power theirs, presenting a single API surface and ensuring that services are resilient to failure. Replicating this amount of engineering effort in most organizations is not feasible. Fortunately, the Crossplane project has taken the capabilities of Kubernetes and constructed a framework that makes building control-plane-based platforms possible for anyone.

Defining Your Platform Foundation

When starting out on their internal platform journey, every organization must determine what APIs will serve as their foundation services. Countless factors could determine where to begin, including size of the company, complexity of the product, and culture of the engineering organization. Starting with a commonly used primitive, such as a database or containerized workload, is usually a safe bet.

In Crossplane, platform APIs are defined using Composition, which allows a platform team to define an API for developers to interact with, and one or more implementations of that API. For example, if developers frequently deploy applications packaged as OCI images, a platform team may define a ContainerizedWorkload API with parameters for an image reference and a set of environment variables.

An implementation of that API could be an AWS Lambda function, an ECS Task Definition or any other set of compute primitives. While these AWS services offer a wide array of customization options, only the parameters that a developer cares about are exposed in the internal platform, allowing the platform team to incorporate any organizational policies, such as tags or memory limits, behind the scenes.

Providing the initial implementations of the ContainerizedWorkload are high-touch activities that require understanding of the cloud-service APIs they rely on. In other words, this is the early part of the logarithmic curve, where there is still significant toil required to add a new API. However, your platform now has its first service that is fully compliant with organizational policy. You’ve defined a foundation service.

Growing with Confidence

In the early stages of building a platform, much of the work will involve building out additional foundation services. Perhaps developers also frequently need to provision databases, so a MySQLInstance API is defined and implemented. However, as the core set of foundation services are rounded out, the cost of adding the next service starts to decrease.

For example, with a workload and database API in place, a StatefulWorkload API can be added to the platform with minimal effort because it relies on foundation services that have organizational policy baked in. In other words: compliance composes.

Furthermore, implicit in this architecture is the propagation of platform changes and improvements from lower-level services to higher-level ones. New organizational mandates, for anything from SOC2 compliance to switching cloud providers, can be carried out at the foundation level but applied to all levels above.

Additionally, transitioning a higher-level API, such as StatefulWorkload, to rely on cloud services directly instead of internal platform APIs can be carried out at any time because the platform is built by stacking APIs, not by stacking implementations.

Your First Platform in 5 Minutes or Less

Defining a platform is inherently iterative, and Crossplane makes getting started on the journey easy. Grab a Kubernetes cluster from your favorite provider, install Crossplane and pick a QuickStart package to implement your first APIs!