Codefresh Runner: Run CI/CD Pipelines Inside a Kubernetes Cluster

Codefresh has recently open-sourced the Codefresh Runner, a native Kubernetes application for running CI/CD pipelines inside any Kubernetes cluster, either on a public cloud or behind the firewall in a private cluster.

Continuous Integration/Continuous Delivery (CI/CD) pipelines are by their own nature a perfect workload for moving to a Kubernetes cluster. In most software organizations the number of active pipelines is always changing across time. For example, just before a software release happens, the number of active pipelines will skyrocket as engineers are rushing to implement last-minute features and fixes.

Auto-scaling workloads is one of the built-in characteristics of a Kubernetes cluster. Placing CI/CD pipelines in a Kubernetes cluster is an obvious choice as this will make them take advantage of the auto-scaling feature without any other configuration.

The Codefresh Runner not only allows you to take advantage of Kubernetes scalability for CI/CD pipelines but also comes with several advantages for dealing with security and compliance of the data and services used in CI/CD pipelines.

Secure Services in a CI/CD Pipeline

Companies that have strict security limitations are always facing a tough choice when it comes to using a new software tool. The usual option is to install the tool on-premise and spend considerable effort on maintenance and monitoring it. The other choice is to use the public cloud and accept all the risks associated with it.

The Codefresh Runner is taking the best of both worlds by implementing a Hybrid solution. The Runner itself is installed inside a private Kubernetes cluster. This way it gets access to all secure services such as Git repositories, Docker registries, and other databases that the organization already maintains.

The Web UI for managing the Runner is running on the public cloud, enjoying all the benefits of a SaaS platform. The important thing to notice is that the network communication is only outgoing. The Codefresh Runner is polling the cloud UI for pipelines to execute. No firewall ports need to open making installation painless.

At the same time, the source code used inside a CI/CD pipeline always stays behind the firewall. The same method is followed for databases, Docker registries, and other secure services.

The end result is that a company using the Codefresh Runner can enjoy the reduced maintenance of a SaaS solution (the Web UI is monitored and managed by the Codefresh team) while still making sure that the intellectual property of the company (i.e. source code and secure data) is still safe behind the company firewall.

Kubernetes Scalability in CI/CD Pipelines

A Kubernetes cluster can be set up to scale according to workload metrics in different ways (i.e. increasing the number of pods or nodes). This is one of the biggest selling points of Kubernetes as it allows administrators to utilize the underlying resources in the best way possible.

The Codefresh Runner will automatically take advantage of any auto-scaling facilities present in the cluster it runs on. This means that as more pipelines are launched, the Runner will adapt to the increased load without any extra configuration.

A classic scenario for an increasing number of CI/CD pipelines is at the end of the sprint. Developers might open a large number of pull requests, or conversely, they will merge a large number of pull requests back to the mainline branch.

With the Codefresh Runner, all pipelines will run as soon as they are queued instead of creating a bottleneck due to a lack of resources. All Codefresh pipelines also have automatic workspace and Docker layer caching enabled without any extra configuration.

Multicloud CI/CD Pipelines

The Codefresh Runner is using the standard Kubernetes API without making any assumption about the underlying cloud provider. This means that the Runner can be installed in any compliant Kubernetes distribution in the same way (even on a local Kubernetes cluster running on your laptop).

Codefresh Pipelines are composable (i.e. you can call one pipeline from another pipeline). There is no restriction on where a child pipeline runs. This key characteristic allows you to chain pipelines that run on completely different clusters or even cloud providers.

As an example, you could run a parent pipeline that starts on the public Google Cloud, and it launches a second pipeline that runs in AWS. Or one pipeline can run inside your on-premise datacenter, but the second pipeline is running on the Codefresh SaaS resources. The Codefresh Runner is agnostic and behaves in the same manner regardless of what cluster it is running on.

Pipelines can be specifically targeted at different clusters or environments, giving maximum flexibility on the resource constraints and security measures present in the runtime environment.

Reusing Kubernetes Knowledge for CI/CD Pipelines

Many CI/CD solutions have been traditionally very difficult to set up and maintain. The Codefresh Runner takes a new approach when it comes to installation and maintenance.

First of all, while you can install the Codefresh Runner in many different ways (e.g. using Kubernetes manifests or a Helm chart), the quickest method of installation is by using a friendly command-line wizard.

The wizard asks some basic questions about your cluster and then proceeds to download, install, and set up all Runner resources inside the Kubernetes cluster in a single step.

After installation is complete, the Runner is self-sufficient. It will automatically look at your pipeline queue and run the pipelines that are assigned to the respective cluster along with all existing integrations of Docker registries and Git providers already set up in the Codefresh account. The Runner is using Docker images as build tooling, which means that new tools don’t have to be installed on the Runner itself.

The Runner components are all native Kubernetes resources. This means that you can monitor the Runner with your existing Kubernetes tools that are already in place inside the cluster. Especially in large organizations, this is a significant advantage as it allows operators to reuse their existing Kubernetes knowledge and investment on tools for cluster monitoring.

Conclusion

Running CI/CD pipelines on a Kubernetes cluster does not have to be a complicated process.

With the Codefresh Runner you can:

• Execute pipelines  on any compliant Kubernetes cluster regardless of cloud provider
• Access secure services behind the firewall in any CI/CD pipeline while keeping the source code inside the data center/private cloud
• Automatically gain the scalability benefits of Kubernetes for handling large numbers of pipelines
• Run pipelines in multicloud configurations and mix parent/child pipelines on completely clusters and cloud installations
• Maintain and monitor your CI/CD runtime like any other Kubernetes application

The Codefresh Runner is hosted in GitHub (part of the Codefresh CLI). See the documentation page for more details on how to install it on a Kubernetes cluster and start running CI/CD pipelines.