Getting Started with GitOps 

Software development. Software engineering. Whatever you want to call it, the process is becoming more and more complex. It’s getting better at solving problems computationally too, but at the same time, the learning curve involved in building resilient, distributed systems is getting flatter and flatter. (Yes, flatter learning curves are the tough ones, y’all).

While I would argue it’s not killing the industry, it’s certainly making it harder for people to become software engineers. When I interviewed Tyler Jewell for the Art of Modern Ops podcast, he explained that while the population of developers is still growing, the rate of growth is so slow that the population is starting to plateau.

Growth in demand for software, meanwhile, is accelerating. In other words, demand for solid software is beginning to outstrip supply of the skills needed to craft it. The key to success is therefore to make today’s developers as empowered and productive as possible. Hence the success of GitOps for cloud native development.

Software Is Hard, and GitOps and Kubernetes Came to the Rescue

Let’s step back for a second. Yes, software development has always been tough. Computers are not easy to understand and technology doesn’t always lend itself to simple operation, although, arguably, machines only resolve primitives.

This isn’t exclusive to computational technology. My mother, a geologist, used to say that planning any intervention – whether a tunnel, an underground parking lot or just the foundations of a building – meant managing a set of variables that were almost beyond any certainty.

That’s why complex systems, whether the Earth’s crust or computers, are incredibly hard to master. It is, as Emily Freeman brightly puts it, a game of managing ever-increasing entropy.

So how does the cloud native community approach this growing entropy and complexity? As a culture that embraces collaboration, blameless improvement and a holistic, incremental approach to software development, DevOps would be the likeliest candidate to figure this out successfully.

However, a subset of this community — mostly those involved in CD at scale, and with Kubernetes in particular — gradually came up with an answer we now call GitOps.

Before Kubernetes, there was the rise of continuous integration (CI), config management/automation and continuous delivery (CD). Yet the orchestration of a containerized application’s deployment remained fairly complex. It still involved a combination of CI engines, configuration management database (CMDB) servers, package repositories and, of course, a ton of human coordination.

Regardless of how cumbersome these processes were, the results were fairly evident: fewer errors occurred when rolling out applications. In other words, value was seamlessly delivered to the end-user much more frequently. This was thanks in part to Kubernetes’ declarative nature.

In Comes Flux: Operationalize Complexity

A similar process happened at Weaveworks. When building products cloud natively back in 2017, we wanted, like everyone else, to release with as few problems as possible. So we automated our build, testing and release processes.

Although not evident at the outset, several requirements would soon reveal themselves to be critical:

• Store the system in git in declarative terms
• Store configuration manifests in git
• Have these observe the build pipeline’s artifact update
• Deploy any changes to the declared system automatically

With that in mind — what, at the time, we called Operations by Pull Request — Flux was born. It was actually born as a daemon that monitored an image artifact repository, looking for new versions of images referenced in Kubernetes manifests that were deployed in Weave Cloud.

When the build pipeline we used at that time updated the image the Flux daemon was watching, it would automatically update the deployment manifest, so that the latest semver artifact would be deployed.

In time, it would also invoke kubectl and have that image deployed. In effect, we were connecting an automated CI pipeline with a CD one.

In this way, Flux in particular, and GitOps in general, have gradually become the operational model for cloud native development. Flux is now an incubation projection at the CNCF, and it incorporates a GitOps toolkit — an SDK for building Kubernetes controllers – to further extend GitOps’ reach. 

From Making Sense to Becoming Productive

The process of simplifying software development for the cloud native era has only just begun. Products like Flux and Weave GitOps are used by site reliability engineers or DevOps engineers to build platforms.

Platforms should be development environments in which tooling, workflows and policies are served to developers for them to peruse. The idea is to enable developers to manage Kubernetes from within git.

This way, the entry barrier is pretty darn low, since almost all developers feel comfortable with git’s most basic commands (arguably git init, git add, git commit, git status, git log and git revert would suffice).

Change becomes a gated process that requires approvals, reviews or just visibility across teams. It also becomes cheaper, since branching in git is cheap.

So ultimately, if developers are enabled to manage workloads, config and infra from git, and spinning up a cluster is just two commands away, then scaling any environment becomes fairly easy, and promoting workloads across them even more so!