Engineering Best Practices of CI Pipelines

Before the popularity of microservices architecture, developers had to build and deliver large software applications as single entities. Today a number of smaller applications are used simultaneously, each one focusing on a single aspect of business functionality. With smaller applications and teams with members with diverse backgrounds often working remotely, increased collaboration and reduced barriers between teams are essential features of an effective production environment.

DevOps and CI/CD

DevOps brings development and operations teams together, creating a cultural shift characterized by cooperation, collaboration and partnership between development and operations teams. This shift integrates the processes involved in the life cycle of microservice applications, enabling organizations to deliver products to market faster and more efficiently.

Continuous integration and continuous delivery or deployment (CI/CD) are DevOps practices that streamline the process of creating and deploying applications. The term “continuous” in CI/CD represents two essential characteristics:

• First, the application’s integration and delivery will be continuous, with minimal human interaction and a high level of automation.
• Secondly, the software packages built and deployed will each be as small as possible, allowing the incremental delivery of software updates.

Shift Left

The most significant advantage of CI/CD systems in the cloud native era is that they enable “shift left” practices. Shift left means working to guarantee an application’s security at the earliest possible stage in its development, which in practice means performing tests, integrations and quality checks during the initial stages of the software development process.

Although CI/CD has been part of the software development landscape for a number of years, as yet there is no one-size-fits-all CI/CD platform. Since organizations have different needs and objectives, their expectations from their CI/CD platforms are also diverse. However, there are well-established best practices when it comes to cloud native CI/CD, no matter which platform you’re using.

Best Practices for CI/CD

Let’s look at strategies implemented by software giants including Google, Facebook and VMWare, so you can benefit from their experience and apply their strategies in our own CI/CD pipelines.

Conduct Smaller Changes and Frequent Builds

Continuous integration relies on the fact that all source code, configuration and dependencies are defined in the source control repository. Any time a change is made to data in the repository, the CI system will trigger a set of tasks to ensure that the change has been integrated successfully, without any conflicts.

To benefit from CI/CD systems, each team member should deliver their changes in small chunks to the source code repository. By performing minor changes and frequent builds, engineering time used on integration will be minimized, reducing inefficiency and related costs. Small changes also help to keep delivered and deployed packages manageable and controllable. This means that both users and operators will be presented with less complexity in the updated application.

Automate Tests and Green Builds

The essence of a CI/CD system is to aim for green builds and to resolve issues quickly when a red build occurs, meaning a test failed. When the automated tests run, any failure results should be visible to all team members. Then, it should be a top priority for the team to make the build work again.

Green builds and rapid fixes are critical for two reasons. First, when tests are failing, it is not possible to test forthcoming development and changes accurately. Secondly, continuous deployment will be halted, because no new and validated packages exist. Although it may seem like a frustrating situation to stop active development and instead focus on fixing failed tests, this mindset will ensure optimal application stability.

Create a Gatekeeper for Production

An efficient CI/CD system should be the only path that leads to the production environment. In other words, if you have confidently built a CI/CD system with a comprehensive set of tests, there should be no other way to deploy applications to the production system. It can be highly tempting — and common — to maintain administrator privileges and deploy an application to the production systems just once. Most often, this problematic practice happens when someone is tempted to test a new feature quickly in production, or fix an issue with untested code. Since these changes haven’t been tested thoroughly, they can cause bigger breaks and create security issues. However, you undermine your processes and prior efforts when these applications have yet to pass all the CI/CD system stages and tests.

Use Deployment Strategies

Deployment strategies are the techniques used to launch a new version of the software. They are a key aspect of DevOps, since they help to minimize downtime and risk, and detect problems early in the development cycle. In addition, deployment strategies fit in well with the concept of making frequent but minor changes, performing frequent deployments and automation. Some well-known deployment strategies include:

• Rolling upgrade deployment: Incrementally releases new updates and removes old versions if there are no issues indicated in testing.
• Canary deployment: Releases new features to a subset of users initially, checking if there are any issues in practice before rolling out the feature to all users.
• Blue/green deployment: Requires two production-ready environments, one idle, often termed “blue,” and one active, labeled “green.” Changes are tested on blue; if everything looks OK, the blue environment is set to active while the green becomes idle.

Most cloud native platforms, such as Kubernetes, support zero-downtime runtime deployment strategies out of the box. You only need to configure your deployment strategies in the continuous delivery stages of your CI/CD pipeline.