Continuous Verification: The Missing Link to Fully Automate Your Pipeline

It would be hard to find any organization shipping software that is not under pressure to deploy more often with fewer defects and greater feature velocity. Inconsistent processes and manual operations are some of the common issues preventing teams from realizing their potential with regard to software delivery. In response to these challenges, much as been written about the benefits of adopting Continuous Integration and continuous delivery (CI/CD) practices on software velocity.

However, many applications are not ready for use at the completion of their deployment pipelines. Post-deployment work to “operationalize” or “harden” services is still fairly common practice and generally slows the overall rate of software delivery. The numbers tell the story: a by Enterprise Management Associates (EMA) found that for 63% of the organizations surveyed, less than half of the end-to-end continuous deployment processes were automated. Only 6% characterized their process as 90 to 100 percent automated.

In this post, we describe how an alternative practice, Continuous Verification (CV), can reduce and potentially eliminate these post-deployment actions in order to accelerate the pace and reliability of software delivery.

VMware defines CV as:

“A process of querying external system(s) and using information from the response to make decision(s) to improve the development and deployment process.”

This definition depends on the existence of a deployment process, which includes one or more pipelines for CI and/or CD. Continuous verification is an optimization practice, where continuous integration and delivery are output driven, resulting in an artifact or deployment. In many organizations, a series of post-deployment activities are required before a deployed application is ready for its intended users to take advantage of its features. Often, these steps have not been included in the pipeline because they are manually performed or customized scripting drives them, which was not designed to be part of a broadly orchestrated process.

Each human-driven interaction with the deployed application creates additional time in which the application is not delivering value to the organization. Shifting these activities into the pipeline increases the overall efficiency of the process and reduces the operational “lag” between deployment and availability. These efficiency gains are realized through an overall reduction in human-driven actions.

CV can augment the process pictured above by moving some or all of the post-deployment steps into feedback loop(s) within the pipeline. Once these activities are codified into the pipeline, they are executed reliably and predictably according to the procedure of the pipeline(s) and the logic defined by the organization. This uniformity of approach and action decreases variability between deployments and increases the reliability of deployments. These trends, in turn, allow humans to focus on designing the policies necessary to further increase the scope and efficiency of the organization’s pipeline(s). We’ve written about these factors in greater detail in a piece titled “”

As these post-deployment steps are incorporated into the pipeline, the overall efficiency increases. These “verification” steps create feedback loops capable of altering the eventual output of the pipeline. As more of these operations occur, the time to deployment (Td) trends lower and the deployments become a more accurate reflection of organizational policy.

What are some examples of human-initiated verifications?

• Verifying the utilization and/or cost of the resources is not exceeding authorized limits;
• Validating that the infrastructure (AWS VPC/AWS EKS/etc) configuration follows organizational guardrails;
• Understanding if developers have exposed vulnerabilities in artifacts from CI;
• Ensuring the application performs within latency limits on the infrastructure
• Confirming the application authorized to use the correct set of resources and services.

Depending on your internal requirements and processes some of these post-deployment steps can be integrated into one or multiple stages of a CD pipeline.

In most cases, the cost of deployment is managed on an ongoing basis. While business operations is continuously monitoring costs (pre/post-deployment), check on whether your budget is exceeded is done post-deployment. However, automating a simple check with tools like Cloudability, or Cloudyn in the CD pipeline prior to deployment helps reduce not only the time of deployment but also removes any rollbacks due to overages. Our team has detailed out how you can achieve this in one of our blog posts: “.”

Prior to deploying software, it is important to know if the infrastructure on which that software resides is correctly configured according to organizational policy. Without that assurance, deployments may fail or be compromised from the start. There are many options available, from open source tools (Cloud Custodian, etc) to off-the-shelf products (Redlock, , etc.). An example of this concept in action is detailed in our article, “.”

One very important verification must be done prior to initiating ANY deployment is ensuring the built artifacts are not exposing any vulnerabilities. Whether its an AWS AMI, an executable, container image, etc, that artifact should be checked with tools like Clair (for container vulnerabilities) post build or during the build with tools like OWASP dependency-check (java builds, etc). There is an abundance of vulnerability analysis tools, and these should always be added in the CI process and just prior to deployment. We’ve detailed out how to analyze vulnerabilities with docker containers in our blog post: “”

These are just a few examples of how CV helps organizations to deploy production-ready software more quickly. Many other policy-driven tasks can be moved “into the pipeline” in order to increase the efficiency of the deployment process and bring “production-ready” deployments closer to reality.

To continue the discussion on this topic, or others related to modern applications and public cloud, please visit us at .