Are Monolith CI/CD Pipelines Killing Quality in Your Software?

When development and DevOps teams started adding testing to their CI/CD pipelines for monolithic applications, they unlocked enormous benefits. Not only could they deploy faster, with less manual process and intervention to accompany every merge, but they could also deploy with the confidence they weren’t introducing bugs into their production environment.

But microservices changed pretty much everything about CI/CD pipelines. With more complex microservice architectures, spanning multiple clusters and dozens or hundreds of individual nodes, one way for teams to keep up with the complexity of their cluster was to build enormously complex CI/CD pipelines. They tried to reclaim their confidence through sheer robustness.

In the end, they built CI/CD pipelines that resembled the monolithic applications that cloud native got us away from in the first place. Instead of smaller and independent processes for builds, code coverage, reviews, testing and more, CI/CD pipelines were tightly-coupled daisy chains, where each step depended on the last to function correctly.

While monolithic pipelines have implications for every part of the application life cycle CI/CD touches, they pose the greatest threat to the business value of robust testing.

Are Monolithic CI/CD Pipelines Right for Testing?

Unfortunately, testing often gets tightly coupled with monolithic CI/CD pipelines in that they’re entirely dependent on one another. You can’t run any tests without your CI/CD pipeline, and your pipeline can’t finish running without being able to complete your tests and receive a pass/fail in return.

This creates some obscure and complex challenges for both developers trying to push their commits with confidence and the DevOps teams responsible for fine-tuning their pipeline to perfection:

• You might have to re-run your entire CI/CD workflow when you only need the result of a single test. This slows down development. It’s hard to context-switch for a few minutes to continue work on another branch while you wait for your result and excessive runs also tally up into an expensive bill at the end of the month.
• You can’t reuse any test logic across multiple CI/CD tools or workflows. You can, of course, copy-paste your CI/CD logic into multiple steps of your pipeline, but then you’ve also duplicated your maintenance efforts. Every time you make an important change to one iteration of your test, you’ll need to do the same everywhere else.
• You end up with inconsistent execution and reporting. As part of your testing suite, you might use end-to-end tests (Cypress.io), load tests (k6), API tests (Postman and SoapUI), reliability tests (Artillery.io) and more. With all these tests tightly integrated into a monolithic pipeline, you have a half-dozen services and web apps to log into just to see the results of your latest execution. And forget about being able to collate your organization’s testing results into valuable reports without a ton of manual intervention.

This problem got us thinking: What if we built CI/CD pipelines that behaved more like microservices than monoliths? That’s exactly what you get when you decouple your CI/CD pipeline from the way you create, manage, execute and view your tests. It frees your teams to work with the same agility that got you into cloud native and microservices.

Why You Should Decouple CI/CD and Testing

When we talk about decoupling, we’re not talking about disconnecting CI/CD and testing entirely. We are talking about the two being able to operate independently. A change in your CI/CD solution (GitHub Actions, Jenkins, ArgoCD, etc.) shouldn’t mean you need to change your testing workflow or scripts, and vice versa.

Practically speaking, this decoupling means that your CI/CD pipelines can trigger tests to run directly against your test environments through a sequence of simple API calls. This results in small, reusable and composable CI/CD components, which require none of the maintenance of their monolithic past. And because your testing platform (and your testers!) can execute specific tests independent of the state of the CI/CD pipeline, it’s trivial to re-run that one test you need to pass to move your pull request along.

👁 Image

Decoupling unlocks immediate technical benefits:

• You can reuse testing-related steps in multiple CI/CD pipelines by doing API calls to trigger the tests.
• You can easily add or edit existing tests without worrying over inadvertently affecting your CI/CD pipeline.
• Tests can be run whenever needed, both manually by your engineering team and by automation via CI/CD, APIs, etc.

The business benefits might be less obvious, but they’re just as important. Your decoupled CI/CD pipeline lets developers work at their highest velocity by stripping away all the pains and time-wasters from your monolithic past, giving them the confidence to work their magic without worrying they’re going to break production.

Because your tests are more editable and nimble, it’s much easier to fine-tune them to perfect the end-user experience. That’s fewer bugs, better performance and a more stable live application, which all directly tie into your organization’s bottom line, whether your priority is reducing retention or maximizing revenue.

While we at Testkube have been thinking about decoupling CI/CD and testing for a while, it’s just beginning to make inroads in the cloud native world. From the feedback we’re getting from our community already, it’s bound to make an extraordinary difference for DevOps and QA teams — a fast-track to cloud native testing that’s even more powerful than the monoliths of the past.

Using Testkube for Cloud Native Testing in Your CI/CD Pipeline

Testkube is an open source testing framework that allows you to execute, orchestrate and visualize all of your tests in the Kubernetes-native way. It uses custom resource definitions (CRDs) to define and configure tests, letting you execute tests from inside your cluster through API calls instead of complex, tightly-coupled CI/CD pipelines.

There are two approaches you can take when you are trying to decouple your CI/CD pipeline and your testing:

First, there’s the familiar synchronous deployment pipeline. In this case, you remove your tests from your CI/CD pipeline and replace them with API calls to Testkube, which runs inside your cluster. If your tests fail, your pipeline stops on a dime, preventing you from taking your bugs any further down the production life cycle. To figure out where your code went wrong, you jump into the Testkube UI to view your execution and its logs, and diagnose the problem.

👁 Image

The second approach, if you want a more GitOps-style strategy to building a CI/CD pipeline, you can configure Testkube to trigger tests based on the changing state of your cluster or its resources. In this case, Testkube isn’t just operating independently and securely from inside your cluster, with tests decoupled from your CI/CD pipeline.

👁 Image

Your CI/CD and pipeline are two separate processes, with your test executions agnostically reporting just on what it finds in your application. You can then connect Testkube to Slack, or through any webhook, to inform your team when a test fails and how.

Two viable ways to maximize the effectiveness of your CI/CD pipeline in the cloud native era, and as an added bonus, not only does Testkube help you decouple your testing pipelines, it also ensures that:

• You’re free from maintaining test-specific Docker images or lots of script files.
• You’re leveraging Kubernetes’ core functionality, plus the compute of the cluster you’re already maintaining and paying for, to scale test executions, minimizing your reliance on any CI/CD provider.
• You can use multiple testing tools—Postman, Cypress, k6, and more—and view your results in a single dashboard.

Get Started with Testkube

Ready to decouple?

Install Testkube for Mac, Linux, or Windows to build simpler, reusable, and composable tests. Our plug-and-play architecture integrates with the most popular testing tools available like Cypress, Postman, k6, SoapUI, and more.

For tips from other DevOps and GitOps enthusiasts, check out our active Discord or find our entire open source project on GitHub! We’re also on Twitter (@testkube_io), where we’re always sharing the latest news in our journey to become the most complete and beloved Kubernetes-native testing framework.