Solving 3 Pervasive Enterprise Continuous Testing Challenges

Global 2000 organizations are increasingly being pitted against nimble startups set on disrupting their industry. Accelerating application delivery is a key part of maintaining a competitive advantage, but it’s hard for enterprise organizations to keep pace. They’re burdened by everything from complex application stacks, including mainframes, packaged applications and decades-old custom applications, as well as modern interfaces, to deeply-entrenched processes built for dramatically different delivery models, to strict compliance requirements. At times, it can seem like you’re competing in the Tour de France while towing a child in a bike trailer.

All too often, it’s testing that’s holding back the speed of application delivery — specifically, lack of a mature continuous testing practice that provides near real-time feedback on whether the application has an acceptable level of business risk. Nobody doubts that continuous testing is essential. But making it a reality is a challenge in any development environment, and it’s an especially pointed pain in enterprise environments, with all the burdens and baggage mentioned above.

Here’s a look at three of the challenges enterprises commonly face when trying to implement an effective continuous testing practice and how top organizations have solved them. These stories were all presented publicly at Tricentis conferences.

1. Introducing and Expanding Test Automation

Having a core set of automated tests that expose business risks as they’re introduced is a key component of continuous testing. Yet, the latest World Quality Report found that test automation has been stalled around a dismally low 15% for years now. You can’t, and shouldn’t, automate all your testing efforts. But if you want fast feedback on whether the latest changes broke any critical business processes, ramping up automation is essential.

The problem is that getting started can be hard and keeping it going is, unfortunately, even harder. Most large organizations have at least one or two failed test automation initiatives under their belt, so quality leaders are understandably reluctant to spearhead another.

Yet, many brave QA leaders are up for the challenge and achieve success. What’s their secret? At a high level, they remain laser-focused on the long-term goal of sustainable test automation, so they architect the deep-seated process and cultural change to support that. Getting it started requires finding a test automation approach that crosses the various technologies involved in your business processes (technology-agnostic or broad technology support) and can be adopted rapidly and broadly by your existing team members, whatever their current skill sets may be. Keeping it going also requires addressing what I call “the three nightmares of test automation”: test maintenance, test data and test environments.

Here are some tips from how Linde, the world’s largest gas company, introduced test automation into the 100+-year-old company’s complex web of highly-customized SAP, Salesforce, web and mobile apps:

• Think long and hard about where automation will have the greatest impact, then focus on creating stable automation for those business-critical use cases. You can gain much more attention and internal support by automating a handful of high-impact end-to-end processes than you can by automating thousands of ill-conceived tests.
• Root out the “number of tests” mindset. For years, many organizations have been using a number of tests to measure the extent of testing and even to compensate testing providers. More tests that address the same business risks don’t help. In fact, they hurt. They consume resources to create as well as resources to maintain, eating into your test automation ROI.
• Sometimes you need to cut your losses. If an approach or tool really isn’t working for your existing resources (e.g., it’s too technical or doesn’t suit your ecosystem), understand what’s needed instead and resolve to address it going forward.

2. Assessing the Release Readiness of New Functionality 

A primary goal of continuous testing is to determine if a release candidate is ready for production. As described above, you absolutely need to ensure that the changes in each release don’t break existing functionality. But you also need to test the new functionality to ensure that it works and meets expectations.

Making the ultimate go/no-go release decision can be a bit of a guessing game when different teams are responsible for different components and layers of the application: the browser interface, the mobile experience, the various packaged apps at work behind the scenes (SAP, Salesforce, ServiceNow), and all the microservices, APIs and integration platforms that are probably gluing it all together. They’re likely developing new functionality at different cadences and testing their parts in different ways, using different testing practices and different tools. But the user doesn’t make those distinctions. They expect it all to just work, flawlessly.

Moët Hennessy-Louis Vuitton (LVMH), the parent company behind luxury brands such as Christian Dior, TAG Heuer and Dom Perignon, recently decided to streamline its testing process to support ambitious plans for e-commerce growth. It perfected the art of testing new functionality efficiently and making release decisions that ensure the ultimate user experience:

• Take a structured approach to testing new functionality. To better defend both the customer and the brand, each QA team took a deep dive into the DNA of the brand and its core customer profiles (personas). They then developed new test strategies for checking how the customer experiences the brand and captured them as reusable rollout kits.
• Reuse, reuse, reuse. They built a library of strategically designed test building blocks that enable 70 to 90% reuse across launches. These building blocks map to an array of test automation tools that they combine to view, interact with and evaluate the site like a human on a broad variety of devices. With this head start on automation, they can start testing early in each release cycle and gain fast feedback as they implement and refine new user stories.
• Connect the dots. All the disparate testing data for web, mobile, Salesforce eCommerce, ERPs, warehouse and customer relationship management systems, etc., is integrated into a single pane of glass that provides real-time insight into release readiness, by persona, feature and tech layer. They can always tell at a glance what’s ready to release, what’s holding the release back and who’s responsible for getting it on track.

3. Balancing Collaboration With Autonomy

For years now, the trend has been toward autonomous, self-governing development teams that choose the practices and tools best suited for their culture and projects. This is great for team motivation, but it’s not ideal for collaboration. At the same time, another competing trend, toward highly interconnected applications, has made collaboration even more critical. At the points of intersection, there’s a huge opportunity to share test artifacts as well as code and deployment artifacts. However, that’s easier said than done now that so many teams have grown accustomed to using their own processes and tools.

How do you promote cross-team continuous testing collaboration without homogenizing all the distinct ways of working that different teams have invested considerable resources in developing and perfecting? Dell has devised an effective strategy to tap synergies across 30 divisions working across more than 20 different test automation frameworks:

• Abstraction is key. They analyzed the elements of DevOps success from a high-level philosophical perspective, then crafted an abstract CI/CD/CT architecture that specified what activities to address (source control, requirements management, test management, etc.) without prescribing low-level implementation details on how to complete them.
• For continuous testing, they established an overarching layer that enables them to find and run relevant test assets without even thinking about which test automation framework is used behind the scenes. This way, everyone can share, but no team has to compromise.
• For new projects, they plan to standardize on a test framework that’s flexible enough to cover their various application stacks and delivery cadences. They will make it available to existing teams and projects, but nobody is being forced to change.

Focus on the Right Approach

There’s no such thing as enterprise continuous testing in a box. As you can see from just these few examples, the top challenges and potential solutions vary widely, and yours will too.

To succeed, you really need to get real. Take a long, hard look at what you’re working with and craft an approach accordingly. Don’t expect long-time manual testers to download some open source testing tools and automate processes that span packaged applications, custom applications, mainframes and more. Likewise, don’t expect a high-performing, tech-savvy team to give up the home-grown or highly customized approach they poured significant time and resources into over the years.

Continuous testing itself is all about assessing the impact of change. Your approach to continuous testing must also be built around change: What can you do from a technology, process and change management perspective to ease the transition from your current state to an enterprise continuous testing process that’s achieving your organization’s goals with respect to quality, speed and efficiency.

If you’d like to explore these and other enterprise Continuous Testing challenges in-depth, I invite you to read my recent book, “Enterprise Continuous Testing: Transforming Testing for Agile and DevOps.”