DevOps in 2024: Automate First, AI Second

Enterprise IT teams are struggling to keep up with business needs. They are constantly being asked to do more with less. This is why it’s important to identify and overcome the bottlenecks they face. The problem most often comes down to their inability to automate more of their processes.

The news is full of articles about how AI copilots will boost productivity, but success in DevOps is about people and processes in addition to tools. Most users are typically so far behind that they don’t have the time to focus on improving their processes and educating their people to use these new tools properly.

Going Faster and Safer

Orchestrating CI/CD processes is one aspect of improving DevOps performance, but testing automation should come first. Why?

This can be answered by the old trope about the purpose of brakes in your car. The purpose of brakes is not to stop your vehicle — that’s what they do — but to enable the car to go fast and safely. That’s also the purpose of testing. Investing in testing automation is about allowing your teams to go faster and safer.

According to a 2023 survey by Forrester, 57% of global developers sometimes or always skip writing unit tests for their code. And 27% of organizations don’t even do formal code reviews! Got no time! Got to get it shipped!

The majority of users are manually testing new features before release with no automated regression testing. When we ask why they are not automating, they cite the lack of resources and time. I get that. When I ran the development and quality team at my previous company, I also struggled with that challenge. But that’s like paying the minimum on your credit card bill. You will never get out of debt that way.

So, how do you automate your testing when you lack the resources to do it?

Code copilots are well-known and adopted by many developers, whether their companies have approved their use. The trend that does not get the same visibility is using generative AI (GenAI) for test development and code review.

Developing Tests

Both unit tests and functional tests can be created using GenAI.

In the case of unit tests, the developer generates the code first, either on their own or using the help of a code copilot, and the AI then generates the test coverage. The same GPT copilot that generates the code is used to generate the unit tests. Remember that just as it is the developer’s responsibility to review the code generated by AI, it is also critical that they review the unit tests, especially if AI generates the code.

The code doesn’t have to exist for functional tests before the test is created. The same user story that defines the feature can be used to generate a happy path functional test. This is called intent-based test generation, and it can generate a happy path test.

This supports the notion of shift left by providing the developer with a test to validate that their code works. Negative tests are still quite essential, and while AI can help there, a subject matter expert (SME) is better equipped today to identify the critical corner cases. Stay tuned for developments in this area shortly.

The key to good unit and functional tests is ensuring the user story is complete and well-written. As AI generates more of the solution, the user story will require more information than before. The good news is that AI can be used to help enrich user stories as well.

While developers will spend less time writing code and tests in the future, they should prepare to spend more time providing architectural guidance and reviewing code and AI-generated tests.

Reviewing Code

AI can also help review code. Developers can ask the AI to walk them through the code and even ask specific questions about the libraries and APIs used. If a section of code is not up to snuff, the developer doesn’t have to ask for an entire rewrite; they can request that a specific section or class be rewritten to address a perceived gap. This method of interacting with AI will be a necessary skill in the future.

Likewise, with unit tests. The developer should review the test to ensure both the positive and negative tests are sufficient, and request an update or expansion if appropriate. Again, the developer will need to learn how to ask for the specific changes they require. This is an acquired skill that should be practiced.

The developer must learn to interact with the AI agent to get what they need. Some platforms require a minimum percentage of code coverage. If there is insufficient coverage in the first pass, the developer can explain the goal and ask for more. Results may vary between different tools, but with AI, half the battle is learning how to ask for what you need.

Benefiting Business Users

Even business users can benefit from AI regarding user acceptance tests and expansion of the test automation library. Business users are the SMEs when using the applications, but they are not test engineers. New tools are emerging. However, even now they can watch a user explore the application’s functionality, create automated tests from their session and even file defects directly to the developers with details on how to duplicate the bug. In the future, AI agents can learn how SMEs approach the application and gain a better understanding. Eventually, the AI will create the first-pass tests for the SME to review.

So, the important trends in DevOps this year are the increased focus on using GenAI to improve testing to increase deployment frequency, reduce the lead time for changes, and reduce the change failure rate, three of the four DORA metrics. This will not only boost developers’ productivity but hopefully free up more time for them to focus on training and improving their skills using these new tools.