Many developers still claim they don’t like writing tests, so the idea of generating them using AI was always going to appeal.
Of course, tests are not necessarily a secondary issue — with Test-driven development (TDD) you write the tests first. While it is a good thing every team should try, common practice sees the creation of a minimum viable product (MVP) first, and when there is some evidence this has a future, to then continue the project with full unit testing.
I had not seen
CodiumAI before, but as it has generated an easy-to-understand elevator pitch — “Generating meaningful tests for busy devs” — right on its front page, I’m immediately ready to give it a go. Clearly, they are trying to jump into the tool space opened up by OpenAI’s GPT models — indeed, it is “powered by GPT-3.5&4 & TestGPT-1”.
CodiumAI’s implementation only works with Visual Studio Code and JetBrains for now; I’ll use the former. I find VSC a bit awkward, but at least it should be happy with C#, my preferred tipple. But I know JetBrains tools are extremely popular too.
As a test example, I will use the very simple bank account class with three methods: withdraw, deposit and balance. Let us assume a simple account with
no overdraft facility. We should already have a strong feeling for what we want to test for:
- We cannot withdraw more money than we have.
- If the input deposit or withdrawal value is zero, we should not proceed.
- Neither the requested deposit or withdrawal amounts should be negative.
Our reasoning for these rules come from different places. The first test reflects the business rules of the account. The second test reflects that some expense is involved with using a real banking system system; so while zero may be a valid value, we should not make a call if this is the case. And the third rule is just a restriction on the input range — and that should be handled by using unsigned inputs.
Back to Codium. On the marketplace page, it appears to prefer Python, JavaScript and TypeScript, but it also clearly shows a menu operation to generate examples for other languages. As usual, Visual Studio Code opens in a messy blizzard of windows, but I am able to proceed.
I am asked to log into Github — which is the norm pretty much everywhere — by a small dialog note at the bottom of the screen:
👁 Image
After some authentication tango, I am in. At some point .NET was loaded in and I got it to create a simple console project. Here is the code:
public class BankAccount
{
private uint balance;
public string ShowBalance() => $"{CURRENCYSIGN}{balance}";
private const string CURRENCYSIGN = "$";
public void Deposit(uint funds)
{
balance = balance + funds;
Console.WriteLine($"{CURRENCYSIGN}{funds} deposited. Balance now {ShowBalance()}");
}
public void Withdraw(uint funds)
{
if (funds > balance) throw new Exception($"Cannot withdraw {CURRENCYSIGN}{funds}, balance is {ShowBalance()}");
balance = balance - funds;
Console.WriteLine($"{CURRENCYSIGN}{funds} withdrawn. Balance now {ShowBalance()}");
}
static void Main(string[] args)
{
BankAccount ba = new BankAccount();
try {
// Money in
ba.Deposit(20);
//Money out
ba.Withdraw(5);
//Too much ba.Withdraw(25);
}
catch (System.Exception ex)
{
Console.WriteLine($"Error {ex.Messag}");
}
}
}
From the console, the main method tests give the expected results:
BankAccount> dotnet run
$20 deposited. Balance now $20
$5 withdrawn. Balance now $15
Error Cannot withdraw $25, balance is $15
Now this is not devilishly complex, and there will be no need for any mocking frameworks for example. However, what we are interested in is whether the tool presents usable tests at all.
OK, we can now ask CodiumAI to produce tests. The prompt to do so was shown neatly just above the class:
👁 Image
First of all, the AI-generated a very acceptable English summary of the class:
It gives an English explanation for the three methods. It also generated “code suggestions”, and these were all sensible to some extent. For example, it suggested using decimal values instead of an unsigned int. I will leave the reader, if they are interested, to consider the pros and cons of that.
This gives me confidence that the tests it generates will be good. And they are.
The presentation is very nice. It generates some tests and summarizes other tests it could generate. First, the Happy Path:
👁 Image
Here are two of the generated Happy Path examples:
[Test]
public void test_deposit()
{
BankAccount ba = new BankAccount();
ba.Deposit(20); Assert.AreEqual("$20", ba.ShowBalance());
}
[Test]
public void test_withdraw_more_than_balance()
{
BankAccount ba = new BankAccount();
Assert.Throws<Exception>(() => ba.Withdraw(25));
}
Technically, throwing an exception is not part of a “happy path”, but let’s face it — two years ago, I would have been surprised if an app even grokked what the “happy path” was. So, rejoice.
Each test is nicely presented for consumption in your test framework:
👁 Image
OK, let’s look at the edge cases:
👁 Image
These deal with the zero unit withdrawals — this implies that I can extend these tests and my code to satisfy my need to check that callouts to real systems are not done if not required. Note that you can generate and regenerate a test as needed.
Finally the “other” cases:
👁 Image
The one of interest here is the one that tests the maximum value — if we were not sure about using an unsigned integer, we would want to check on any regression cases concerning changes in the size of the account.
Conclusion
Looking at this tool in the context of available AI tools, I think it is a good example of AI helping with the grunt work of developer testing, while not doing anything too unexpected. When you build up a test framework, you will need set up and tear down methods, and supporting functions that might mean you can’t just drop generated tests directly into your code without alteration. But from my own testing, I would recommend trying CodiumAI out to see if it can be part of your work day. Well, give it a test run anyway.
