This week OpenAI gave us access to GPT-4o (“o” for “omni”), which aims to compete better in speech recognition and conversation. It’s is almost certainly a stronger LLM, too. But can it do a code review? I put it through some tests to find out.
To start with, I checked if it could do long multiplication:
👁 Image
This is correct, as you can confirm on any online calculator. Note that it doesn’t attempt to “work through” the result for you — it has just passed it to a calculator. The “view analysis” icon at the end basically confirms this. Finally! This has failed to work on previous models.
I have no personal interest in formal code reviews. Yes, sometimes a junior developer might actually demand one, but unit tests and pairing sessions will usually suffice in sanding away bad code practice. But a few mumbled comments can help indicate mild unease with code, without appearing to be imposing some kind of authoritarian order over easily rewriteable snippets. In short, a code review is more of a social interaction than anything. Focus on engineering issues, not code semantic detailing.
However, code reviews are a good LLM use case for developers. Here I’ll look at three different bits of code, each of which simply draws a box with a diagonal line. I’ll add problems and see how GPT-4o (via ChatGPT) reviews them.
Here we start with our simple diagonal line routine in C#:
using System;
public class Program
{
public static void Main()
{
BoxWithDiagonalLine(5);
BoxWithDiagonalLine(6,1);
BoxWithDiagonalLine(6,-1);
BoxWithDiagonalLine(2);
BoxWithDiagonalLine(-1);
}
// Example 1
public static void BoxWithDiagonalLine(int boxsize, int linepos = 0)
{
for(int i=0; i < boxsize; i++)
{
for (int j = 0; j < boxsize; j++)
{
if (j == linepos) Console.Write("x ");
else Console.Write("o ");
}
linepos++;
Console.Write("n");
}
Console.Write("n");
}
}
You can throw this into
https://dotnetfiddle.net/ (set the compiler to .NET 6 at least, though) and see the responses:
👁 Image
This is fairly simple, and relatively self-explanatory. An outer loop controls the rows, an inner loop places the “x” in the right place. The variable naming is minimal, but the code is short. There is a default value for the method to give a full diagonal line from top left to bottom right. Whereas we can move the start line position left or right, it doesn’t make much sense to have a negative boxsize. So we could improve it by turning the method parameter to an
unsigned int. This would immediately stop the last example from even compiling. A wildly large linepos will not be visible, but it isn’t dangerous.
So far, there is little to comment on.
In the second example, I will replace the
for loops with
while loops:
// Example 2
public static void BoxWithDiagonalLine(int boxsize, int linepos = 0)
{
int i = 0;
while(i++ < boxsize)
{
int j = 0;
while (j++ < boxsize)
{
if (j == linepos + 1) Console.Write("x ");
else Console.Write("o ");
} linepos++;
Console.Write("n");
}
Console.Write("n");
}
Now this works, but deserves a slight eyebrow raise. By post-incrementing the loop variables (the
++ after the
i and
j mean the values are read first before they are increased by one) we save some space, but we need that ungainly
linepos + 1 expression. A
while doesn’t express the beginning and end state quite so nicely as a
for loop does — but we are not sliding into any madness.
We make one last change. We now have a variation that does the wrong thing. It tries to use
linepos to control the outer loop:
//Example 3
public static void BoxWithDiagonalLine(int boxsize, int linepos = 0)
{
while(linepos < boxsize)
{
int j = 0;
while (j++ < boxsize)
{
if (j == linepos + 1) Console.Write("x ");
else Console.Write("o ");
}
linepos++;
Console.Write("n");
}
Console.Write("n");
}
In an unnecessary attempt to remove one loop variable, the box itself will no longer be complete — although it will work with the default value. In fact, this solves a different problem, always reliably finishing with a cross in the bottom corner.
👁 Image
This poor code could probably come about because the developer continued to use a
while inappropriately.
So let’s summarize the way we can look at the different versions:
- The first example is fine, but we could force the parameter to be positive. Although, it is not crystal clear what the code does without comments.
- The second uses a
while loop, which doesn’t quite have the certainty of a for loop. It also makes for an ugly expression to take into account the post-increment.
- The third example is plain wrong, even though it gets the result right in the default case
What do we expect from an
AI code review? First of all, I have given it no comments, so will it work out what the code does? Only the name of the method reveals this.
Here is GPT-4o response:
“Your code is functional and effectively generates a box with diagonal lines based on the provided parameters. However, there are a few improvements and edge cases to consider:
- Input Validation: Ensure that the box size is positive and greater than 1, since a box with a size less than 2 doesn’t make much sense.
- Handling
linepos: Improve handling of diagonal positions to ensure the diagonals are correctly drawn when linepos is negative or greater than the box size.
- Code Readability: Add comments and improve variable naming for better readability and maintenance.”
So it kind of did understand what the code did — even if it thought the program drew “lines” not a “line”. This is still very good.
The code it suggested was the following:
// ChatGPT4o suggested code
public static void BoxWithDiagonalLine(int boxSize, int linePos = 0)
{
if (boxSize < 2)
{
Console.WriteLine("Box size must be greater than 1.");
return;
}
// Adjust linePos if it's out of the box boundaries
if (linePos < 0) linePos = 0;
if (linePos >= boxSize) linePos = boxSize - 1;
for (int i = 0; i < boxSize; i++)
{
for (int j = 0; j < boxSize; j++)
{
if (j == linePos) Console.Write("x ");
else Console.Write("o ");
}
linePos++;
// Reset linePos if it goes out of the box boundaries
if (linePos >= boxSize) linePos = 0;
Console.Write("n");
}
Console.Write("n");
}
Here are the results of putting this into dotnetfiddle.net:
👁 Image
I think most people would be happy with the size less than two guards. It also camel-cased the params nicely, as this is right for C#. Unfortunately, it missed the idea of changing the position to allow for the line to start lower down. And by resetting the
linepos, we get that unfortunate error at the bottom of the second example.
However, as code review, it hit the right note.
More Examples
Let’s feed it the second example and see how it does. Let me pick out the interesting comments:
“The use of
while loops is functional but could be less intuitive compared to
for loops for such iterations.”
Excellent. This is exactly the right note.
“The condition
j == linepos + 1 is used to determine where to place “x”. This works but can be made more intuitive by aligning with the index
i.”
Again, this is good. It understands the difference between functionality and readability.
The produced code clamped the value of
linepos, and rather oddly used the modulo function to change the value. But it ended up with the same result. It reverted to
for loops, which is excellent.
Finally, let’s deliver the bad Example 3, and see how it fairs. Oddly, it didn’t spot the bug, but did suggest the following:
“Edge Cases:
– When
linepos is negative, it works but may not be intended.”
This is an indication that it didn’t quite understand the code — although, again, I never explicitly stated what I was doing. So perhaps a wrapping line may have been a reasonable guess?
Conclusion
Overall, I was quite impressed with the nature and the delivery of the review. The code GPT-4o produced was less promising, but it did ok. There was no “great clomping foot of nerdism” in the criticism of the code; instead, it gently nudged the reader in a better direction.
This is perhaps not where I expected to end up — with a good attitude but slightly inaccurate code. However, I would be happy to let this review be seen by a nervous junior; in that respect it does a slightly better job than some humans.
