Testing an AI-First Code Editor: Good for Intermediate Devs

In an earlier post, I looked at Visual Studio running with Copilot, but that was somewhat clouded by the hype around Large Language Models (LLMs) — in the far-off era of April 2023. Now that the hype has deflated somewhat, we can just look at an LLM-based assistant and determine what it can actually deliver for a software developer.

On the one hand, writing an editor that is built specifically to use AI makes sense, and this is where Cursor AI comes in. I have already heard developers recommend this tool. Cursor describes itself as “an AI-powered code editor that helps you move faster. You can use Cursor to edit code in natural language, fix runtime errors, and locate hard-to-find code.”

But there is clearly a problem with having a straight dependency on OpenAI, which Cursor AI does. This is revealed when you look at the pricing plan and realize they are mixing up ‘slow’ and ‘fast’ GPT-4 calls and GPT-3.5 calls. From a technical point of view this can be changed or placed in the background where it belongs, but they are still having to tie their roadmap to whatever OpenAI gives them access to.

I downloaded the Mac version 0.8.6 from their site. You can sign up, or use your own OpenAI key. Cursor AI itself is a fork of VSCode, which pulls in language extensions to allow it to work with multiple codebases of many flavors — although it is not an IDE. The windows are set up as you would expect but with a chat window on the right side.

It comes with a built-in tutor including examples, and I started with a rust file with a bug in it. I’m quite happy resolving runtime bugs after executing the code, but this gave me a chance to stare at some rust code. I spotted the bug, but couldn’t quite persuade Cursor AI to actually compile the code. (VS Code does little to help you with environments, but this would not have been an issue if I were a regular Rust developer.)

When I initially asked it to find bugs in the code as it was, it found the missing crate. But I was more interested in the basic design flaw:

use std::io::{self, Write};
use std::collections::HashSet;

const MAX_TRIES: u8 = 7;
const WORDS: [&str; 3] = ["openai", "chatgpt", "hangman"];

fn main() {
 let word = WORDS[rand::random::<usize>() % WORDS.len()];
 let mut guessed_chars: HashSet<char> = HashSet::new();
 let mut tries_left = MAX_TRIES;

 while tries_left > 0 {
 println!("You have {} tries left.", tries_left);
 print!("Guessed characters: ");
 for ch in &guessed_chars {
 print!("{} ", ch);
 }
 print!("nWord: ");
 let mut found = false;
 for ch in word.chars() {
 if guessed_chars.contains(&ch) {
 print!("{}", ch);
 found = true;
 } else {
 print!("_");
 }
 }
 println!();

 if found {
 println!("Congratulations, you've won!");
 return;
 }

 ...
 }

 println!("You've lost! The word was '{}'", word);
}

...

While I didn’t spot the crate (or library) missing, I did spot the less rust-specific problem. What was supposed to be a hangman-style guessing game would let you win with just one correct guess.

The second time I asked Cursor AI to find the bug, it spotted it correctly:

“Yes, there is a bug in this code. The problem is with the found variable and its usage. The found variable is set to true if any character in the word has been guessed correctly. However, the game is declared won if found is true after checking all characters in the word. This means the game will be declared won if the player has guessed at least one character correctly, not necessarily all characters.”

Even given that this was an isolated problem in a provided example, this was still an excellent find. The summary description is sufficient to guide someone unfamiliar with the game or the code. While it is better to find the bug through writing unit tests, which would also prevent you from writing code like that above, if you were given some code to consider then this would save you some time.

I got a nice email from Cursor giving examples and a warning that using AI “takes a bit of learning and effort.” Fair point. I used the examples to explore further.

C Examples

“Select 20 lines of code and hit Cmd+K to prompt an edit”. I used the C example, which featured a linked list and methods using it. When I isolated a bit of code and asked for an edit, it made some sensible suggestions but also wrote explanation text into the editor pane. More appropriately, it used the green and red addition/subtraction interface you might get in a merge dialogue to place the changes:

👁 Image

I didn’t quite understand how this worked with Undo, so did occasionally get a bit lost in the process. The suggestions themselves, designed to avoid a classic null pointer bug, were solid enough.

“Click ‘with codebase’ in the chat to find a piece of code”. In the linked list C code I copied b->prev into the chat window and then hit “with codebase”. It then gave a very good description of the double link list that was being used in the example. It didn’t get confused that the b was just a local variable.

This slightly reminds me of those cinema reviews you can only wholly appreciate after you’ve seen the film. However, for a mid developer given a fresh piece of code, this can quickly get them up to scratch.

“Ask a question about a library using ‘@’ in the chat”. I selected the line #include <unistd.h> from the C example and added it to chat, then asked chat to chat about it. It did the important thing, which was to explain where in the code the library was used and for what. Again, this won’t be any good for an absolute beginner, but for someone using a new language, this is more efficient than continual checks with Stack Overflow. However, the often bickering perspectives in that forum often shine a stronger light.

Rust Translation

Finally, I did a translation. This is undoubtedly a strong feature, to which the tool is well suited. I selected a small Python function from the example files and translated it to Rust. Again, I am using the examples given because they seem perfectly valid:

👁 Image

For your own inspection, the code is:

def generate_random_number(output):
 # Simple pseudorandom number generator using system time
 random_number = time.time() - float(str(time.time()).split('.')[0])
 random_number = int(random_number * (10**6))

 # Ensure the number is different for different processes by adding the process id
 random_number += os.getpid()

 output.put(random_number)

The response was:

👁 Image

The short explanation at the bottom is more functional than useful.

The code generated was:

use std::time::{SystemTime, UNIX_EPOCH};
use std::process;
use std::sync::mpsc::Sender;

fn generate_random_number(output: Sender<u64>) {
 // Simple pseudorandom number generator using system time
 let start = SystemTime::now();
 let since_the_epoch = start.duration_since(UNIX_EPOCH)
 .expect("Time went backwards");
 let in_ms = since_the_epoch.as_secs() * 1000 +
 since_the_epoch.subsec_nanos() as u64 / 1_000_000;
 let random_number = in_ms - in_ms / 1000 * 1000;

 // Ensure the number is different for different processes by adding the process id
 let random_number = random_number + process::id();

 // Send the random number to the output channel
 output.send(random_number).unwrap();
}

While I don’t use either of these languages professionally, I think we can see this code has added an amusing warning if the current date is set to earlier than 1970!

I can’t help but think this is a tool that might fall between the professional developer and the junior getting up to speed. But it looks well suited for intermediate tasks like translation.

I don’t really appreciate a chat window in my editor — but then again, I’ve seen someone build code from a prepared ChatGPT solution. The UI affordances of AI within development are still new, so it isn’t fair to say it doesn’t feel natural yet. Right now it doesn’t quite work with undo, and really explanation text should never enter the editor pane.

And yet the future is probably here. When cars first appeared, to even start the engine needed an intimate knowledge of combustion and a crank handle. While the software profession still venerates the master craftsman, we surely must be entering an age where code manipulation, on average, will be manageable by engineers with much less experience. So technically, the true mass audience for this tool may not arrive for a few years yet.