5 Strategies for Better Results from an AI Code Assistant

Copilots, like all GenAI, are non-deterministic; meaning they vary in outcome. But while using AI code assistants, developers can use prompt engineering to refine and guide the AI for better outcomes, according to Rizel Scarlett — who is a developer advocate, including recently for GitHub Copilot.

Scarlett is now a staff developer advocate at TBD, a business unit operating within Block that’s building open source platforms and protocols for exchanging money internationally. She shared five strategies for improving a copilot’s results at this week’s InfoBip Shift conference in Miami.

First, she set up a scenario: Imagine, she asked the audience, a developer named Dawson who suffers a bit from imposter syndrome. Fortunately for Dawson, she has a friend who can help — a developer and time traveler named Phil, from Disney’s Phil of The Future, except he’s all grown up.

Dawson has a problem: She has to create an identity verification program, but she doesn’t know how and she’s unsure how to really leverage Copilot to help her, Scarlett said. Phil is from the 22nd century when AI assistants are the norm. He helps her jumpstart her efforts with five strategies for prompting Copilot.

Strategy 1: Provide High-Level Concepts

The first step is to provide the GPT with high-level context. In her scenario, Phil demonstrates by building a Markdown editor. Since Copilot has no idea of the context, he has to provide it and he does this with a large prompt comment with step-by-step instructions. For instance, he tells the copilot, “Make sure we have support for bold, italics and bullet points” and “Can you use reactions in the React markdown package.” The prompt enables Copilot to create a functional but unsettled markdown editor.

Strategy 2: Provide Details

Follow up by providing the Copilot with specific details, Scarlett advised.

“If he writes a column that says get data from [an] API, then GitHub Copilot may or may not know what he’s really trying to do, and it may not get the best result. It doesn’t know which API he wants to get the data from or what it should return,” Scarlett said. “Instead, you can write a more specific comment that says use the JSON placeholder API, pass in user IDs, and return the users as a JSON object. That way we can get more optimal results.”

Strategy 3: Provide Examples

There are three terms to understand when it comes to giving AI examples, Scarlett said:

• Zero-shot learning, in which the model is expected to correctly make predictions for tasks on which they have never been explicitly trained. A human example would be trying to defeat a video game without playing it, but using strategies the gamer learned from previous video games.
• One-shot learning, which provides a single example to the AI. The corollary is being expected to competently play any character and defeat any opponent after playing one match in the game.
• Few-shot learning, where the model is provided with a small set of examples. This would be like doing two to five missions in the new game and then being expected to fully navigate the game.

Strategy 4: Keep A Few Tabs Open

This one may seem a bit surprising, but keeping open a tab or two open in the editor allows GitHub Copilot to gain context from the tabs. Too many open tabs can lower the quality of the Copilot suggestions, she cautioned.

Strategy 5: Use Copilot Chat

Our heroine Dawson loves the advice and results, but she actually would like to get code feedback. Copilots come with a chat function, which can be used for tasks such as fixing bugs, formatting dates, refactoring code, testing code and generating tests, Scarlett said. It can be asked to identify any bugs and then asked to provide a solution with a brief explanation, she said. She then demoed asking GitHub Copilot to generate a test using the open source JavaScript testing framework, Jest. (Microsoft’s Copilot also offers a chat interface.)

By using Copilots, developers can do more than increase productivity, Scarlett added. Copilots can also improve psychological safety, especially in new developers or others who might have imposter syndrome, she said.

“Unfortunately, the reality is psychological safety is not always common at work, especially in second history, and especially for minorities,” she said. “Beginner devs can gain reassurance from a Copilot because it can act as a peer, providing us with ideas when using a new tool for the first time.”