VOOZH about

URL: https://thenewstack.io/testing-copilot-and-chatgpt-as-coding-assistants-what-we-found/

⇱ Testing Copilot and ChatGPT as Coding Assistants: What We Found - The New Stack


TNS
SUBSCRIBE
Join our community of software engineering leaders and aspirational developers. Always stay in-the-know by getting the most important news and exclusive content delivered fresh to your inbox to learn more about at-scale software development.
REQUIRED
It seems that you've previously unsubscribed from our newsletter in the past. Click the button below to open the re-subscribe form in a new tab. When you're done, simply close that tab and continue with this form to complete your subscription.
The New Stack does not sell your information or share it with unaffiliated third parties. By continuing, you agree to our Terms of Use and Privacy Policy.
Welcome and thank you for joining The New Stack community!
Please answer a few simple questions to help us deliver the news and resources you are interested in.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Great to meet you!
Tell us a bit about your job so we can cover the topics you find most relevant.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Welcome!

We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.

What’s next?

Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.

Follow TNS on your favorite social media networks.

Become a TNS follower on LinkedIn.

Check out the latest featured and trending stories while you wait for your first TNS newsletter.

PREV
1 of 2
NEXT
VOXPOP
As a JavaScript developer, what non-React tools do you use most often?
Angular
0%
Astro
0%
Svelte
0%
Vue.js
0%
Other
0%
I only use React
0%
I don't use JavaScript
0%
Thanks for your opinion! Subscribe below to get the final results, published exclusively in our TNS Update newsletter:
NEW! Try Stackie AI
From clobbered drafts to real-time sync
Apr 14th 2026 10:00am, by David Moore
TypeScript 6.0 RC arrives as a bridge to a faster future
Mar 14th 2026 9:00am, by Darryl K. Taft
Mastra empowers web devs to build AI agents in TypeScript
Jan 28th 2026 11:00am, by Loraine Lawson
2024-03-07 08:30:11
Testing Copilot and ChatGPT as Coding Assistants: What We Found
podcast,video,
AI / Frontend Development

Testing Copilot and ChatGPT as Coding Assistants: What We Found

In this episode of The New Stack Makers, Joan Westenberg, a tech writer and consultant, demoed Copilot and ChatGPT.
Mar 7th, 2024 8:30am by Alex Williams
👁 Featued image for: Testing Copilot and ChatGPT as Coding Assistants: What We Found

A great copilot for Han Solo? Chewbacca. A not-so-great? Chewbacca as a large language model.

“And I think that’s funny because we are talking here about something called Copilot,” said Joan Westenberg in this episode of The New Stack Makers.

And so it’s kind of like being Han Solo and turning to Chewbacca and saying, ‘The TIE fighters are after us, we need to raise the shields, jump to hyperspace.” And Chewy, he’s just like, ‘I’m sorry, as a large language model, I can’t help with that. Have you tried fighting the TIE fighters yourself?’ You can’t count on it. So it’s not a copilot.”

Westenberg is a writer, technologist and consultant. Check out her blog. She joined us on this episode of Makers and gave two demonstrations: She showed how Copilot integrates into Microsoft 365, and then she showed how ChatGPT works as a coding assistant.

You can see the demos on our YouTube page. The audio is available in The New Stack Makers audio feed.

Putting the Tools to the Test

“So I’ve primarily been using Microsoft 365 Copilot as of late,” Westenberg said. “And that’s just because it’s kind of this catch-all where Microsoft wants this one tool to bridge the gap between your work no matter where your work is and no matter what your work is. So it’s an extension of what Microsoft was trying to do with GitHub Copilot, and it’s an extension of their work with ChatGPT.

“And what they’ve done here is they’ve created an entire toolset that is meant to draw data from your Microsoft 365 account from the work that you do, from your flow, and be able to sit with you and do that work with you. Whether it’s your coding, whether it’s writing presentations, whether it’s writing documents, and it’s kind of become the big thing that they are hanging their hat on of late.”

Westenberg prompted Microsoft 365. The response came back with an apology. She had her email integrated with the service; she had used Microsoft Teams for meetings. Copilot should have worked.

“I’m sorry, but I couldn’t find any information about the Microsoft Teams meetings you attended,” Copilot wrote back. “Can you please provide more detail or specify a date range for me to search?

Westenberg offered more context.

“Okay, I’ll look for Microsoft Teams meetings on February 5, 2024,” said Westenberg. She recited what Copilot offered back: “I’m sorry, but I couldn’t find any information.”

“It is meant to plug into this stuff,” our Makers guest said. “But it’s not reliable when it comes to actually giving you something.”

Copilot for coding can be decent enough; it is helpful for code snippets but not so much for more complex projects. It comes piece by piece, similar to what it requires to write an article using ChatGPT, in snippets, a few paragraphs at a time.

Check out the demos and how Westenberg tested ChatGPT’s coding skills using a Python plugin. It shows some good stuff. It’s very cool. However, reliability still needs to improve.


For more on the risks and benefits of coding assistants, check out The New Stack’s latest ebook, “Better, Faster, Stronger: How Generative AI Transforms Software Development.”

👁 call to action for ebook on generative AI

TRENDING STORIES
Alex Williams is founder and publisher of The New Stack. He's a longtime technology journalist who did stints at TechCrunch, SiliconAngle and what is now known as ReadWrite. Alex has been a journalist since the late 1980s, starting at the...
Read more from Alex Williams
SHARE THIS STORY
TRENDING STORIES
Microsoft is a sponsor of The New Stack.
SHARE THIS STORY
TRENDING STORIES
TNS DAILY NEWSLETTER Receive a free roundup of the most recent TNS articles in your inbox each day.
The New Stack does not sell your information or share it with unaffiliated third parties. By continuing, you agree to our Terms of Use and Privacy Policy.