JetBrains Launches New AI Assistant Powered by Multiple LLMs

JetBrains launched a new AI coding assistant on Wednesday that feeds information from the developer’s IDE to AI software to inform coding suggestions, refactoring of code and documentation support. The development tool company claims its AI Assistant is the first such offering to be vendor-neutral because it uses multiple large language models rather than relying on a single AI platform.

“The reason that we’re calling ourselves neutral is because you’re not vendor-locked into one large language model,” said Jodie Burchell, a developer advocate in data science at JetBrains. “It also means you are not locked into a model that will then age, that potentially will not keep up with the latest developments.”

Underlying the platform is both OpenAI and Google’s LLMs, as well as a number of smaller LLM models created by JetBrains. The different models are used for different tasks. For instance, code generation can be straightforward, so it can be solved more easily than a question such as, “Why is this bug happening?” That requires the more nuanced understanding of language offered by larger models, she said. JetBrains’ AI service automatically handles which model gets which queries. By using an AI service architecture, JetBrains can connect new models without the need for users to change AI providers, the company stated.

“If you have a simpler task, you can maybe use a smaller [model], maybe one of our in-house large language models,” she said. “If you need to do something really complex, you’re going to have to use GPT.”

The underlying third-party LLM companies do not store the data, although information is shared in order to create the prompt for the underlying large language models, she said.

“The agreement we have with our providers is that they cannot store the data. They cannot use it for anything other than generating outputs from the model and also for moderation — so basically security checks,” Burchell said. “And we will never ever work with third-party providers that have this sort of agreement where they can use the data for anything other than that.”

Developers aren’t overly keen on turning all coding over to AI assistants — a recent JetBrains survey found that only 17% said they would delegate code crafting to an AI assistant. But, when it comes to helping, 56% of respondents said they would let an AI assistant write code comments and documentation if given the opportunity.

So far, the offering is only available to paying customers, due to the cost of using large language models underneath the hood, Burchell said. The plan is to make it available in other products, but for now, the AI Assistant is available as a subscription add-on for most IDEs, including IntelliJ IDEA, PyCharm, PhpStorm and ReSharper. The exceptions are the community-licensed IDEs and users with educational, startup or open source licenses.

The Dream: On-Premise AI

One possibility that eludes JetBrains and other AI Assistants is the ability to offer such solutions as an on-premise, “bring your own model” version of AI that relies on in-house models, running on a company’s on-premise servers. Some companies are interested in this approach, due to the security benefits.

“That’s obviously going to be the best you’re going to get in terms of security, but there [are] going to be obviously trade-offs with using models that are potentially not as powerful as the third-party providers we’re working with,” Burchell said. ”It’s a trade-off, but this will be a possibility that we are actively looking into.”

One reason it’s difficult to offer an on-premise model is simply that AI runs on extremely large neural networks.

“What this means is, building the model obviously is one thing — [it] takes a lot of time — but actually running the model and getting answers back in real-time is a significant engineering problem,” she explained.

The model has to take a huge amount of context, such as a piece of code run through the model to predict what comes next, she explained. It then runs the new prediction plus the previous information through the model to predict the next part of the sequence. And it keeps doing that, in real time, along with all the networking and security checks.

“Under the hood, what’s needing to happen from an engineering perspective is quite phenomenally difficult,” Burchell said. “It requires GPUs and it requires the cost-effective use of GPUs. So it’s not necessarily an insurmountable problem if you have enough compute to throw at it. But being able to do that in a way that’s not exorbitantly expensive is difficult.”

One interesting side-effect of this process is distillation. This means as the models evolve, the neural networks required do run it to get smaller over time, as it’s able to deliver the same performance with fewer parameters, she said.

Providers are actively working on solving the challenge of LLMs, but it’s one of the reasons why JetBrains is working with third-party providers: Doing so helps keep the cost down and allows JetBrains to offer an affordable product, said Burchell. The company is also exploring open source alternatives, she added.

The other concern some people have is about using their IDEs with an AI system bundled into it, Burchell said. To accommodate those developers, JetBrains has incorporated the ability to disable the AI.