Opportunities and Limitations of Deploying Large Language Models in the Enterprise

Generative AI is all the rage. In the last few months, we’ve seen an explosion of large language off-the-shelf and open source models (LLMs), like Meta’s Llama 2, OpenAI’s GPT-4, Anthropic’s Claude 2 and tools like ChatGPT code interpreter and GitHub Copilot. The ecosystem is exploding and it’s quickly transforming various industries.

Customer service and support is one area seeing massive gains. By leveraging LLMs, organizations are providing faster and more personalized responses to customer inquiries than ever before. One example is Delta Air Lines’ “Ask Delta” chatbot that uses generative AI to help customers find flights and check-in and track their bags, which in turn has reduced call center volume by a whopping 20%.

In marketing and sales, many organizations are using ChatGPT and other generative AI solutions to generate marketing copy and score leads. In HR, many CHROs are now using large language models for recruiting, performance management, and coaching.

And let us not forget the strides Generative AI is making in the world of software development. Solutions like GitHub Copilot and Amazon CodeWhisperer are enabling developers to generate code faster and more accurately, reducing the time and effort required for routine tasks.

I can go on, but to sum it up: every business wants to take advantage of generative AI, but actually applying it is more difficult and laborious than it seems.

But while off-the-shelf models are helping many companies get started with generative AI, scaling it for enterprise use is difficult. It requires specialized talent, a new tech stack to manage and deploy models, ample budget for rising compute costs, and guardrails to ensure security.

The Good: Businesses Can’t Afford to Sit This One Out

The progress we’ve seen in the last few months is nothing short of impressive. While natural language understanding and processing is not net-new, it’s now much more accessible. Not to mention that models have gone from 0 to 60 in terms of depth and capabilities.

But, for many CIOs, the value may not be immediately obvious. Many organizations have been slashing budgets in the last year and making blind investments is not in their agenda. But this is not the inning to sit out. AI has the power to shape your business in unimaginable ways. Here’s a quick list of benefits:

• Instant access to the world’s knowledge: These models are trained on all publicly available data, making the entirety of human knowledge easily accessible through APIs or chat prompts.
• Human-level understanding of language: These models possess the ability to understand and generate language, enabling partial or full automation of various enterprise workflows involving language comprehension and writing.
• Code interpretation and generation: Advanced models like GPT-4 Code Interpreter can understand and generate code, allowing seamless interaction with traditional software systems in enterprises.
• International support: With 20+ language support out of the box, these models enable global reach and multilingual applications effortlessly.

The Limitations, for Now

Large Language Models (LLMs) like GPT-4 are based on neural networks, which are inherently probabilistic in nature. This means that given the same input, they might produce slightly different outputs each time due to the randomness in the model’s architecture or during the training process. This is what we mean when we say LLMs are “non-deterministic.”

This non-deterministic behavior can be a limitation in building enterprise-grade business applications for several reasons:

• Consistency: Businesses often need reliable and consistent results, especially when dealing with sensitive areas like finance, healthcare, or legal matters. The non-deterministic nature of LLMs can lead to inconsistencies, which can be problematic in these contexts.
• Auditability: In many industries, it’s important to be able to audit and trace back decisions made by automated systems. If an LLM makes a decision or recommendation, and later can’t reproduce the same output, it makes auditing and accountability difficult.
• Predictability: In many business scenarios, it’s crucial to be able to predict the system’s behavior based on certain inputs. With non-deterministic models, it’s harder to guarantee specific outputs, which can make planning and strategy development more challenging.
• Testing: Testing is an essential part of any software development process, including the development of business applications. The non-deterministic nature of LLMs can make it difficult to write and run tests that produce reliable, repeatable results.
• Risk Management: Given the probabilistic nature of LLMs, there’s always a degree of uncertainty in their outputs. This can increase risk in business applications, especially in sensitive domains.

Despite these challenges, there are ways to manage the non-deterministic nature of LLMs, such as using ensemble methods, applying post-processing rules or setting a seed for the randomness to get repeatable results. However, each of these approaches has its own trade-offs and doesn’t completely eliminate the issue.