Cloud Native Ecosystem Containers Databases Edge Computing Infrastructure as Code Linux Microservices Open Source Networking Storage

AI AI Engineering API Management Backend development Data Frontend Development Large Language Models Security Software Development WebAssembly

AI Operations CI/CD Cloud Services DevOps Kubernetes Observability Operations Platform Engineering

C++ Developer tools Go Java JavaScript Programming Languages Python Rust TypeScript

As organizations increasingly look to implement AI solutions without the computational burden, inference AI as a service is emerging as a go-to option, offering the power of real-time decision-making without the need for in-house hardware and expertise. By outsourcing the inferencing workload to specialized cloud services, companies can save on the costs associated with building and maintaining on-premises infrastructure and simultaneously benefit from the latest advancements in AI technology, ensuring optimal performance — and a leg up on the competition. Does that mean you should plunge right into outsourcing the computational burden that comes with managing the deployment of an AI inference model? How do you ensure that managed inference AI service is the right choice for you? You’ll often find cloud service providers talking about numerous benefits and advanced features of a managed AI solution, however, they seldom offer you a clarity about what managing an AI production pipeline actually entails. Let’s understand what the journey from model training to deployment actually looks like, how to traverse this route and how Gcore deploys an inference AI model.

AI Model Training and Inference

In the world of AI, there are two key operations: training and inference. Regular AI encompasses both of these tasks, learning from data and then making predictions or decisions based on that data. By contrast, inference AI focuses solely on the inference phase. After a model has been trained on a dataset, inference AI takes over to apply this model to new data to make immediate decisions or predictions. This specialization makes inference AI invaluable in time-sensitive applications, such as autonomous vehicles and real-time fraud detection, where making quick and accurate decisions is crucial. For self-driving cars, this service can swiftly analyze sensor data to make immediate driving decisions, eliminating latency and increasing safety. In real-time fraud detection, inference AI can instantaneously evaluate transactional data against historical patterns to flag or block suspicious activities.

The Need for Streamlined AI Production Management

Managing AI production involves navigating a complex matrix of interconnected decisions and adjustments. From data center location to financial budgeting, each decision carries a ripple effect. In my experience as head of cloud operations at Gcore, I can say that this field is still defining its rules; the road from model training to deployment is more of a maze than a straight path. In this section, I’ll review the key components that every AI production manager must carefully consider to optimize performance and efficiency. Location and latency should be your first consideration in AI production. Choose the wrong data center location, and you’re setting yourself up for latency issues that can seriously degrade user experience. For example, if you’re operating in the EU but your data center is in the United States, the transatlantic data travel times can create noticeable delays — a non-starter for inference AI. Resource management demands real-time adaptability. Elements like CPUs, memory and specialized hardware — GPUs or TPUs — require constant tuning based on up-to-the-minute performance metrics. As you switch from development to full-scale production, dynamic resource management becomes not a luxury but a necessity, operating on a 24/7 cycle. Financial planning is tightly linked to operational efficiency. Accurate budget forecasts are crucial for long-term sustainability, particularly given the volatility of computational demands in response to user activity. Unlike the more mature landscape of software development, AI production management lacks a standardized playbook. This means you need to rely on bespoke expertise and be prepared for a higher error rate. It’s a field propelled by rapid innovation, and trial and error. In this sense, the sector is still in its adolescent phase — reckless, exciting and still figuring out its standards.

How to Deploy an Inference AI Model

Now that we understand the key components of AI production management, let me walk you through a step-by-step guide for deploying an AI inference model, focusing on the integration of various tools and resources. The aim is to build an environment that ensures swift, efficient deployment and scaling. Here are some tools that will be essential for success:

1. Docker: An industry standard for containerization, aiding in the smooth deployment of your model.
2. Whisper: A leading AI model for speech-to-text that serves as the foundation of our service.
3. Simple Server Framework (SSF): This Graphcore tool facilitates the building and packaging (containerizing) of applications for serving.
4. Harbor: An open source artifact storage software used for preserving Docker images, instrumental in our setup. Use the official docs to get set up.

Here’s what the pipeline looks like: 👁 AI inference pipeline

Preparation

• Model: For this guide, we use a pre-trained model from Hugging Face. Training the model is outside the scope of this article.
• Environment: We have a designated cluster for model building. All commands will be executed via SSH.

Step 1: Set up a Virtual Environment

Create a virtual environment: Activate it:

 Step 2: Install Required Packages

Install SSF: Install additional plugins for Docker:

Step 3: Codebase

Clone the Gcore repository that contains all the necessary files: Change the branch: Two key files here are `ssf_config.yaml` and `whisper_ssf_app.py`. `ssf_config.yaml` is crucial for configuring the package that you’ll build. It contains fields specifying the name of the model, license and dependencies. It also outlines the inputs and outputs, detailing the endpoints and types of fields. For instance, for the Whisper model, the input is a temporary file (TempFile) and the output is a string (String). This information sets the framework for how your model will interact with users. Example for Whisper: SSF provides support for various data types. Detailed information can be found in its documentation. `whisper_ssf_app.py` acts as a wrapper around your Whisper model, making it compatible with the Simple Server Framework (SSF). The script contains several essential methods:

• `build`: This is where the model’s computational graph is constructed. It must run on a host with an IPU.
• `startup`: Manages preliminary tasks before the model can begin serving user requests.
• `request`: This is the heart of the system, responsible for processing user requests.
• `shutdown`: Ensures graceful termination of the model, like completing ongoing requests.
• `is_healthy`: This method allows the model to function both as a standalone Docker container and as part of larger, more complex systems like Kubernetes.

Within the build method, the function `compile_or_load_model_exe` is invoked. This is pivotal when constructing a model’s computational graph on IPUs. Here’s the catch: Creating this graph requires an initial user request as input. While you could use the first real user request for this, keep in mind that graph-building could consume 1 to 2 minutes, possibly more. Given today’s user expectations for speed, this delay could be a deal-breaker. To navigate this, the build method is designed to accept our predefined data as the first request for constructing the graph. In this setup, we use `bootstrap.mp3` to mimic that inaugural request.

Step 4: Building and Publishing the Container

Build and publish the container, specifying your own Docker registry address and credentials: The resulting container holds all necessary components: the model, a FastAPI wrapper, and the bootstrap.mp3 for initial warmup. It will be pushed to the Harbor registry.

👁 Image

Step 5: Deploying to Edge Node

For deployment on the edge node, the following command is used: `gc-ssf` deploy uses SSH to run commands on the target host, so you’ll need to access it using `ssh-key` between nodes. By following this pipeline, you establish a robust framework for deploying your AI models, ensuring they are not just efficient but also easily scalable and maintainable.

Inferring a More Intelligent Future

Inference AI’s growing role isn’t limited to tech giants; it’s vital for any organization aiming for agility and competitiveness. Investment in this technology constitutes a strategic alignment with a scalable, evolving solution to the data deluge problem. Inference AI as a service is poised to become an indispensable business tool because it simplifies AI’s technical complexities, offering a scalable and streamlined way to sift through mountains of data and extract meaningful, actionable insights.

How Gcore Uses Inference AI

Despite the surge in AI adoption, we recognize there’s still a gap in the market for specialized, out-of-the-box AI clusters that combine power with ease of deployment. Gcore is engineered to provide infrastructure and low latency services in order to go global faster. This solves one of the most significant challenges in the machine learning landscape: the transition from model development to scalable deployment. We use Graphcore’s Simple Server Framework to create an environment that’s capable not only of running machine learning models, but also of improving them continuously through Inference AI. For a deeper understanding of how Gcore is shaping the AI ecosystem, consider a visit to our blog and AI infrastructure documentation.

The New Stack does not sell your information or share it with unaffiliated third parties. By continuing, you agree to our Terms of Use and Privacy Policy.