A Guide to Model Composition

Consider an AI-powered image recognition app designed to identify and classify wildlife photos. You upload a picture taken during a hike, and within moments, the app not only identifies the animal in the photo but also provides detailed information about its species, habitat and conservation status. This kind of app can be built through model composition — a technique where multiple AI models collaborate to analyze and interpret the image from various perspectives.

Model composition in this context might involve a sequence of specialized models: one for detecting the animal in the image, another for classifying it into broad categories (e.g., bird, mammal and reptile) and yet another set of models that work together to determine the specific species. This layered approach offers a nuanced analysis that exceeds the capabilities of a single AI model.

What Is Model Composition?

At its core, model composition is a strategy in machine learning that combines multiple models to solve a complex problem that cannot be easily addressed by a single model. This approach leverages the strengths of each individual model, providing more nuanced analyses and improved accuracy. Model composition can be seen as assembling a team of experts, where each member brings specialized knowledge and skills to the table, working together to achieve a common goal.

Many real-world problems are too complicated for a one-size-fits-all model. By orchestrating multiple models, each trained to handle specific aspects of a problem or data type, we can create a more comprehensive and effective solution.

There are several ways to implement model composition, including but not limited to:

• Sequential processing: Models are arranged in a pipeline, where the output of one model serves as the input for the next. This is often used in tasks like data preprocessing, feature extraction, and then classification or prediction.
• Parallel processing: Multiple models run in parallel, each processing the same input independently. Their outputs are then combined, either by averaging, voting or through a more complex aggregation model, to produce a final result. This is commonly used in ensemble methods.

An important concept related to model composition is the inference graph. An inference graph visually represents the flow of data through various models and processing steps in a model composition system. It outlines how models are connected, the dependencies between them and how data transforms and flows from input to final prediction. The graphical representation helps us design, implement and understand complex model composition. Here is an inference graph example:

👁 Image

1. The service accepts a text input, such as “I have an idea!”
2. The service simultaneously sends the prompt to three separate text generation models, which run in parallel to produce results using different algorithms or datasets.
3. The results from these three models are then sent to a text classification model.
4. The classification model assesses each piece of generated text and assigns a classification score to them (for example, based on the content’s sentiment).
5. Finally, the service aggregates the generated text along with their respective classification scores and returns them as JSON.

When Should I Compose Models?

Model composition is a practical solution to a wide array of challenges in machine learning. Here are some key use cases where model composition plays a crucial role.

Multimodal Applications

In today’s digital world, data comes in various forms: text, images, audio and more. A multimodal application combines models specialized in processing different types of data. A typical example of composing models to create multimodal applications is BLIP-2, which is designed for tasks that involve both text and images.

BLIP2 integrates three distinct models, each providing a unique capability to the system:

• A frozen large language model (LLM): Provides strong language generation and zero-shot transfer abilities.
• A frozen pre-trained image encoder: Extracts and encodes visual information from images.
• A lightweight Querying Transformer model (Q-Former): Bridges the modality gap between the LLM and the image encoder. It integrates visual information from the encoder with the LLM, focusing on the most relevant visual details for generating text.

Ensemble Modeling

Ensemble modeling is a technique used to improve the prediction of machine learning models. It does so by combining the predictions from multiple models to produce a single, more accurate result. The core idea is that by aggregating the predictions of several models, you can often achieve better performance than any single model could on its own. The models in an ensemble may be of the same type (e.g., all decision trees) or different types (e.g., a combination of neural networks, decision trees and logistic regression models). Key techniques in ensemble modeling include:

• Bagging: Train multiple models on different subsets of the training data and then average their predictions. This is useful for reducing variance.
• Boosting: Sequentially train models, where each model attempts to correct errors made by the previous ones.
• Stacking: Train multiple models and then use a better model that leverages the strengths of each base model to improve overall performance and combine their predictions.

A real-world use case of ensemble modeling is a weather forecasting system, where accuracy is important for planning and safety across industries and activities. An ensemble model for weather prediction might integrate outputs from various models, each trained on different data sets, using different algorithms, or focusing on different aspects of weather phenomena. Some models may be more capable of predicting precipitation, while others perform better at forecasting temperature or wind speed. By aggregating these predictions, an ensemble approach can provide a more accurate and nuanced forecast.

Pipeline Processing

Machine learning tasks often require a sequence of processing steps to transform raw data into actionable insights. Implementing model composition can help you structure these tasks as pipelines, where each step is handled by a different model optimized for a specific function.

One of the common use cases is an automated document analysis system, capable of processing, understanding and extracting meaningful information from documents. The system might use a series of models, each dedicated to a phase in the processing pipeline:

• Preprocessing: The first step might require an OCR (Optical Character Recognition) model that extracts text from scanned documents or images. This model is specialized in recognizing and converting varied fonts and handwriting styles into machine-readable text.
• Prediction: Following text extraction, a text classification model can be used to categorize the document based on its content, such as a legal document, a technical manual, or a financial report. This classification step is important for routing the document to appropriate downstream processes.
• Postprocessing: After classification, a summarization model can be used to generate a concise summary of the document’s content. This summary provides quick insights into the document, informing decision-making and prioritization.

In addition to sequential pipelines, you can also implement parallel processing for multiple models to run concurrently on the same data (as shown in the first image). This is useful in scenarios like:

• Ensemble modeling: Predictions from multiple models are aggregated to improve accuracy.
• Computer vision tasks: Models for image segmentation and object detection may run in parallel to provide a comprehensive analysis of an image, combining insights into the image’s structure with identification of specific objects.

What Are the Benefits of Model Composition?

Model composition provides a number of operational and developmental advantages. Here are some key benefits:

Improved Accuracy and Performance

In some cases, the synergy of multiple models working together can result in improved accuracy and performance. Each model in the composition may focus on a specific aspect of the problem, such as different data types or particular features of the data, ensuring that the combined system covers more of the entire problem space than any single model could. This is especially true in ensemble modeling, as aggregating the results from multiple models can help cancel out their individual biases and errors, leading to more accurate predictions.

Dedicated Infrastructure and Resource Allocation

Model composition allows you to deploy the involved models across varied hardware devices, optimizing the use of computational resources. They can be assigned to run on the most appropriate infrastructure — whether it’s CPU, GPU or edge devices — based on their processing needs and the availability of resources. This dedicated allocation also ensures that each part of the system can be scaled separately.

Customization and Flexibility

One of the most significant advantages of model composition is the flexibility it offers. Models can be easily added, removed or replaced within the system, allowing developers to adapt and evolve their applications as new technologies emerge or as the requirements change. This modular approach simplifies updates and maintenance, ensuring that the system can quickly adapt to new challenges and opportunities.

Faster Development and Iteration

Model composition supports a parallel development workflow, allowing teams to work on different models or components of the system simultaneously. This helps accelerate the development process, which means quicker iterations and more rapid prototyping. It also enables teams to provide more agile responses to feedback and changing requirements, as individual models can be refined or replaced without disrupting the entire system.

Resource Optimization

By intelligently distributing workloads across multiple models, each optimized for specific tasks or hardware, you can maximize resource utilization. This optimization can lead to more efficient processing, reduced latency and lower operational costs, particularly in complex applications that require substantial computational power. Effective resource optimization also means that your application can scale more gracefully, accommodating increases in data volume or user demand.

Composing Multiple Models With BentoML

Different model-serving or model-deployment frameworks may adopt different approaches to model composition. In this connection, BentoML, an open source model-serving framework, provides simple service APIs to help you wrap models, establish interservice communication and expose the composed models as REST API endpoints.

The code example below demonstrates how to use BentoML to compose multiple models. In BentoML, each Service is defined as a Python class. You use the `@bentoml.service` decorator to mark it as a Service and allocate CPU or GPU resources to it. When you deploy it to BentoCloud, different Services can run on dedicated instance types and can be separately scaled.

👁 Image

In this BentoML `service.py` file, GPT2 and DistilGPT2 are initialized as separate BentoML Services to generate text. The BertBaseUncased Service then takes the generated text and classifies it, providing a score that represents sentiment. The InferenceGraph Service orchestrates these individual Services, using `asyncio.gather` to concurrently generate text from both GPT-2 models and then classifying the output using the BERT model.

After they are deployed to BentoCloud, Services can run on separate instance types, as shown below:

👁 Image

Monitor the performance:

👁 Image

For detailed explanations, see this example project.

Frequently Asked Questions

Before I wrap up, let’s see some frequently asked questions about model composition.

What is the difference between ensemble modeling and multimodal applications?

These two machine-learning concepts serve different purposes and are applied in different contexts.

• Purpose and application: Ensemble modeling improves prediction accuracy by combining multiple models. Multimodal applications integrate and interpret data from multiple sources or types to make better decisions or predictions.
• Models vs data: Ensemble modeling focuses on using multiple models to enhance predictions. Multimodal applications focus on integrating different types of data (e.g., text, image, audio).
• Implementation: Multimodal systems often require data preprocessing and feature extraction techniques to handle different data types effectively. Ensemble modeling, on the other hand, needs strategies for combining model predictions, which might involve direct averaging or more complicated voting systems.

I am using a single model for my application. Should I move to multiple models?

It’s important to note that while model composition offers different benefits as mentioned above, it’s not always necessary. If a single model can efficiently and accurately accomplish the task at hand, I recommend you just stick with it. The decision to compose multiple models and the design of the processing pipeline should be guided by your specific requirements.

How does model composition affect production deployment?

The integration of multiple models into a single application affects production deployment in several key ways:

Increased Complexity

• Configuration and management: Each model in the composition may require its configuration, dependencies and environment. Managing them across multiple models adds complexity to the deployment process.
• Service orchestration: Composing multiple models often requires careful orchestration to ensure that data flows correctly between models, and that each model is executed in the correct order or in parallel as required.

Resource Allocation

• Hardware requirements: As mentioned above, different models may have different hardware requirements. Some models may need GPUs for inference, while others can run on CPUs. The serving and deployment framework you select should support flexible resource allocation to meet your needs.
• Scaling strategies: Scaling multiple models in production may not be as straightforward as scaling a single model. Different components of the application may have varying loads, requiring dynamic scaling strategies that can adjust resources for individual models based on demand.

Monitoring and Maintenance

• Monitoring: Keeping track of the performance and health of different models in production requires comprehensive monitoring solutions that can provide insights into each model’s performance, resource usage and potential bottlenecks.
• Versioning and updates: Updating one model in a composite application can have cascading effects on other models. Proper version control and testing strategies must be in place to manage updates without disrupting the application’s overall performance.

Deployment Strategies

• Microservices architecture: Adopting a microservices architecture can simplify the deployment of multiple models by encapsulating each model as a separate service. This approach simplifies scaling, updates and management, but requires flexible service orchestration tools.
• Containerization: Using containers for deploying AI models can help manage dependencies and environments for each model. Container orchestration tools like Kubernetes can help manage the deployment, scaling and networking of containerized models.

Model composition can affect deployment by requiring more resources and potentially more complex deployment strategies. However, as shown in the example above, platforms like BentoML and BentoCloud can help developers build AI applications of multiple models by allowing them to package, deploy and scale multimodel services efficiently.

Final Thoughts

While the benefits of model composition are clear, from enhanced performance to the ability to process multiple data types, it’s important to acknowledge the complexity it introduces, especially related to production deployment. Successful implementation requires careful planning, resource management and the adoption of modern deployment practices and tools to navigate the challenges of configuration, scaling and maintenance.