5 Steps to Deploy Efficient Cloud Native Foundation AI Models

The five steps to deploy cloud native sustainable foundation AI models starts with the obvious two: containers to manage the workloads and Kubernetes to deploy across a distributed infrastructure.

They may use PyTorch for programming and Jupyter Notebooks for debugging and evaluation, said Huamin Chen, who works in R&D at Red Hat’s Office of the CTO, in an interview recorded at the Open Source Summit North America in Vancouver earlier this Spring. Due to their facile deployment, Chen said Docker community files work well to containerize workloads for deployment.

With Kubernetes in deployment, the challenge is resource allocation and gaining efficiencies.

That may sound easy, but Chen asks, “How do you take care of the little difference when apportioning resources?”

The third step: measurement.

Chen cited Prometheus, the open source tool for event monitoring and alerting. Applications and infrastructure create metrics. With Prometheus, developers can correlate the workloads in foundation models and the runtime environments in a system, allowing for correlations and analysis.

Analytics represents the fourth step.

Chen said developers might use their analytics, but it’s helpful to have guidelines or some heuristics to build upon. To achieve bigger impacts, Chen said they place queries into Prometheus to get the basic metrics in place. They use that information to establish benchmarks, for example, to determine, for example, energy usage from foundation models, then correlate with performance metrics.

Energy usage relates to the performance of the model. It’s assumed that better performance comes with more energy expended.

“Your intuition may not always work,” Chen said. “And our discovery is that you can get the same performance without using more energy.”

The action taken from the analytics represents the fifth step. It culminates with applying the efficiencies and performance attained through the five stages.

“Number five is what we believe is most important for the community, for society, and for the environment,” he said. “Once you are able to optimize the energy profiles for our foundation models, then the more energy we can save, and the better environment we are going to have in the future.”