ML and LLM Adoption Challenged Most Often by Observability

Observability and monitoring is the most cited challenge when moving ML models into production. The Institute for Ethical AI & Machine Learning conducted a survey on the state of production ML in the fourth quarter of 2024. The other key takeaway is that custom-built tools dominate user roadmaps, since few vendor tools have gained significant traction.

Overall, 44% of the 170 practitioners surveyed were machine learning engineers with about the same amount identifying as a data scientist or a MLOps engineer. Many of the respondents are subscribers to Alejandro Saucedo’s The ML Engineer newsletter.

Only 7% say that ML security is one of their top three challenges and only 17% say the same about governance and domain risks. That finding is significantly different from what we’ve seen in other studies, where security and AI governance are cited as among the biggest obstacles to increased adoption. We believe the practitioners view ML security as pertaining just to the ability of a model to be hacked, while other IT decision-makers worry more about general access to corporate and personal data.

It seems like every enterprise is at least experimenting with generative AI and AI agents that rely on large language models (LLMs). At the same time, the adoption of predictive analytics and computer vision continues to grow. As these applications scale up, developers require data engineers, SREs and others to handle Day 1 and Day 2 challenges. Rising to the challenge, MLOps became a real discipline, followed by LLMOps and GenAIOps.

Regardless of the terminology used, LLM observability and monitoring is something that has to be addressed.

Custom-Built vs. Vendor Tools

The survey asked about nine different parts the technology stack needed to utilize AI and machine learning. Here are some noteworthy findings:

• A managed model or LLM API service is used by 65% of the survey use. Among those that use this type of service, OpenAI (38%), AzureAI (20%) and Amazon Bedrock (12%) were used most often.
• MLflow is the leader for model registry and/or experiment tracking. Among those that have adopted these tools, 48% use MLflow most often. Custom built tools (16%) and Weights & Biases (12%) were the next most used tools in this category. Note that CoreWeave, which itself just had an IPO, recently announced its acquisition of Weights & Biases.
• Among users of ETL / workflow orchestrators, 40% use Airflow most often. Custom-built tools (17%) and Argo Workflows (11%) were the next most used tools in this category.
• Among users of real-time model serving, 46% use FastAPI/Flask Wrapper most often. Data scientists were more likely to use this tool (70%). Custom-built tools (16%) and AWS SageMaker (12%) were the next most used tools in this category.