Model Monitoring on Azure - 6 Weeks PoC

M-AInA, a unified AI/ML model monitoring template that provides a view of monitoring metrics of deployed model on Azure platform.

M-AInA, provides a new way to monitor your AI/ML models along with providing insights around drift management, ground truth evaluation and model explanations along with what-if analysis. The outcome being full visibility on model performance, easing the process of managing any misbehavior in the models.

M-AInA provides easy interoperability with the Azure cloud services. It provides in-built functionality with pre-defined KPIs to help monitor the model performance and drift using key Azure cloud components like Azure monitor, Azure ML.

The key axes which help provide a frictionless view of AI/ML models with M-AInA are: Model Drift: Helps evaluate the model performance over a period and a ground truth comparison of the models.

Explainable AI: Insights covering the model explanations and importance of features for a particular model, to help decipher model insights for business teams.

Data Drift: Provides a view of data comparison across multiple versions and features of the models contributing to the drift.

Service Health: Offers a detailed view across key performance metrics and includes automated triggers and alerts (for ex. Model retraining).

The framework today covers drift management in structured data and classification, Regression and Forecasting models. Within the scope of PoC a model monitoring setup would be done on a selected model registered in Azure ML.

Note: The Scope of PoC will include setting up of model monitoring framework and end to end monitoring for one model. Model drift requires availability of ground truth.

To help exemplify the outcome of M-AInA, our 6-week POC engagement plan covers monitoring setup of one model:

Week 1:

1. Understanding current Azure Machine learning environment along with reviewing the types of models which are in production and selecting the model which will be part of POC.
2. Setting up Metadata stores, covering key aspects like database and schema for the selected model.

Week 2 and 3:

1. Setting up and configuring M-AInA, along with performing Azure Authentication, database authentication, Azure ML authentication and integrated it with app insights.
2. Modify existing pipelines to log necessary parameters to metadata store by leveraging Azure monitor, python libraries, and evidently AI for setting up the foundation of M-AInA.
3. Modifying/Updating KPIs for the selected model to showcase the model performance.

Week 4, 5 and 6:

1. Setting up ground truth evaluation mechanism for selected models
2. Configuring M-AInA for the customer environment and deployment on Azure Cloud, develop dashboards to showcase the centralized model monitoring across different parameters.

The Outcome:

• Ease the efforts to evaluate the performance of models over time with a ground truth performance check.
• Manage various data and feature elements on-the-go, leveraging an in-built mechanism.
• Service view of the models in production to give a snapshot over infrastructure requirements.