Monitoring Machine Performance: Strategies For Success

Machine learning (ML) models are complex and probabilistic in nature, and monitoring their performance in production can be challenging due to the dynamic nature of real-world data and the technical debt inherent in ML systems. However, monitoring ML models is crucial to ensure consistent performance, maintain accuracy, and address data or concept drift. Functional monitoring, which focuses on data, model, and predictions, and operational monitoring, which focuses on system utilisation and cost, are the two primary methods for monitoring ML models.

Functional monitoring involves tracking the performance of the model in relation to its inputs and outputs. This includes monitoring data quality and integrity, such as checking for data processing issues, data schema changes, data loss at the source, and broken upstream models. It also involves detecting data and concept drift, where changes in the input data distribution or the relationship between input features and the target variable can lead to model degradation over time. Additionally, functional monitoring can include model and prediction monitoring, where the model's performance is evaluated against ground truth data or proxy metrics when ground truth data is not available.

Operational monitoring, on the other hand, involves tracking system performance metrics such as CPU/GPU utilisation, memory utilisation, number of failed requests, total number of API calls, and response time. It also includes monitoring the health of data and model pipelines, as well as tracking the cost of hosting the ML application and performing inference.

By implementing effective monitoring strategies, data scientists and ML engineers can gain insights into model behaviour, identify and address issues, and continuously improve the performance and accuracy of their ML models.

Monitor data quality and integrity

Monitoring data quality and integrity is a critical aspect of machine performance management. Here are some detailed instructions and best practices to ensure effective data quality and integrity monitoring:

Define Data Quality and Integrity Criteria

The first step is to establish clear definitions of data quality and integrity for your organisation. Data quality can be assessed based on dimensions such as accuracy, completeness, timeliness, consistency, validity, and relevance. On the other hand, data integrity dimensions include authenticity, availability, confidentiality, and non-repudiation. It is important to set objectives, targets, and thresholds that align with your specific business goals and requirements.

Implement Data Quality and Integrity Controls

Implement robust data quality and integrity controls throughout the data lifecycle, from creation to consumption. This includes establishing data governance policies, standards, and procedures, as well as data encryption, backup, and recovery processes. Ensure that data access, authorisation, and authentication mechanisms are in place, along with data audit trails, logs, and signatures. Document and communicate these controls to all relevant stakeholders for a cohesive approach.

Regularly monitor data quality and integrity performance using appropriate tools, methods, and indicators. Collect and analyse key metrics such as error rates, completeness ratios, timeliness scores, consistency indexes, validity rates, and relevance scores. Compare your performance against predefined objectives and identify any gaps, issues, or risks that may impact the overall machine performance.

Audit Data Quality and Integrity Compliance

Conduct periodic audits of your data quality and integrity compliance using independent and objective auditors, reviewers, or assessors. Verify that your data quality and integrity controls are effectively implemented and meet your predefined standards and expectations. Ensure that your data compliance adheres to any applicable laws, regulations, or contractual obligations. Document and report the findings, recommendations, and actions taken to address any compliance gaps.

Continuously Improve Data Quality and Integrity

Foster a culture of continuous improvement by regularly reviewing and updating your data quality and integrity criteria, objectives, targets, and thresholds. Implement corrective and preventive actions based on monitoring and audit results, evaluating their effectiveness and impact. Encourage a sense of ownership and accountability among your staff, partners, and customers regarding data quality and integrity.

Additional Considerations

• Data Profiling: Examine, analyse, and understand the content, structure, and relationships within your data. Identify patterns, anomalies, and inconsistencies to gain insights into potential quality issues.
• Data Auditing: Assess the accuracy and completeness of your data by comparing it against predefined rules and standards. Identify and track data quality issues such as missing, incorrect, or inconsistent data.
• Data Quality Rules: Establish predefined criteria that your data must meet to ensure accuracy, completeness, consistency, and reliability. Enforce these rules using data validation, transformation, or cleansing processes.
• Data Cleansing: Identify and correct any errors, inconsistencies, or inaccuracies in your data to maintain high-quality data for effective decision-making.
• Real-time Data Monitoring: Continuously track and analyse data as it is generated, processed, and stored within your organisation to address data quality issues promptly.
• Data Performance Testing: Evaluate the efficiency, effectiveness, and scalability of your data processing systems and infrastructure to ensure they can handle increasing data volumes and complexity without compromising data quality.
• Metadata Management: Organise, maintain, and utilise metadata to improve the quality, consistency, and usability of your data. Implement robust metadata management practices to enhance the overall quality of your data.

Identify data distribution changes

Identifying data distribution changes is crucial for monitoring machine performance. Here are some steps and strategies to help you achieve this:

Understand Data Distribution

Before identifying changes, it's important to grasp the concept of data distribution. Data distribution refers to the arrangement or pattern of values within a dataset. It involves understanding the central tendency, spread, and shape of the data. Common types of distributions include normal (bell-shaped), skewed (positively or negatively), uniform, bimodal (two peaks), and multimodal (multiple peaks). Visualisation tools such as histograms, density plots, box plots, and quantile-quantile (Q-Q) plots can aid in understanding data distribution.

Monitor Data Quality and Integrity

The quality and integrity of input data are vital for machine learning systems. Issues with data processing, such as pipeline problems resulting in missing or corrupted data, can have significant impacts. Monitoring data quality involves checking for data availability, data schema changes, data loss at the source, and broken upstream models. Feature processing issues, where the transformation of data into model features goes awry, should also be addressed.

Detect Data Drift

Data drift occurs when there are changes in the distribution of training data and production data. Monitoring data drift involves tracking feature-level changes using statistical measures such as mean, standard deviation, minimum and maximum values, and correlation. Techniques like Kullback-Leibler divergence, Kolmogorov-Smirnov statistics, and Population Stability Index (PSI) can be employed to detect drift. Monitoring at the feature level is crucial as it provides insights into model performance and behaviour.

Monitor Model Drift

Model drift, or concept drift, happens when the relationship between features and labels changes over time, leading to degraded model performance. This can occur due to natural changes in the business landscape or sudden events. Monitoring model drift involves tracking model performance metrics such as accuracy, AUC, precision, etc. Comparing model predictions with ground truth data, when available, is essential for evaluating model drift.

Implement Functional and Operational Monitoring

Functional monitoring focuses on the model's performance, inputs, and outputs. It involves tracking data quality issues, data/feature drift, model drift, and model configuration. Operational monitoring, on the other hand, deals with system-level metrics such as CPU/GPU utilisation, memory utilisation, response time, and system performance. Both types of monitoring are crucial for a comprehensive understanding of machine performance.

Establish Alerting Mechanisms

Setting up alerts is an integral part of monitoring. Alerts should be configured based on defined thresholds and sent to relevant stakeholders when issues arise. Alerts should be tested beforehand and include context and suggested actions for effective troubleshooting.

Choose a Monitoring/Observability Platform

Selecting an appropriate monitoring platform is essential. The platform should be easy to use, configurable, and offer out-of-the-box metrics and integrations. It should also provide customisation options, collaboration features, model explainability, and the ability to detect outliers and adversarial attacks. Examples of monitoring platforms include Prometheus + Grafana, Kibana and the ELK stack, and specialised ML observability platforms like Arize AI and Superwise.ai.

Best Practices

Some best practices for monitoring machine performance include focusing on people first and encouraging a culture of data ownership. Decentralising knowledge and tasks among cross-functional teams can also enhance efficiency. Additionally, it is important to monitor from the experimentation stage onwards, not just after deployment. Finally, ensure that logging is strategic and focuses on issues with consequential impacts.

Identify training-serving skew

Training-serving skew is a common problem in machine learning model deployment, which occurs when there is a difference in the model's performance during training and serving. It is caused by discrepancies in data handling between the training and serving pipelines, changes in data between training and serving, or a feedback loop between the model and algorithm.

Training-serving skew can lead to reduced model performance over time and is challenging to detect. It is important to address this issue as it can cause erratic behaviour in the model and induce logic discrepancies, requiring additional engineering efforts for debugging.

To avoid training-serving skew, engineers should aim to reuse the same feature engineering code during training and deployment to ensure that any given raw data input maps to the same feature vector. This can be challenging when there is a mismatch in computational resources at training and deployment, requiring multiple feature engineering codebases to maintain. In such cases, it is crucial to test for training-serving skew before deploying a new model by passing raw data through both pipelines and comparing the output. If the raw input vectors do not map to the same output feature vector, there is training-serving skew.

Training-serving skew is distinct from data drift, which assumes a change during the production of the model. In contrast, training-serving skew is a mismatch between the training and deployment stages, without any "drift" occurring.

Identify model or concept drift

Model or concept drift refers to the changes in the data patterns and relationships that a machine learning model has learned. It occurs when the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This can cause problems as the predictions become less accurate as time passes.

To detect model or concept drift, you can monitor model quality using metrics such as accuracy or mean error. When ground truth labels are unavailable, proxy metrics such as prediction drift, data drift, correlation changes, or domain-specific heuristics can be used.

Data drift refers specifically to changes in the input data distributions, while concept drift refers to changes in the relationship between model inputs and outputs. However, they often coincide. For example, in a spam detection model, shifts in email characteristics such as average length, languages used, or delivery time could signal a significant change in the environment or the emergence of a new attack strategy.

There are several types of model or concept drift:

• Gradual concept drift: This is the most frequent type of drift, where the underlying data patterns change over time. For example, user preferences evolve, or new movies are released, causing a model that predicts user preferences for movies to make less relevant suggestions.
• Sudden concept drift: This is an abrupt and unexpected change in the model environment, such as a new competitor entering the market and completely changing customer behaviour.
• Recurring concept drift: This refers to pattern changes that happen repeatedly or follow a cycle, such as sales increasing during holidays or Black Fridays.
• Temporary concept drift: This is difficult to detect using rule-based methods and is often detected using unsupervised methods. It occurs due to strange, one-off events such as adversarial attacks or system performance issues.

To detect model or concept drift, you can use various techniques:

• Statistical hypothesis testing: Compare the distribution of incoming data with a reference distribution (e.g. the first month of training data) using tests such as the Kolmogorov-Smirnov test or the Chi-Square test.
• Distance metrics: Quantify the difference between two probability distributions using metrics such as Wasserstein distance or Population Stability Index (PSI).
• Rule-based checks: Define simpler conditions without comparing distributions, such as setting a rule that triggers an alert when the share of emails predicted as spam falls below a certain threshold.

To address model or concept drift, you can:

• Retrain the model using the most recent data: This helps the model adapt to changing patterns, but it may be costly or require a major approval process.
• Adjust the decision-making threshold: Modify the decision thresholds for classification models to adjust the model's sensitivity to changes in the data distribution.
• Human-in-the-loop: Return to the "classic" decision-making process or manual review for critical models or unusual inputs.
• Alternative models: Consider heuristics or other model types, such as rule-based systems or ensemble techniques.
• Pause or stop the model: If the model quality is unsatisfactory, you may need to turn it off temporarily.

Identify health issues in pipelines

Identifying health issues in pipelines is crucial for maintaining optimal machine performance. Here are some strategies to identify and address these issues:

• Data Processing Issues: Ensure that the machine learning model receives complete and accurate data. Address any pipeline issues that may lead to missing, corrupted, or limited data. Monitor upstream systems and data sources to prevent disruptions.
• Data Schema Changes: Be vigilant about changes in data formats, types, and schemas. Communicate with data owners to stay informed about updates and their potential impact on the model. Establish data validation checks to catch schema changes and prevent errors.
• Data Loss at the Source: Monitor for data loss due to failures at the source, such as application clickstream data loss or sensor malfunctions. Detect and address irreversible data loss promptly to minimize the impact on retraining data.
• Broken Upstream Models: In complex setups with interconnected models, monitor for errors in upstream models that can propagate downstream. Establish feedback loops and error-handling mechanisms to identify and rectify issues in upstream models.
• Data Quality and Integrity: Implement data quality monitoring to catch issues before they affect the model's performance. Track data subsets consumed by the model and establish custom monitoring for feature processing code. Validate input and output data at each step of the pipeline to facilitate error detection and troubleshooting.
• Pipeline Bloat: Keep an eye on deals that have been stagnant in the pipeline for extended periods. Analyze the reasons for stagnation, such as low activity, stage-specific issues, or delayed close dates. Optimize the pipeline by addressing these issues and ensuring efficient progression of leads.
• Insufficient Pipeline Coverage: Ensure you have enough opportunities in the pipeline to meet your sales targets. Diversify lead generation strategies and prospecting approaches to maintain a healthy pipeline coverage ratio, typically 3-4x the sales target.
• Poor Pipeline Data Hygiene: Improve visibility into leading and lagging indicators of pipeline health. Establish clear criteria for a healthy pipeline, including factors such as deal quality, opportunity age, deal size, stage progression, and win rates. Assign scores and weights to these criteria and calculate a pipeline hygiene score to assess the overall health of your pipeline.
• Inflow/Outflow Report: Monitor the inflow and outflow of opportunities in your pipeline daily. Analyze the number of new opportunities created and the number of deals closed or pushed out to identify trends and potential issues.
• Pipeline Size and Balance: Evaluate the total number of leads and the dollar value of deals in your pipeline. Ensure you have sufficient prospects to meet your sales targets while considering the quality of your leads and their likelihood of conversion. Maintain a balanced distribution of leads across different sales stages, forming a funnel-like structure.
• Lead Velocity: Measure the speed at which leads move through the sales pipeline. Benchmark your lead velocity against industry standards or historical data to identify areas for improvement and ensure a smooth flow of deals.
• Lead Generation Rate: Track the rate at which new leads are added to the pipeline. Understand the most effective channels for lead generation, such as inbound marketing, outbound sales efforts, or partner referrals, and optimize your strategies accordingly.
• Pipeline Cleanliness: Maintain accurate, up-to-date, and easily accessible data in your pipeline. Establish a unified source of truth, such as Salesforce, to ensure consistency in data analysis across the organization. Regularly review and update your pipeline data to ensure transparency and straightforwardness in assessing your sales process.

Frequently asked questions