Data drift happens when the data leveraged for machine learning changes over time. This can be caused by different factors, such as ways the data is collected or changes in the underlying patterns of the data.

Data drift can lead to a deterioration in the performance of the model, as the model may no longer be able to accurately generalize to new data. To avoid data drift, it is key to keep an eye on the performance and feed the model updated data when necessary.

Why is data drift important?

Data drift is an important concept in machine learning because it can cause a trained model to become less accurate over time, leading to incorrect predictions and suboptimal decision-making. This is particularly problematic in scenarios where the model is deployed in real-world applications, where changes in the data can be frequent and unpredictable. The consequences of inaccurate predictions can range from missed business opportunities to safety risks in critical systems.

Detecting and addressing data drift is essential to ensure that machine learning models remain accurate and effective over time. Regularly monitoring the performance of a deployed model and retraining it with updated data as needed can help maintain its accuracy and ensure it continues to make reliable predictions. Data drift is an ongoing challenge in machine learning, and as such, researchers and practitioners are continually developing new techniques and tools to mitigate its effects. 

Methods of detecting data drifts

There are several ways to detect data drift in machine learning models.

  1. The Population Stability Index 

The Population Stability Index (PSI) is a statistical method used to detect data drift in machine learning models. PSI measures the distributional difference between two datasets by comparing their decile groups. It is calculated by dividing the data into deciles and comparing the percentage of observations in each decile between the reference (training) dataset and the new dataset. If the distribution of observations in any decile group changes by a significant amount, it suggests data drift.

PSI is a widely used method to monitor changes in the data distribution, and it has the advantage of being able to detect changes in the distribution of both continuous and categorical variables. PSI values can range from 0 to infinity, with higher values indicating a greater degree of data drift.

PSI is often used in conjunction with other techniques for detecting data drift, such as statistical methods, visualization, and drift detection algorithms, to provide a more comprehensive picture of changes in the data. By regularly monitoring the PSI of a model, data scientists can identify when data drift occurs and retrain the model with updated data to maintain its accuracy.

  1. The Kolmogorov-Smirnov Test

The Kolmogorov-Smirnov test is a statistical method used to detect data drift in machine learning models. It compares the cumulative distribution function (CDF) of the training data with that of the new data. If the CDF of the new data deviates significantly from the CDF of the training data, it suggests data drift. 

The Kolmogorov-Smirnov test is a widely used method for detecting changes in the data distribution, and it is effective for both continuous and categorical variables. It expects two datasets to be identical and tests their divergence.

  1. Kullback–Leibler Divergence

Kullback-Leibler divergence measures the difference between two distributions. In the context of machine learning, it can be used to detect data drift by measuring the divergence between the distribution of the training data and the distribution of the new data. 

KL divergence quantifies the amount of information lost when the new data distribution is used to approximate the training data distribution. If the KL divergence is high, it indicates a significant difference between the two distributions and suggests data drift. By regularly monitoring the KL divergence of a model, data scientists can identify when data drift occurs and retrain the model with updated data to maintain its accuracy.

  1. The Page-Hinkley Method 

Page-Hinkley (PH) is a statistical technique used to detect data drift in machine learning models. It works by monitoring the difference between the cumulative sum of errors made by the model on the training data and the cumulative sum of errors made by the model on the new data. In case it exceeds a certain limit, data drift has occurred.

To use the PH method to detect data drift, follow these steps:

  1. Train the machine learning model on the training data and use it to make predictions on the new data.
  2. Calculate the cumulative sum of errors made by the model on the training data and the new data. This can be done using a rolling window approach, where the cumulative sum is calculated over a fixed number of observations.
  3. Calculate the difference between the cumulative sums of errors for the training data and the new data at each time step.
  4. Calculate the PH statistic at each time step using the formula: PH_t = max(0, PH_t-1 + (d_t - λ - μ)), where d_t is the difference between the cumulative sums of errors at time t, λ is a constant representing the sensitivity of the test, and μ is the average difference between the cumulative sums of errors over a window of previous time steps.
  5. If the PH statistic exceeds a predetermined threshold, it indicates that data drift has occurred.

Cloud-base data drift detection 

Detecting data drift in the cloud can be done using various tools and services provided by cloud-based machine learning platforms. Here are some examples:

  1. AWS SageMaker Clarify: It is a service that helps detect bias and data drift in machine learning models. It provides a set of tools for monitoring data and model performance, including statistical analysis and visualization of drift metrics.
  2. Amazon CloudWatch: It is a monitoring and logging service that can be used to track metrics and detect changes in datasets. It can monitor data stored in S3 and generate alerts when changes are detected.
  3. Azure Machine Learning: It is a cloud-based service that provides tools for training, deploying, and monitoring machine learning models. It includes tools for detecting data drift, such as built-in data drift detection modules and support for custom drift detection algorithms.

To monitor datasets, data scientists can set up monitoring tools and services to track key metrics and detect changes over time. These tools can include statistical methods, visualization tools, and drift detection algorithms. 

Author:

Gilad David Maayan, a technology writer for SAP, Imperva, Samsung NEXT, NetApp and Check Point, the head of Agile SEO marketing agency.

LinkedIn: https://www.linkedin.com/in/giladdavidmaayan/