Model Performance & Drift Monitoring

Covariate Drift (Amount)

KS Test vs Training Data

N/A
P-Value
SAFE

Label Drift

Anomaly Rate Shift

0.0%
Current Anomaly Rate
STABLE

Concept Drift (DDM)

Model Error Rate Monitor

0.0%
Error Rate
STABLE
Model Precision Recall F1 Score Training F1 (Ref)
isolation forest 10.34% 26.58% 14.89% 95.00%
lof 7.58% 26.58% 11.80% 95.00%
one class svm 7.03% 27.85% 11.22% 95.00%
elliptic envelope 9.46% 26.58% 13.95% 95.00%
mlp 100.00% 5.06% 9.64% 95.00%

Metric Definitions

  • Precision: Accuracy of anomaly alerts.
  • Recall: Ability to catch actual fraud.
  • Drift: If > 2.0, the live data is significantly different from training data, suggesting models need retraining.

Drift Types & Detection Methods Reference

Drift Type Definition Detection Methods & Metrics Notes/Examples
Covariate Drift Change in feature/input distributions over time. KS test, PSI, JSD, Wasserstein Distance, Chi-square Typical in shifts of customer behavior or environment variables.
Prior Probability Shift (Label Drift) Change in label/class distribution without feature change. Monitor class proportions, Chi-square test Example: sudden increase in fraud cases changing label distribution.
Concept Drift Change in the relationship between inputs and labels. Monitoring error rates, DDM, EDDM, Page-Hinckley, ADWIN Model performance degradation signals this; can be gradual or abrupt.
Virtual Drift Change in input data distribution without label change. Similar to Covariate Drift (KS, PSI), feature importance monitoring Implies changes in data but not necessarily in the task or predictions.
Feature Drift Specific shift in particular features or groups of features. Segment-based drift analysis, adversarial validation Useful for identifying drifting subpopulations or cohorts.
Anomaly Drift Appearance of outlier patterns that were not present during training. Isolation Forests, One-Class SVM, clustering Detects novel or rare shifts that cause model uncertainty.
Partial Drift Drift affecting only some features or subclasses of data. Feature-wise drift tests, segment/subgroup analysis Requires granular inspection; tied to business segments or demographics.