Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees

Authors: Lei Yang

Submitted: 4 July 2026

Subjects: Machine Learning (cs.LG)

arXiv ID: 2607.21623

DOI: https://doi.org/10.48550/arXiv.2607.21623

Abstract

We present EaaS, a cloud-native reference architecture that operationalizes AI evaluation methods as six stateless Kubernetes microservices. The platform enables scalable, real-time monitoring of AI systems with rigorous statistical guarantees. The key components are:

1. Conformal Prediction – using finite-sample-corrected Adaptive Prediction Sets for uncertainty quantification
2. Calibration Assessment – continuous evaluation of model calibration
3. Drift Detection – via Random Fourier Features-approximated Maximum Mean Discrepancy (RFF-MMD)
4. Fairness Monitoring – with bootstrap confidence intervals for demographic parity
5. DAG-based Pipeline Orchestrator – for flexible composition of evaluation workflows
6. Result Storage API – for persistent storage and querying of evaluation results



Key Findings




Conformal Coverage

Empirical coverage is consistent with the marginal conformal guarantee across K=50 random calibration/test splits, with mean coverage within 1.4 percentage points of the nominal target.




Logprob Imputation Robustness

All four MMLU answer tokens appear in the top-20 logprobs with 0% imputation needed. Simulated imputation at 10% produces less than 1.5% coverage impact.




Drift Detection Performance

RFF-MMD achieves 100% detection power for mild and severe drift at the median heuristic bandwidth, with Type I error between 5–8.5%.




Fairness Monitoring

Analysis of the UCI Adult Income dataset reveals significant demographic parity disparities by race (DP gap = 0.33), with stable alerts across sequential batches.




Latency Benchmarks

• Conformal prediction and calibration services achieve sub-2ms p99 latency at batch size 100.
• RFF-MMD requires ~500ms, making it suitable for periodic batch monitoring.



Comparison with Existing Tools




A comparison with four open-source tools suggests that, to the best of our knowledge, no current platform combines conformal-prediction-as-a-service, microservice decomposition, and DAG-based orchestration.




Technical Details



• Format: 23 pages, 15 figures, 12 tables
• Primary Subject: Machine Learning (cs.LG)
• Comments: Includes full experimental setup and reproducibility guidelines



Significance for 2026




As AI systems become increasingly embedded in production environments, the need for continuous, scalable, and statistically rigorous monitoring grows. EaaS addresses this gap by providing a modular, cloud-native platform that can be deployed on Kubernetes for organizations requiring trustworthy AI evaluation at scale. The architecture supports both real-time inference monitoring and periodic batch evaluation, making it suitable for a wide range of deployment scenarios.

via ArXiv LG