Psychometric Characterization of Silent Reliability Degradation Detection in Automated Decision Aids

Abstract

Automated decision aids may undergo silent reliability degradation without explicit alerts, requiring operators to infer state changes from sparse and partly stochastic error cues. The current research conducts a secondary analysis of the raw first-report behavioural data from Tan et al. (2026) to psychometrically characterize silent degradation detection. First-report trajectories from a within-subject 3 x 2 x 2 experiment (reliability loss magnitude: 0%, 5%, 10%; initial reliability: 75% vs. 90%; error type: false alarms vs. misses; N = 60) were reconstructed into window-based cumulative outcomes at 20, 40, 60, 80, and 100 trials. A logistic mixed-effects model estimated window-specific psychometric functions mapping reliability loss to cumulative failure-report probability, from which operational detection thresholds were derived as the loss required to reach a 50% report criterion. Failure-report probability increased with reliability loss and monitoring window, and was lower under 90% initial reliability in the low-loss region. The 90% condition also showed steeper psychometric slopes. Operational detection thresholds declined with monitoring duration in both conditions, but remained higher under 90% initial reliability in the mid-to-late windows, with stable separation at 80 and 100 trials. These findings show that silent degradation detection is not only a matter of when the first report occurs, but also of how much degradation is required, within a given monitoring duration, to trigger a failure judgment.

Keywords: Automated Decision Aids, Silent Reliability Degradation, Psychometric Function, Operational Detection Threshold, Monitoring Duration

DOI: 10.54941/ahfe1007550