When the model card is the hazard: a silent preprocessing failure in a CT foundation model and a label-free check for trustworthy deployment
Abstract
Clinical teams increasingly build decision support on frozen foundation models (FMs), preparing inputs as the public model card shows. We report a case in which the documentation itself was the hazard. A linear probe on CT-FM embeddings was trained to predict COVID-19 outcome in 1,205 patients and applied unchanged to 309 patients at another hospital. Following the model card, external mortality AUC (area under the receiver operating characteristic curve) was 0.64; restoring the resampling used in pretraining, documented only in the training code, gave 0.71 (0.73 at 2 mm; Holm P = .10). In 319 examinations reconstructed both thin and thick, the model-card pipeline raised predicted risk by 1.76 standard deviations (SD) and scrambled patient ranking (Spearman 0.34): a thicker slice looked like a sicker patient. The corrected pipeline was stable (−0.02 SD; 0.96). The failure was reproduced by removing resampling from two other FMs on 1,110 CTs from a third cohort, and model-side repairs did not fully fix it. A Korean cohort (255 patients) analysed under a prospectively specified plan added kernel-driven instability and, post hoc, inflated predicted risk. We frame the omission as a latent condition that is invisible to clinicians and unequal across sites, and propose a three-step pre-deployment check that clinical teams can run: reproduce the pretraining preprocessing, test stability on paired reconstructions without labels, and recalibrate on about ten local cases.
Keywords: Trustworthy AI, Model Documentation, Foundation Models, Medical Imaging, Deployment Safety, Human Factors in Healthcare, Acquisition Bias
DOI: 10.54941/ahfe1008459
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image


AHFE Open Access