When the model card is the hazard: a silent preprocessing failure in a CT foundation model and a label-free check for trustworthy deployment

Open Access
Article
Conference Proceedings
Authors: Woo Hyun Shim, Chae Ri Park, Sang Cheol Kim, Sukyoung An
Abstract

Clinical teams increasingly build decision support on frozen foundation models (FMs), preparing inputs as the public model card shows. We report a case in which the documentation itself was the hazard. A linear probe on CT-FM embeddings was trained to predict COVID-19 outcome in 1,205 patients and applied unchanged to 309 patients at another hospital. Following the model card, external mortality AUC (area under the receiver operating characteristic curve) was 0.64; restoring the resampling used in pretraining, documented only in the training code, gave 0.71 (0.73 at 2 mm; Holm P = .10). In 319 examinations reconstructed both thin and thick, the model-card pipeline raised predicted risk by 1.76 standard deviations (SD) and scrambled patient ranking (Spearman 0.34): a thicker slice looked like a sicker patient. The corrected pipeline was stable (−0.02 SD; 0.96). The failure was reproduced by removing resampling from two other FMs on 1,110 CTs from a third cohort, and model-side repairs did not fully fix it. A Korean cohort (255 patients) analysed under a prospectively specified plan added kernel-driven instability and, post hoc, inflated predicted risk. We frame the omission as a latent condition that is invisible to clinicians and unequal across sites, and propose a three-step pre-deployment check that clinical teams can run: reproduce the pretraining preprocessing, test stability on paired reconstructions without labels, and recalibrate on about ten local cases.

Keywords: Trustworthy AI, Model Documentation, Foundation Models, Medical Imaging, Deployment Safety, Human Factors in Healthcare, Acquisition Bias

DOI: 10.54941/ahfe1008459

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← Development and Evaluation of Applications for Oral and Cognitive Rehabilitation: A Facial Rock–Paper–Scissors Program with Psychological EffectsSystem Safety Analysis Considering Human Factors in Aircraft Development →
View all articles in Human Factors in Design, Engineering, and Computing →