When One in a Million Matters: Developing Metrics for Human-AI Collaboration in Rare Disease Diagnosis

Abstract

Rare diseases affect approximately 300 million people worldwide, yet physicians rarely encounter individual conditions, creating significant risk of delayed or missed diagnosis. AI diagnostic tools offer potential to reduce this uncertainty. However, metrics for assessing Human-AI Collaboration (HAIC) quality in this context remain underexplored — existing frameworks lack empirical operationalisation for the Human-Centric collaboration mode, where the physician retains full decision-making authority. This study aims to operationalise quality metrics for Human-Centric HAIC within rare neuromuscular disease diagnosis by exploring neurologists’ experiences with a conversational AI diagnostic assistant. An exploratory qualitative design employing thematic analysis is planned. Semi-structured interviews following a critical incident technique protocol will be conducted with 10–12 neurologists. Questions address ten collaboration quality dimensions: clarity of communication, ease of use, user satisfaction, feedback frequency, teaching efficiency, error reduction rate, task completion time, confidence, trust score, and safety incidents. Thematic analysis will identify context-specific subdimensions of each metric. The study will lay the groundwork for a domain-specific HAIC assessment instrument, provide design recommendations for clinical AI systems, and establish a basis for future psychometric validation and research on AI adoption in healthcare.

Keywords: Human-AI Collaboration Assessment, Human-centric Mode, Rare Disease Diagnosis, Qualitative Methodology, Collaboration Quality Metrics

DOI: 10.54941/ahfe1007466