A Rank-Based Analysis of AI Consistency in Shortlisting Top Software Systems by Usability: Learning Management Systems as an Example
Abstract
This study serves as a furtherance of a prior work, which demonstrated a moderate-to-strong correlational consistency between usability factor scores generated by Large Language Model (LLM)-based Artificial Intelligence (AI) systems and a human benchmark score from Capterra. While the prior work focused on parametric and non-parametric correlations, this paper investigates the consistency of the rankings produced by three AI systems (Claude-Opus-5, GPT-5.6-Sol, and Gemini-3.5-Flash) when shortlisting top-performing Learning Management Systems (LMSs) by usability as gauged by seven usability attributes based on ISO/IEC 25010, 9241 Standards and usability models proposed in the literature so far: effectiveness, efficiency, error protection, accessibility, operability, learnability, and satisfaction. Rank-Biased Overlap (RBO) was employed to compare the AI rankings against each other and against Capterra’s human benchmark score. The results show that all AI systems exhibit moderate-to-high ranking agreement with each other (RBO up to 0.878). However, the agreement between the AI rankings and the human benchmark score is substantially lower, diminishing from a moderate level at a shallow depth for top 10 LMSs (RBO ~ 0.7) to a rather weak level for the full list of 97 LMSs (RBO < 0.2). These findings indicate that while AI systems are highly consistent among themselves in ranking LMSs by their usability, their rankings somewhat diverge from the human benchmark as more LMSs are considered. This suggests that AI may be a viable tool for rapidly identifying a shortlist of potential top-usability LMSs but is not yet a reliable replacement for comprehensive human evaluation. Such findings on rankings offer more practicality than the prior work on correlations.
Keywords: Rank-Biased Overlap, Artificial Intelligence, Usability Evaluation, Learning Management Systems, Ranking Consistency
DOI: 10.54941/ahfe1008371
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image


AHFE Open Access