A Rank-Based Analysis of AI Consistency in Shortlisting Top Software Systems by Usability: Learning Management Systems as an Example

Open Access
Article
Conference Proceedings
Authors: Victor K Y Chan
Abstract

This study serves as a furtherance of a prior work, which demonstrated a moderate-to-strong correlational consistency between usability factor scores generated by Large Language Model (LLM)-based Artificial Intelligence (AI) systems and a human benchmark score from Capterra. While the prior work focused on parametric and non-parametric correlations, this paper investigates the consistency of the rankings produced by three AI systems (Claude-Opus-5, GPT-5.6-Sol, and Gemini-3.5-Flash) when shortlisting top-performing Learning Management Systems (LMSs) by usability as gauged by seven usability attributes based on ISO/IEC 25010, 9241 Standards and usability models proposed in the literature so far: effectiveness, efficiency, error protection, accessibility, operability, learnability, and satisfaction. Rank-Biased Overlap (RBO) was employed to compare the AI rankings against each other and against Capterra’s human benchmark score. The results show that all AI systems exhibit moderate-to-high ranking agreement with each other (RBO up to 0.878). However, the agreement between the AI rankings and the human benchmark score is substantially lower, diminishing from a moderate level at a shallow depth for top 10 LMSs (RBO ~ 0.7) to a rather weak level for the full list of 97 LMSs (RBO < 0.2). These findings indicate that while AI systems are highly consistent among themselves in ranking LMSs by their usability, their rankings somewhat diverge from the human benchmark as more LMSs are considered. This suggests that AI may be a viable tool for rapidly identifying a shortlist of potential top-usability LMSs but is not yet a reliable replacement for comprehensive human evaluation. Such findings on rankings offer more practicality than the prior work on correlations.

Keywords: Rank-Biased Overlap, Artificial Intelligence, Usability Evaluation, Learning Management Systems, Ranking Consistency

DOI: 10.54941/ahfe1008371

Cite this paper
Downloads
0
Visits
2
Download PDF

More from this volume

← The AXTENI Framework: Redefining Leadership Authority in the Age of Agentic AIDesigning with and Beyond Humans: An LLM-Based Human–AI Co-Creation Framework for Remote Curatorial Collaboration →
View all articles in Human Factors in Design, Engineering, and Computing →