Consistency of AI-Driven Usability Evaluations for Software: Learning Management Systems as an Example

Open Access
Article
Conference Proceedings
Authors: Victor K Y Chan
Abstract

The evaluation of usability is a critical, yet resource-intensive, component of Human Systems Integration (HSI) for complex software systems like Learning Management Systems (LMSs). This study investigates the potential of using Large Language Model (LLM)-based Artificial Intelligence (AI) systems to conduct automated usability evaluations by analyzing textual user feedback. The author compared the consistency of usability assessments for 100 LMSs generated by three leading AI systems (Claude-Opus-5, GPT-5.6-Sol, and Gemini-3.5-Flash) against each other and against a baseline of human user aggregate scores. The AI systems scored seven usability attributes based on ISO/IEC 25010, 9241 Standards and usability models proposed in the literature so far: effectiveness, efficiency, error protection, accessibility, operability, learnability, and satisfaction. Principal Component Analysis (PCA) was used to derive a single usability factor score for each AI system’s evaluations. Subsequent correlation analyses (Pearson, Spearman, Kendall’s tau-b) revealed strong, statistically significant positive correlations between the AI-generated factor scores and the human benchmark scores, with correlation coefficients ranging from 0.459 to 0.626. Notably, the inter-AI correlations were exceptionally high, with values as strong as 0.850 between Claude-Opus-5 and Gemini-3.5-Flash. The findings suggest that LLMs possess a significant capability to perform human-like usability evaluations (or at least humans’ aggregate product rating) from text, supporting their potential as a powerful tool for rapid, scalable, and cost-effective usability analysis in HSI and systems engineering. However, the moderate strength of the correlations with human benchmark scores also indicates that current AI systems should be seen as complementary to, rather than a complete replacement for, traditional human-centered evaluation methods.

Keywords: Artificial Intelligence, Usability Evaluation, Learning Management Systems, Human Systems Integration, Principal Component Analysis, Text Analytics

DOI: 10.54941/ahfe1008375

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← Qualifying NEV User Experience: LLM-Based Dimensional Encoding and Hierarchical Clustering of Online CommentsHuman–AI Collaboration in Supply Chains: A Framework for Resilient Decision Making Under Disruption →
View all articles in Human Factors in Design, Engineering, and Computing →