Consistency of AI-Driven Usability Evaluations for Software: Learning Management Systems as an Example
Abstract
The evaluation of usability is a critical, yet resource-intensive, component of Human Systems Integration (HSI) for complex software systems like Learning Management Systems (LMSs). This study investigates the potential of using Large Language Model (LLM)-based Artificial Intelligence (AI) systems to conduct automated usability evaluations by analyzing textual user feedback. The author compared the consistency of usability assessments for 100 LMSs generated by three leading AI systems (Claude-Opus-5, GPT-5.6-Sol, and Gemini-3.5-Flash) against each other and against a baseline of human user aggregate scores. The AI systems scored seven usability attributes based on ISO/IEC 25010, 9241 Standards and usability models proposed in the literature so far: effectiveness, efficiency, error protection, accessibility, operability, learnability, and satisfaction. Principal Component Analysis (PCA) was used to derive a single usability factor score for each AI system’s evaluations. Subsequent correlation analyses (Pearson, Spearman, Kendall’s tau-b) revealed strong, statistically significant positive correlations between the AI-generated factor scores and the human benchmark scores, with correlation coefficients ranging from 0.459 to 0.626. Notably, the inter-AI correlations were exceptionally high, with values as strong as 0.850 between Claude-Opus-5 and Gemini-3.5-Flash. The findings suggest that LLMs possess a significant capability to perform human-like usability evaluations (or at least humans’ aggregate product rating) from text, supporting their potential as a powerful tool for rapid, scalable, and cost-effective usability analysis in HSI and systems engineering. However, the moderate strength of the correlations with human benchmark scores also indicates that current AI systems should be seen as complementary to, rather than a complete replacement for, traditional human-centered evaluation methods.
Keywords: Artificial Intelligence, Usability Evaluation, Learning Management Systems, Human Systems Integration, Principal Component Analysis, Text Analytics
DOI: 10.54941/ahfe1008375
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image


AHFE Open Access