A Comprehensive and Quantitative framework of User Experience Evaluation in GenAI software

Abstract: Generative AI (GenAI) is transforming the software market by introducing innovative yet complex intelligent experiences across various applications. However, traditional user experience (UX) evaluation methods, such as SUS, UEQ, and CSAT, are inadequate for capturing key aspects of these AI-driven experiences including output diversity and relevance. Relying solely on user feedback also overlooks broader commercial objectives. To address these challenges, we propose a structured evaluation framework that balances user experience and business goals. This paper: a) defines four core metrics for AI-driven experiences—Functionality, Ease of Use, Intent Understanding, and Generation Quality—further broken down into 27 influential factors; b) establishes a quantitative approach that combines product decision-makers’ weighted metrics with user satisfaction ratings to create a comprehensive satisfaction scoring model. Empirical validation with six GenAI software products and 30 user surveys confirms that when weight data meets consistency validation (CR < 0.1), prioritizing high-weight, low-satisfaction metrics enables precise UX issue identification and targeted enhancements. This approach resulted in notable improvements in user satisfaction and NPS, showcasing the practical value of aligning weighted metrics with user feedback for effective product optimization.Our primary contribution is a measurement framework for evaluating GenAI software. It's designed to overcome the limitations of traditional metrics while aligning user experience with business strategy, providing actionable insights for product iteration. This framework is currently being tested across various domains. We will present its definitions, evaluation approach, metrics, and results in poster sessions to foster cross-industry discussions on GenAI software UX evaluation.

Keywords: User Experience, UX Evaluation, UX Metric, User Satisfaction, GenAI software

DOI: 10.54941/ahfe1006673

Cite this paper:

​