Evaluation of ChatGPT-Generated vs. Self-Created Study Reviewers: A Case Study Comparing Student Test Scores, Efficiency, and Cognitive Workload
Abstract
This study compared ChatGPT-generated and self-created reviewers in terms of engineering students’ test scores, time efficiency, and perceived cognitive workload. Using a within-subjects mixed-methods design, 30 engineering students participated. Each subject experienced using both a ChatGPT-generated reviewer and a self-created reviewer. Quantitative data were analyzed using descriptive statistics, ANCOVA, and correlation analysis, while qualitative responses were examined thematically. Results showed that ChatGPT-generated reviewers were associated with lower perceived cognitive workload and shorter average study time, while self-created reviewers produced slightly higher average test scores. ANCOVA results showed that the reviewer method significantly affected test scores, NASA-TLX workload, and time efficiency, even after controlling for covariates such as AI usage frequency, note-taking frequency, reviewer-making habits, mental exhaustion, and reviewer satisfaction. Note-taking frequency was also a significant predictor of test scores, with students who reported taking notes more frequently generally achieving higher test scores regardless of the reviewer method used. The reviewer-making habit was also a significant predictor of NASA-TLX scores. Participants who regularly create reviewers as part of their study routine reported higher perceived workload across both methods, showing differences in participants’ study habits. For time efficiency, none of the covariates were significant. Qualitative findings supported these results, showing that ChatGPT was preferred for speed and reduced effort, while self-created reviewers were valued for deeper understanding and retention. Recommendations for future studies include increasing the sample size, involving a wider population, and exploring a hybrid study approach.
Keywords: Student Learning, Reviewer Method, Generative AI, Workload, Efficiency
DOI: 10.54941/ahfe1008497
Cite this paper
More from this volume
- AI-Enhanced Live-Streamed English Language Learning: Designing and Evaluating an Intelligent Platform for Chinese Learners
- "They Will Move": Confirmation Bias, Expectation Frames, and Premature Monitoring Drop Off in COLREG Encounters
- Automation transparency communication does not improve supervisory control performance, but shapes operator trust
- The influence of neuroticism on startle and surprise response: a flight simulator study
- EEG-Based Classification of Reward, Penalty, and Movement Events Across Four Video Games of Varying Complexity
- The Effects of Task Similarity and Time Pressure on Task Performance Following Interruptions in Complex Human-Computer Interaction Systems
- Trusting the Chain, Not the Black Box: A Provenance-Based Architecture for Human-Machine Teaming in Military Autonomy
- Design of a Simplified Humanoid Robot Face: Cultural Differences and Emotional Synchronization
- From Visual Observation to Auditory Interaction: A Human-Centered Tangible Music System Based on Physical Objects
- The Impact of Scarcity-Based Dark Patterns on User Performance, Perceived Darkness, and Workload in Subscription Services
- A Behavioral Observation Tool for Diagnosing Human–AI Interaction Profiles in Engineering Design Education
- The Mediated Moon: Semantic Mediation Between Gesture and Generative Image


AHFE Open Access