Evaluation of ChatGPT-Generated vs. Self-Created Study Reviewers: A Case Study Comparing Student Test Scores, Efficiency, and Cognitive Workload

Open Access
Article
Conference Proceedings
Authors: Aliyah Maxine Mallari, Yngrid Bless Cadena, Aleeza Fiona Montinola, Lorelie Grepo-Jalao
Abstract

This study compared ChatGPT-generated and self-created reviewers in terms of engineering students’ test scores, time efficiency, and perceived cognitive workload. Using a within-subjects mixed-methods design, 30 engineering students participated. Each subject experienced using both a ChatGPT-generated reviewer and a self-created reviewer. Quantitative data were analyzed using descriptive statistics, ANCOVA, and correlation analysis, while qualitative responses were examined thematically. Results showed that ChatGPT-generated reviewers were associated with lower perceived cognitive workload and shorter average study time, while self-created reviewers produced slightly higher average test scores. ANCOVA results showed that the reviewer method significantly affected test scores, NASA-TLX workload, and time efficiency, even after controlling for covariates such as AI usage frequency, note-taking frequency, reviewer-making habits, mental exhaustion, and reviewer satisfaction. Note-taking frequency was also a significant predictor of test scores, with students who reported taking notes more frequently generally achieving higher test scores regardless of the reviewer method used. The reviewer-making habit was also a significant predictor of NASA-TLX scores. Participants who regularly create reviewers as part of their study routine reported higher perceived workload across both methods, showing differences in participants’ study habits. For time efficiency, none of the covariates were significant. Qualitative findings supported these results, showing that ChatGPT was preferred for speed and reduced effort, while self-created reviewers were valued for deeper understanding and retention. Recommendations for future studies include increasing the sample size, involving a wider population, and exploring a hybrid study approach.

Keywords: Student Learning, Reviewer Method, Generative AI, Workload, Efficiency

DOI: 10.54941/ahfe1008497

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← FULL PAPER ONLY: Beyond Tools: A Task-Based Human-AI Studio Scaffold for Critical Thinking in Architectural EducationA Conceptual Framework for Modeling Team Adaptation in Cooperative Games through Ludic Knowledge →
View all articles in Human Factors in Design, Engineering, and Computing →