Evaluating AI-generated Research Plans: Expert Insights from a Blind Authorship Study

Abstract

Artificial Intelligence (AI) is increasingly being adopted in academia to enhance various research activities and has proven to be a valuable research accelerator in recent studies. This study examines the capabilities of large language models (LLMs), specifically ChatGPT and Gemini, in generating comprehensive research plans during the early stages of research. We tasked both models with developing research plans on the topic of “Gen Z's willingness to sacrifice convenience for environmental benefits” including interview guidelines and survey questions. Eight expert researchers evaluated these research plans without knowing they were AI-generated. Our findings provide in-depth insights into the perceptions of expert researchers regarding the quality of AI-generated research plans, identifying missing elements and pitfalls of utilizing AI in the planning activity of research. The necessity for researchers to oversee and intervene in AI outputs is emphasised in our research to fully leverage the advantages offered by this technology.

Keywords: Artificial Intelligence, Large Language Model, ChatGPT, Gemini, Academic Research

DOI: 10.54941/ahfe1005843