Can LLMs assist in job interview preparation? Assessing the quality and effectiveness of LLM-generated feedback

Abstract

Large language models (LLMs) have demonstrated strong reasoning capabilities, making them potential candidates for generating formative feedback in learning contexts. This paper evaluates the ability of LLMs to provide formative feedback on interviewees' responses in a job interview task. Specifically, the degree of explanation in an interviewee’s response, a key communication skill, was used as the focal assessment criterion. Combinations of LLM models (i.e., GPT-3.5-Turbo, Gemini-1.5-Pro) with various chain-of-thought (CoT) prompting strategies, including task definition, domain knowledge, and contrastive prompting, are examined across multiple self-reported metrics of feedback quality effectiveness. Data was collected from 663 participants on Amazon Mechanical Turk using a between-subjects design with six experimental conditions, each corresponding to a combination of LLM model and prompting strategy. Results indicate that users perceived LLMs as having a moderate ability to provide formative feedback for job interviews, though the feedback was at times viewed as irrelevant or potentially harmful. The choice of LLM model and prompting strategy significantly influenced perceived feedback quality, with GPT-3.5-Turbo generally rated more favorably than Gemini-1.5-Pro. While stronger task performance occasionally aligned with higher user ratings, the relationship between performance and perception was not strictly linear. These findings are discussed in terms of design implications for enhancing the quality and effectiveness of LLM-generated feedback in interview training contexts.

Keywords: Large language models, formative feedback, job interview, human-AI interaction

DOI: 10.54941/ahfe1006917