AI-based Chatbot Coaching for Interdisciplinary Project Teams: The Acceptance of AI-based in Comparison to Rule-based Chatbot Coaching

Abstract

Project-based work is integral in corporate and academic settings, where coaching plays a crucial role in enhancing team performance and project success. To streamline this process and improve scalability, we developed a coaching chatbot at TH Köln/University of Applied Sciences to assist interdisciplinary teams. Utilizing a systemic coaching approach, the chatbot prompts self-reflection through solution-focused questions. We collaboratively created it with student facilitators and lecturers and tested it during a University-wide Interdisciplinary Project Week in November 2023. The pilot study involved two versions of the chatbot: a rule-based system and a hybrid model incorporating generative AI capabilities. As part of the field test, we analysed its acceptance: How effective is the chatbot in supporting projects groups and facilitating reflection processes? Are there differences in acceptance between the two chatbots? Half of the project groups in the one-week course used the rule-based chatbot, while the other half of the project groups were provided with the AI-based chatbot. 134 students participated and used the chatbots at the end of each day of the project week. The results of this study indicate that our test subjects accepted both types of chatbots with moderate to good scores in acceptance. However, the AI-based chatbot fared significantly worse in terms of performance expectancy and effort expectancy. This is possibly due to the fact that hybrid coaching chatbots are neither widely developed nor researched. We conclude that regardless of the technical basis of such a chatbot, conversation design and prompting is an essential part of chatbot development and contributes significantly to acceptance. This study demonstrates the potential of chatbots in supporting group coaching, not only in educational settings but also in corporate environments where they can aid agile project teams. This research marks one of the initial explorations into the acceptance of group coaching through chatbots.

Keywords: chatbot, coaching, AI-based coaching, rule-based coaching, group coaching, interdisciplinary project work

DOI: 10.54941/ahfe1005484