Design and Evaluation of an AI Academic Advisor: Insights from Student Interactions

Abstract

Artificial intelligence (AI) is increasingly being explored to improve the efficiency and accessibility of academic advising in higher education. Traditional advising systems often face challenges, including limited advisor availability and increasing administrative workload, which can delay student support. This study presents and evaluates a conversational AI academic advisor built using a Retrieval-Augmented Generation (RAG) architecture that combines a large language model (LLM) with a curated knowledge base. The system allows students to ask advising-related questions and receive responses grounded in official documents. A pilot study with 20 university students was conducted in which participants interacted with the system to explore tasks for advising tasks before completing a usability survey. Results indicate generally positive perceptions of the system’s convenience and accessibility, with some participants reporting that the system was easy to use and helpful for obtaining information, and some indicating that responses were clear and understandable. Students highlighted the speed and immediacy of responses as key advantages. However, some participants reported response delays and limitations in handling complex questions. Overall, the findings suggest that AI advisors can effectively complement traditional advising services for routine inquiries.

Keywords: Artificial Intelligence, Academic Advising, Chatbots, Conversational Agents, Higher Education

DOI: 10.54941/ahfe1008077