Advanced Chunking and Search Methods for Improved Retrieval-Augmented Generation (RAG) System Performance in E-Learning

Abstract: Our study evaluates different search methodologies—Hybrid Search and Semantic Search—within a Retrieval-Augmented Generation (RAG) framework specifically for E-Learning. The primary objective is to enhance the accuracy and efficiency of using Large Language Models (LLMs), such as GPT-4, by employing advanced Prompt Engineering Techniques in E-Learning environments. Efficient search and chunking methods are critical for optimizing the quality of answers provided by these systems.To achieve this, we utilized the RAGas testing framework, focusing on performance parameters including Answer Correctness, Context Recall, Context Precision, Faithfulness, and Answer Relevancy. In our implementation, documents were divided into text chunks and indexed in a database using both vector and keyword indexing. This allowed for searches by vectors for similar records and keyword searches for exact matches. These records were then incorporated into prompts as context to improve LLM responses. The AI model used for generating embeddings, such as OpenAI's text-embedding-ada-002, plays a crucial role in this process by creating high-dimensional representations that capture deep semantic meanings.Current retrieval methods, like keyword and similarity-based searches, often fall short due to limitations in chunk quality, which directly impacts the accuracy of the RAG system. This study aims to improve the retriever component and, consequently, the overall accuracy of the RAG system by comparing three different chunking methods and two search approaches. We conducted tests using 57 questions across multiple files under various configurations.This research examines different search methods, including Hybrid Search, which integrates traditional keyword search with semantic search in order to provide more accurate and contextually relevant results. In comparison, Semantic Search utilizes deep learning models to comprehend the context and meaning of search queries and documents, thereby providing more precise information retrieval. The analysis also compared different chunking methods, such as Recursive Chunking, which divides text into hierarchical sections that are further subdivided until the desired granularity is reached. BERT Chunking utilizes the BERT model to segment text, taking semantic meaning into account to ensure coherent chunks. Token Chunking segments text based on individual tokens, offering fine-grained control over segmentation.Our results, evaluated using the RAGas testing framework, highlight the strengths and weaknesses of each search method and chunking technique. This study provides valuable insights into optimizing RAG Systems for E-Learning through advanced Prompt Engineering Techniques, aiming to improve knowledge transfer regarding efficiency and accuracy.

Keywords: Advanced Chunking, Semantic Search, Hybrid Search, Retrieval-Augmented Generation (RAG), E-Learning, Large Language Models (LLMs), Generative AI, Prompt Engineering Techniques.

DOI: 10.54941/ahfe1005756

Cite this paper:

​