Method to assess students summaries in an intelligent tutor systems using coherence and content analysis in a reading comprehension task

Abstract: In this paper, a discourse-based method that merges syntactic and semantic models for developing an automated system for reading comprehension assessment is proposed. The approach combines shallow linguistics features and discourse patterns to assesssemantic content and coherence of free-text responses by using computational linguistics and machine learning techniques.For evaluating semantic content, we use the classical models from the literature: Vector Space Modelling and Latent Semantic Analysis. For evaluating the coherence of a text, we use two types of models: semantic model and syntactic model. The semantic model uses word embeddings, this is, representing sentences from a document as mathematical vectors, and establishing the coherence of a text, as the mean distance between sentence vectors. We used two popular models from the literature: Latent Semantic Analysis and Word2Vec.For the syntactic modeling, we used an entity grid representation of the texts, which extracts syntactic patterns from the texts and relies on the assumption that coherent texts will have similar underlying syntactic patterns.To train the computational model which is in charge of assessing responses, we developed a rubric evaluating 2 different ítems: content and coherence (functional and referential). Both items are independent, thus two automated graders were trained.The contribution of this work is twofold: firstly, we develop a new methodology for free-text responses in which we assess student‘s texts by semantic content and coherence. Secondly, we develop an automated system for assessing student’s reading comprehension for Spanish Language using features that can be computed automatically. Experiments using experts’ annotated data show that the proposed system can provide useful feedback for students and that the assessments from the system correlate with the ones provided by a human.Moreover, our experiments show that we can get accuracies of 90% when assessing text content, and of 55% - 60% when assessing text coherence. With this model, we developed a web application in which a student provides a free text response from different prompts and the system provides automatic feedback which helps the student in improving his answer and thus, improving his ability to put ideas into text. After an intervention in Chilean schools consisting of a sample of 200+ students, we analyzed the coherence and content assessment from the tool. We computed correlations between the results and linguistics features from student texts. Results show that there is a significant correlation between discourse markers and average content and coherence scores. Now, if we consider separately the dimensions of the summary assessor, we found that several indices predict significantly the student's performance. In the case of summary content, indices explain nearly 20% of the variance in the data. In the case of coherence, the indices explain 33% of the variance in the data. Most of the indices are related to lexical properties and discourse markers of the text. Among the best predictors of coherence are closed class words, discourse markers signaling cause relationships, type-token ratios of adjectives, and syntactic overlap between verb synonyms across sentences.

Keywords: NLP, Natural Language Processing, Readability, Lexical Diversity, Text Complexity, Intelligent Tutor System Education, Distant Learning

DOI: 10.54941/ahfe1001172

Cite this paper:

​