Confidence Estimation in Multiple Choice Questions Using Eye Movements

Abstract: In recent years, with the spread of electronic devices, e-learning has become a common form of learning, especially among students and other young people. One of the most common forms of e-learning is multiple-choice questions. While multiple-choice questions allow the learner to grasp the answer instantaneously and reliably, they also allow the learner to answer correctly by guesswork or chance, which may cause the learner to ignore content that should be reviewed. Therefore, it is important to estimate the confidence level from the learner's mental information. Eye movements are often used as a method of ascertaining learners' mental information. Previous studies have suggested that saccades are effective in discriminating comprehension of sentences, and pupil size is effective in evaluating English word acquisition. In addition, it has been confirmed that the fixation of multiple choice questions changes depending on the learner's confidence level. The purpose of this study was to evaluate the relationship between eye movements, such as saccades and pupil size, and the level of confidence in multiple-choice questions. As an experiment, we measured eye movements using four-choice questions on Japanese translation of English words. Thirty questions were designed to have varying degrees of difficulty. A tabletop eye movement measurement device was used for eye tracking. The flow of one trial is described below. The participant gazed at the English word on the monitor for 5 seconds. After that, four choices were displayed on the monitor, and the participants answered the questions. The participants were instructed to answer the question immediately after the choice was decided. Finally, a post-questionnaire was conducted. In the post-questionnaire, the participants were asked to answer one of the following questions: "I could answer the question without any choice," "I knew the answer by looking at the choices," "I did not know the answer but guessed from the choices," or "I do not know the answer at all. The experiment was conducted on seven Japanese male university students (23.3±1.6 years old) in a random order of 30 trials each. Saccade frequency and mean pupil size were used as evaluation indices. The percentage of fixation time in the answer choices was also evaluated based on previous studies. The confidence level was defined as "recall," "recognition," "guess," and "intuition," in descending order based on the responses to the post-questionnaire, and classified into four groups. The results of multiple comparison tests showed that the percentage of fixation time of the answer choices was significantly larger when the answer choice was "recall" than when the answer choice was "guess. On the other hand, there was no significant difference in saccade frequency and mean pupil size. The reason for the lack of differences may be that there were some trials in which the pupil size did not change because the time to answer the questions was too short. Therefore, as an experiment for improvement, we changed the contents of the multiple-choice questions and are studying the contents of questions that require more time. By increasing the answer time, we expect to see different characteristics of eye movements depending on the level of confidence from the previous experiment.

Keywords: eye movement, pupil, saccade, multiple choice question

DOI: 10.54941/ahfe1001801

Cite this paper:

​