Development of an AI Literacy Scale Using Multiple-Choice Questions

Abstract: With continuously emerging and developing artificial intelligence (AI) technologies, we now have more opportunities to interact with AI agents, to use AI applications to assist our jobs, and to assess the solutions provided by AI. This ability to properly identify, use, evaluate, and collaborate with AI-related products is referred to as AI literacy (Long & Magerko, 2020; Wang et al., 2022). The objective of the current study is to develop an instrument to measure general users’ AI literacy by replacing subjective self-report questions with objective, multiple-choice questions. 12 questions were derived from four dimensions of AI literacy (i.e., awareness, evaluation, ethics, and future AI), and a total of 230 validated responses were collected through the online survey. After deleting an unqualified item, the explorative factor analysis revealed a 3-factor structure of the remaining 11 items in the AI literacy scale: interacting with AI products, understanding AI’s capabilities, and understanding AI’s limitations. Each sub-scale is of acceptable reliability and validity. Furthermore, we examined the relationships between AI literacy and actual use of AI products, digital literacy, the attitude towards AI agents, and individual characteristics such as gender and education. The results suggested that a higher level of overall AI literacy was associated with better digital literacy, and a more positive attitude towards AI agents. The ability to interact with AI literacy, however, was correlated with more negative feelings about AI agents. Gender and differences in education were shown to have a significant impact on AI literacy. This exploratory study contributes to developing a more objective measurement of general users’ AI literacy and provides some insights on users’ attitudes towards AI.

Keywords: artificial intelligence literacy, digital literacy, attitude towards artificial intelligence

DOI: 10.54941/ahfe1004683

Cite this paper:

​