A Global Questionnaire? An International Comparison of the System Usability Scale in the Context of an Infotainment System

Abstract: The System Usability Scale (SUS) is a widely used questionnaire to assess the subjective usability of interactive products or services. It is a cost-effective and time-efficient method which makes it a convenient instrument in the industry context worldwide. Past research has already demonstrated psychometric reliability and validity of the SUS in different languages. The translated SUS questionnaires were considered to be capable of measuring perceived usability. However, there are no international psychometric studies that empirically proof that the SUS can be applied for the same product in different international markets.Therefore, the aim of this study was to investigate if the SUS provides comparable results for the evaluation of the same product, an automotive infotainment system, in different markets. Participants from China (n = 36), Germany (n = 30) and the USA (n = 36) were asked to perform nine different use cases using the infotainment system of a series production vehicle. After completing the use cases, they evaluated the infotainment system using the SUS in their corresponding language. We assessed various psychometric quality measures to evaluate the SUS in the different markets.Cronbach’s α was used to measure reliability of the questionnaires. For further evaluation of sensitivity, an ANOVA with country being the independent variable and the score of the SUS being the dependent variable was conducted. Construct validity was examined by conducting factor analyses.The mean SUS scores ranked similar across the three languages and did not differ significantly between countries, F(2,99) = 1.07, p = .346, ηp2 = .02. Reliability analyses revealed high internal consistency in all three countries with Cronbach’s α ≥ .87. Item analyses indicated that correlations between single items and total scores were high across languages. However, there are inconsistencies in single item reliability analyses. Despite the overall consistent results on reliability, differences were found when comparing which item deletion would increase internal consistency in each language. This finding might question the assumption of the SUS being a global questionnaire with regard to its item structure. Furthermore, the assumption of a unidimensionality of the SUS could not be confirmed by factor analyses conducted for each country. The one-factor solution yields acceptable results, but an explorative two-factor solution is capable of explaining a higher total variance and also shows higher factor loadings. In addition, the items of each SUS translation load differently on the respective factors in both factor solutions, making it difficult to assess the globality of the SUS.Based on the results, the English, German, and Chinese translations of the SUS received validation support to a certain extent, but further research or adjustments of the translations are necessary to validate the SUS as a global questionnaire in the context of an infotainment system.

Keywords: usability, method development, HMI, SUS, intercultural comparison, infotainment system

DOI: 10.54941/ahfe1001711

Cite this paper:

​