From Paper to Pixels: Transferring Handwritten Note-Taking Into Virtual Reality

Abstract

While in the past, Virtual Reality and Augmented Reality needed hardware dedicated exclusively to one or the other, newer Virtual Reality headsets like the Meta Quest 3 combine both functionalities in one. This opens up possibilities to more easily implementable applications that use a combination of VR and AR, namely Augmented Virtuality, where a mostly virtual world is augmented with parts from the real one. We explore the possibilities that Augmented Virtuality can offer to enhance VR applications both from a theoretical perspective on the example of VR in driving automation and by offering a concrete prototype for note-taking in VR. This prototype, PaperVR, uses augmented virtuality concepts to enable handwriting on real paper inside a virtual environment. For that purpose, the front-facing cameras of a Meta Quest 3 were used, together with the MetaXR plugin, to create a tracked passthrough window, revealing the physical paper inside the virtual environment. As a point of comparison, a second prototype was developed which is intended to represent the current standard in handwritten VR note-taking, namely tablet writing. A user study was conducted, to compare both prototypes with each other and their equivalent writing methods outside of VR. It shows that paper writing is superior to tablet-based writing in VR for a synthetic task in our study. Additionally, in a practical note-taking task, it was possible to reach the same objective results inside VR as using physical writing outside a virtual environment, both regarding the number of correct answers and the answering speed per question.

Keywords: Augmented Reality, Augmented Virtuality, VR Text Input

DOI: 10.54941/ahfe1006712