Franco-Taiwanese Research on Extended Reality Experience

This technical note provides descriptions on a collaborative research project between Université Paris 8/École Pratique des Hautes Études-Laboratoire des Usages des Techniques d’Information Numériques (Paris8/EPHE-LUTIN) and National Taiwan Normal University-Science Education Center (NTNU-SEC). This project, entitled “Franco-Taiwanese Research on Extended Reality Experience”, is about research and development of new, innovative, performant eXtended Reality (XR) contents, methods, procedures, and services in order to enhance science teaching and learning and to bring subsequent great benefices to education. The aim of the collaboration is the making of eXtended Reality eXperience (XRX) for teachers and pupils of the 12-year basic education curricula. As a result of reaching our aim, we expect that the XR systems and instructional modules developed through our project will be learning efficient for the students acquiring knowledge and the know-how of Science Education.


INTRODUCTION
As technology continues to advance, people around the world are eager to take advantage of the rapid development. With the help of the media, more people become interested in Augmented Reality (AR) and Virtual Reality (VR) or Mixed Reality (MR). AR can be defined as "a situation in which a real world context is dynamically overlaid with coherent location or context sensitive virtual information" (Klopfer & Squire, 2008, p.205). It carries the properties of combining real and augmented object, running interactively in real times, and registering real and augmented objects with each other (Azuma et al., 2001). VR refers to "the technological hardware that creates the dimensions of experience affording different levels of vividness and interactivity in an immersive or parareality environment" or "artificial simulations, usually recreation of a real-life environment, that enhance an imagery reality or situation" (Tham et al., 2018, p. 180). In VR, users can be completely immersed in a virtual world which allows a user to step through the computer screen into a 3D world (Fast-Berguld et al., 2018). According to Heim (1993), there are seven elements of VR: simulation, interaction, artificiality, immersion, telepresence, full-body immersion, and networked communications. MR can be defined as the blending of the physical world and digital world  or any merging of the any merging of real-world objects or information into the virtual world (Tham et. al., 2018). As for eXtended Reality (XR), it includes different forms of combined real and virtual environments (Milgram et al., 1995). XR is an emerging umbrella term for all the immersive technologies which covers all the descriptive forms of AR, VR, and MR.
Based on potential market size considerations, most of the attention and interest surrounding XR is focused on entertainment and advertising. However, it is now gaining more applications in the area of education, learning and training. For instance, XR technology can enable students to understand the tools and technologies they face after entering the industry through simulation and reducing the gap between academics and industry. In XR Virtual Lab, students can experience the fun of immersive learning through the interface of humancomputer interaction which can enhance their learning motivation (Akcayir et al., 2016). XR has great potential for development in education. With the continuous advancement of XR technology in educational applications, it can become the trend of emerging 2 / 11 technology teaching in the future and will bring more possibilities to education. Expected results are twofold. First, we expect that recommendations and specifications about the specific processes of teaching/learning with the usage of XR, and also about (i) how to assess learning on line to provide operational feedback in the loop of adaptive learning (ii) how to predict and correct misunderstanding or learning decay; and (iii) how effective supervision of the teacher and learner Experience of XR can improve motivation and be inspiring the best didactic strategies. Second, we expect main result would be the transferability of the expertise of how to design efficient XR environments, to other Science Education matters and to others countries at international. These two goals are pursuit by the labs briefly described as follows:

Laboratoire des Usages des Techniques d'Information Numériques (Paris 8/EPHE-LUTIN)
LUTIN is a platform for usability observations and experimentations hosting most of the analytical equipment required for its work. It provides access to various shared equipment such as eyes-tracking systems, evoked potentials systems, physiological recording systems, video recording, and analysis. Advantages are participants for observations, technologies for observation and experimentation, cognitive simulation, interface between disciplines. The working principle involves new products or services being co-innovated by researchers, companies, and individuals. LUTIN is a meeting point between technologies and usages and a place where new experimental methods can be applied, with consideration being given at every stage to the final user, thus facilitating the transition between R&D prototypes and innovative products that are totally adapted to their proposed uses.
LUTIN's main activities are about Knowledge Technologies (Web tools for topologies, Web trails analysis, virtual class tools, e-learning, serious gaming, etc.), Usability testing with an observational approach, both at the user lab level and in the real environment of work or life (usability and acceptability). For this purpose, LUTIN is partnering with several Academies (Créteil, Versailles, Rouen, Orleans), with universcience (Palais de la Découverte and Cité des Sciences et de l'Industrie, visited by 3 million people a year, who are given the opportunity of participating in observation studies as well as experiments and usage tests of new knowledge digital devices), with universities, content publishers, developers devices for production, transmission and display of contents (from eBooks to immersive 3D rooms, through Augmented Reality and Virtual Reality or Mixed Reality, interactive whiteboards and video enhanced). There is a longstanding working evaluation on AR since more than a decade ago, (Pinska & Tijus, 2007), VR (Vandi &Djebbari, 2011) andMR (de los Rios et al., 2015), with observation and experimental methods and instruments such as those based on eyetracking (Benedetto et al., 2013;Krejtz et al., 2016;Le Meur & Baccino, 2013) as well as modeling and

Contribution to the literature
• The current project has brought positive responses for teachers to create their own innovative AR/VR instructional modules used in STEM education • Several XR-enhanced teaching and learning workshops and training for in-service teachers have been held to help equip more teachers to enhance their skills in incorporating XR technology in their teaching. • Through XR technology, learners can interactively explore a variety of interesting subjects and topics on their own in a highly simulated environment. • The research can serve as a framework for the future design of innovative VR/AR enhanced learning environments to enhance efficient and effective teaching and learning.

National Taiwan Normal University-Science Education Center (NTNU-SEC)
Through XR technology, learners can interactively explore a variety of interesting subjects and topics on their own in a highly simulated environment. Even in the case of experimental errors, learners will not result in serious consequences. In the field of education, XR seamlessly integrates the virtual and real world, and can enhance the sensory experience of learning from 2D to 3D, especially for textbooks, maps and stars. Students may find it especially useful for those who have difficulty in spatial cognition. Moreover, teachers can guide students to develop their logic thinking through virtual environment. This is merely impossible to achieve in the past, especially in the educational environment that requires high-risk, high-cost equipment, and macroscopic and microscopic issues that are invisible to the naked eye, such as astronomy or microbiology. With innovative technology, SEC has developed a VR lab which relates to chemistry, physics, and earth science. Users can easily interact with and manipulate images via mouse and keyboard. It is anticipated that these VR technologies can in some ways facilitate students in problem solving, reasoning, and critical thinking abilities.
In addition, SEC has also developed an AR lab to facilitate students to learn concepts in different science courses. AR Software Development Kit is used to create an image target. Through the image target made in the AR Extension, learners are enabled to view 3D structure models, interactive animations and videos as they point them in-device camera at the designated tagged objects (as shown in Figure 1). For example, once the in-device camera is pointed at the Image Target, the 3D model will show up. When students touch the model on the handheld device, more detail information will appear as interactive animations and videos. Students may rotate and view the model at different angles. Base on this AR Science Education Lab, AR system can be integrated in the formal classroom settings of any science courses to promote active learning and turns passive learners into active learners by engaging them into interactive assignments.

METHOD: APPLICATION OF MULTI-SOURCE COGNITIVE TECHNOLOGIES TO LEARNING
The methods are based on the online collection of sequential multi-source behavioral data for the study of learning in the context of the use of immersive devices. This multi-source data technology is already at work in the eLab project of the City of Sciences and Industry 3 . These methods are applied to the use of autonomous cars, robotics, virtual and augmented reality (VR-AR) and real-virtual hybridization systems 4 , by implementing synchronization, sampling and analysis tools (unsupervised and supervised research of patterns) and more generally on the design, accessibility, acceptability, learnability and usability of digital technologies. All these methods are carried out within the frame of the Living Lab methodology of a responsible research and innovation that requires the voluntary participation of people.
A number of technologies designed, tested and evaluated are intended for the general public and many for use at school, like technologies for the classroom of the future, and for which teachers come to LUTIN experiment, or even like technologies for people with Autism Spectrum Disorder (ASD) for whom the accompanying persons travel to the LUTIN 5 .

VR Equipment: FOVE, HTC VIVE, and OCULUS RIFT
VR equipment such as FOVE, HTC VIVE, and OCULUS RIFT increase the powerful utility of extended reality experience by having eye-tracking as an innovative kind of interaction time series data in virtual reality headset.
Among LUTIN's headset, FOVE 6 was mainly used in order to get eye-tracking in the Virtual environment. The FOVE device is WQHD OLED (2560X1440), frame rate of 70fps, field of view up to 100 degrees, equipped with eye-tracking sensors made of infrared eye-tracking system x2 with a tracking accuracy less than 1 degree and a frame ratio 120 fps.
Unlike most eye control systems, FOVE does not rely on the definition of zones that are activated during looks, it exploits a natural and instinctive behavior of the look, which in return requires only minimal concentration, and learning on the order of a few seconds. The visual coordinates are interpreted as a continuous flow of information, which will modify the positioning of the subject in the environment, which will reciprocally, by reflex, readjust their gaze to adapt to these modifications. These two principles will talk to each other according to a virtuous loop which will aim to cancel each other out, in which case this will mean that the target is reached, or conversely to talk, as long as the desired target (looked at) is not reached. The result is a "hands-free" movement system that combines simplicity, comfort and precision, without equivalent currently known.
The FOVE 3D and raised interfaces of VR headsets offer immersive contexts of use, often inviting the user to move physically to move around the environment. The fact that virtual reality headsets are enslaved to users provides ideal conditions for using this system. The proximity between the eye and the gaze tracking device makes it possible to obtain a very good quality of data, which will allow a reliable and stable interpretation on command. Figure 2 shows the eye-tracking data collected with FOVE.
These conditions would allow not only to move and aim vertically and laterally, but also back and forth depending on the distance at which the point of view is carried, all this without effort or decision-making and of acting. This property of facilitating the use of the device and paying more attention to content and to the learning task at hand might be of benefit. For example, LUTIN analyzed and tested a virtual environment dedicated for teaching children to take the bus by themselves with such a virtual environment of Bus taking simulator. Results with ASD children were that they were taking into account (quite as usual) details of the virtual environment for their decision making about the process of bus travelling. Figure 3 depicts he VR environment for taking the bus at the station, with eye-tracking showing where the participant is looking at.
Within the study, the Learning objective were about the training of the "key" elements of taking the bus (buying a ticket, reading a timetable, and requesting a stop). The target population were people with cognitive deficits such as ASD and observational data were about interface assessment (eye-tracking, knowledge assessment, semi-structured interviews) and the issue was the creation of a guide of educational and ergonomic suggestions.

Collaborations
Collaborations with Research Institute as well as SMEs Developing VR as well as XR Solutions and with France Immersive Learning Lab ensure the precision of the immersive technologies that could be modelled within VR.
In addition to the SEC at the international level, at the national level, for new media such as video games and serious games, LUTIN collaborates with federative structures, innovative companies and researchers (in relation to CAP DIGITAL, PARIS REGION ENTREPRISES, and PLAINE COMMUNE) and has indepth collaborations with the Ecole Nationale

Validating the Improvement of the Realism of Art-of-Corner of VR Content with Photogrammetry
According to Frédéric Purgal 10 , CEO of Art-of-Corner from photo capture and 3D model creation using Capturing Reality software to the development of an interactive application using Unity via SteamVR*, Artof-Corner "has built an end-to-end workflow that gives exceptional and enchanting VR results. Art-of-9 https://www.youtube.com/watch?v=h615SgCfuDQ 10 https://software.intel.com/content/www/us/en/develop/articles/artofcorner-and-capturing-reality-bring-a-130-year-old artists-studio-to-life.html?fbclid=IwAR09IZxUx5H_rKaqXmgZvB1uXsZPLy7Apio4gnBFc4xQA6T_cdANYfCL-PY 11 https://www.youtube.com/watch?v=MSkVheZQ5FM&feature=youtu.be Corner's choices were determined by the studio's overall objective of creating a highly attractive VR experience with a suitably high level of visual quality within a compact creation timeframe. Given this criteria, the input to the VR application creation process was fixed at 3,000 photos at FHD quality. An Intel® Core i9 X series hardware platform was chosen to run Adobe* Lightroom, DxO Photolab*, RealityCapture*, and Unity* software applications." LUTIN has also evaluated how much different is the real visit of Bourdelle's Atelier 11 with the VR experiment. First results state that details are better and deeply cognitively processed, through focused attention, than scanning distributed attention. This could be due to the difficulty to change location while looking at things ( Figure 6).

Measuring the Precision of Presence When Interacting with an Avatar
In a context where AR, VR, and XR technologies are evolving rapidly (native integration of an eye tracker, better screen resolution, increase in the degrees of field of vision and in the number of frames per second, external cameras to locate the user in the environment, etc.), we need to ensure that presence in the virtual is efficient and effective.
Presence, that we define as how much cognition is "embodied" in the virtual environment, is evaluated as in Figure 7 by measuring for instance how much the body movement of your avatar (or of the Rodin sculpture, Figure 7-left) is processed in advance with visual anticipation, when the saccadic eye-tracking data (Figure 7-right) indicates that the visual perceiver is looking outside the sculpture at a next location where a body-part would be if the sculpture was moving as a living person.
The goal of the experiment on the contribution of device in immersion about the feeling of presence in a virtual environment is to measure the effect of factors influencing this feeling and the quality of interaction with the environment and with the virtual characters that populate this environment, including our avatar in a third person view.
Immersion allows the belief that the person is "present" in the virtual environment. Thus, it follows that the success of a virtual reality experience requires a high level of immersion in the virtual environment, although it does not erase the original environment, which justifies the study of the effects of locationrelocation, or even bilocation, and an approach to the embodiment, physiological and cognitive states. Thus, a study defines immersion as a psychological state characterized by the perception of being or feeling "enveloped by", "included in" and "interacting with" an environment offering a continuity of various stimuli and experiences. It suggests that the factors affecting immersion include (i) getting away from the physical environment, (ii) perception of feeling included in the virtual environment; (iii) the "natural" state of interactions and the perception of control, and (4) the perception of movement in a virtual environment (Witmer & Singer, 1998).
The research of presence is about the relation between the body and the avatar (Figure 8). To understand how the human participant is "captured and transported in interactive virtual environment", Figure 7. Evaluation of being there, here and now, by modeling the next movement of the avatar or of a Rodin sculpture (left) and verifying with eye-tracking data how much the human visual perceiver is looking at the body movement by anticipating it (right) "Become an avatar" based on the paradigm known as the "Astroscopic Flying Avatar", in which a user is video-taped on a green background to be re-embedded in the virtual environment, so that s/he can see herself through a head-mounted display. Thanks to a direct control by a gestural sensor that controls its movements, gestures and change of location by its own movements. A neurophysiology research station has been designed, as well as an interactive installation, with possible use in museography and scientific mediation. Two variants scenarios were developed. The exploitability of this realization for the benefit of different lessons comes from the fact that it hybridizes, on the one hand, displayed elements (with a "video" avatar and not of synthesis) with logic of simulation with, on the other hand, principles of VR interaction (wireless sensors, analog control).

Blue & white porcelain Zen painting with AR technology
The topic of cultural heritage and educational significance is currently in the art curriculum of the senior and junior high schools in Taiwan. It is an ongoing teaching course which focuses on student-centered learning and promotes students' active learning. Therefore, the curriculum and instructional materials need to be linked to students' daily life experiences to strengthen their aesthetic cognition and practical application of aesthetic enhancement. The main purpose of this curriculum art course is for teachers to facilitate students in developing their own AR-based scientific blue-and-white porcelain and Zen painting. The general idea of the curriculum is for students to first learn about the fundamental knowledge regarding blue and white porcelain and Zen painting, such as natural science and ecology, and mathematics geometry. Then, students design and develop their own hand-painted art followed by acquiring the related engineering technology, such as kilning of the porcelain. Finally, students incorporate the digital and 3D technology through the use of AR technology into their art piece.

Digitally interactive globe system (DIGS)
DIGS uses a 3D simulated earth system, and utilizes the existing classroom equipment to improve learning performance in the Earth Science course. Our system projects a 3D spherical image of the Earth. In addition, this system allows the operator to use a simple handheld wireless control device to interact with the 3D interactive globe system. DIGS involves a data processing unit, a wireless control unit, an image capturing unit, a laser pointing device, and a 3D hemispheric body imaging unit -a kind of 3D screen. DIGS is a user-friendly system in the classroom and allows the users to navigate from one concept to another many times according to their pace. This provides an opportunity for interaction between the user and the system. Users can use the functions like click, zoom in-out, rotate, etc. based on their learning needs. DIGS can be used in schools, planetariums, museums, and exhibitions. Through this system, we attempt to improve learner's learning performance in the earth science course.

FRANCO-TAIWANESE RESEARCH ON EXTENDED REALITY EXPERIENCE
During our research project, we researched and developed a hybrid virtual-real, innovative, and performant XR contents, based on reliable methods, procedures and services in order to enhance User eXperience, Human-Computer Interaction, Human-Human Interaction, including science teaching and learning intended to bring benefices to education.
Expected results are twofold. First, we expect that recommendations and specifications about the specific processes of using XR, about how to assess learning on line to provide operational feedback in the loop of adaptive learning, about how to predict and correct misunderstanding or learning decay and about how 12 https://www.ibm.com/topics/what-is-a-digital-twin 13 https://www.merriam-webster.com/dictionary/avatar#synonyms 14 https://www.inpi.fr/fr/l-internet-des-objets-et-la-propriete-intellectuelle effective supervision of the teacher and learner Experience of XR can improve motivation and be inspiring the best didactic strategies. Second, we expect the transferability of learning in situations with no Extented Reality Technologies.
A Digital Twin is a virtual representation of a physical entity (object, system, and living beings) that spans its lifecycle, is updated from real-time data, and uses simulation, machine learning and reasoning to help decision-making 12 . As a difference with an Avatar 13 that can enhance the inside of the virtual world with a virtual representation of real entity, a Digital Twin enhances the real world with a real representation of a virtual entity. For instance, on the one hand a gamer can choose or design an avatar of a car and of a human being that will represent, inside the virtual world of the videogame, her car and herself, respectfully. On the other hand, the real physical car and its real physical driver can have in the real physical world the Car Digital Twin (CDT) and the Driver Digital Twin (DDT). Thus, a real level 3 autonomous car can be, for instance, advised by DDT that the real driver stars sleeping while the real driver can be alerted by CDT that the left front wheel of the real car has abnormal wear.
Note that if these two CDT and DDT cooperate, they might improve the safety of the whole driving situation through the modelling and simulation of metacognition. For instance, if the car gets the information through DDT that the real driver does not pay attention to the information about the left front wheel, it might find a way to alert DDT that can alert, in turn, the real driver according to the knowledge he has from itself, thus from its human twin.
According to INPI 14 , Digital Twin should not be confused with Internet of Things (IoT). For IoT, we suck up data from a physical object and its environment using sensors, while for Digital Twins, we inject data into a virtual object, which we evolve as its aging." However, data nourishing Digitals Twins are to be collected from sensors (e.g., environmental sensors, Remote-users' sensor and wireless close-to-physical body sensors).
The questions under investigation are about a) the benefits of Digital Twins for the physical real object as well as for the real human being, b) how Digital Twins can interact for information exchange and decision making, c) how they can coordinate and plan the tasks execution and d) how to design the metacognitive systems of such digital systems and e) their insertion in the XR experience of people for museum mediation through dialogs.
We therefore investigate the design of corresponding Digital Twins to enhance XR. These are done in the context of the science of simulation theory 15 . Thus a Digital Twin (Y) of the simulation is an existing X thing (object, system, living thing) that is intended to be a substitute of X, considering that the Y-to-X relations have X-to-Y physical, eXtended-to context, and cognitive counterparts.

IMPACTS OF THE CURRENT PROJECT
Through this international research collaboration between France and Taiwan, there will be potential mutual benefits for NTNU-SEC and Paris 8/EPHE-LUTIN. For NTNU-SEC, the potential benefits and impacts are about being top-leader in Science Education technologies in designing efficient XR environment using innovative XR technologies and having the opportunity to share the state-of-the-art XR technologies with the Paris 8/EPHE-LUTIN. NTNU-SEC will be able to conduct cross-nation comparison research studies on the XR-based learning outcomes between those of the Taiwanese and French and receive feedbacks from international learners of XR learning environment. Since Paris 8/EPHE-LUTIN specializes in using EEG and Eye Trackers for cognitive development research, NTNU-SEC will be benefitted by receiving useful consultations and advices in research on the learning mechanisms in XR environment. Paris 8/EPHE-LUTIN will benefit from the development of techniques and methods related to cognitive technologies to evaluate and guide learning in the next digital age of education. This collaborative project has brought impacts in the following aspects:

Academic Research Aspect
• The current project has opened up opportunities for close research collaborations between NTNU-SEC and Paris 8/EPHE-LUTIN. The participation of the current project has brought positive responses for teachers to create their own innovative AR/VR instructional modules used in STEM education and has served as a framework for the future design of innovative VR/ARenhanced learning environments to enhance more efficient and effective teaching and learning.

Innovative Technology Aspect
• Research and development of a hybrid virtualreal, innovative and performant XR contents, based on reliable methods, procedures and services (XRT), have been conducted in order to enhance User eXperience, Human-Computer Interaction, and Human-Human Interaction, including science teaching and learning to bring benefices to education. 15 The science of simulation is providing criteria about how much a simulation of X is a valuable substitute of X.

Socio-Economic Impact
• At present, there are limitations of the current Digital Earth (DE) systems used in schools, such as high cost, inability to represent the whole Earth, lack the necessary advanced functions in 3D visualization. However, through our research project, the system of cost-effective DIGS has been developed by integrating low-cost equipment. DIGS includes a data processing unit, a wireless control unit, an image capturing unit, a laser emission unit, and a 3D hemispheric body imaging. DIGS can project a 3D spherical image of the earth and allows the operator to use a simple handheld wireless control device to interact with the 3D interactive globe system. Since most of the systems are very costly to be included in the mainstream educational systems, this low-cost DIGS makes it affordable for schools, even for those in the rural areas, as well as in the developing countries.

Talent Cultivation Aspect
• Due to the lack of connections with foreign researchers, young scholars in Taiwan are less likely to collaborate with world-class research teams and thus have very few chance to play influential roles in academia. To overcome this disadvantage, the current project can help cultivate young domestic scholars by providing the opportunity to collaborate with world-class scholars and their research teams. With the international collaboration experiences, we believe that young scholars who participate in this project will gradually build up their influence in the international community and consequently strengthen the national competitiveness of Taiwan.
• Several XR-enhanced teaching and learning workshops and training for in-service teachers have been held to help equip more teachers to enhance their skills in incorporating XR technology in their teaching.