The Effects of Game-Based Learning and Anticipation of a Test on the Learning Outcomes of 10 th Grade Geology Students

This study examines whether a Role Play Game (RPG) with embedded geological contents and students’ anticipation of an upcoming posttest significantly affect high school students’ achievements of and attitudes toward geology. The participants of the study were comprised of 202 high school students, 103 males and 99 females. The students were divided into four groups: 1. RPG group with pre-announced posttest, 2. RPG group with unannounced posttest, 3. No RPG group with pre-announced posttest, and 4. No RPG group with unannounced posttest. A 2x2 MANCOVA was conducted on the posttest scores with students’ pretest scores as the covariates. The results indicated that: (a) there was no statistically significant interaction effect between RPG and anticipation of posttest on students’ learning outcomes; (b) there was no statistically significant main effect for RPG on students’ learning outcomes; however, (c) whether or not students had anticipated an upcoming posttest significantly affected their geological achievements and attitudes. In conclusion, testing has positive effects as a reinforcement to help students both retain their content knowledge and have positive attitudes towards geology.


INTRODUCTION
Computer and video games are now a pervasive part of today's society.A survey conducted in 2002 (Jones, 2003) to 1,162 college students of 2-year and 4-year public and private colleges and universities in the US indicated that virtually all college students have prior experience playing computer or video games.Among the college students surveyed, 65% of them played video, computer or online games occasionally or regularly.Almost half of the college students (48%) admitted that gaming, to some or great extent, kept them from studying.In particular, 9% of the students even confessed that their main purpose for playing games was to avoid studying.In addition, one third (32%) of the students also confessed to playing games that were irrelevant to instructional activities during their classes.Another study conducted in Slovakia (Fančovičová, & Prokop 2008) surveyed 206 students from eleven different elementary schools suggested that 64% of students' regular activities with computer use were for playing computer games, followed by working on the internet (27%), and writing (26%).
Since electronic games play such a prominent role in young people's lives, researchers and educators therefore hope to combine the intrinsic motivation that students show towards electronic games with educational content and objectives to make learning more fun and enjoyable (Marina Papastergiou, 2009a).As a result, harnessing the power of computer games in instructional designs to increase students' learning incentives and to positively affect their learning outcomes has become of great interests among researchers and educators (Hwang, Wu, Chen, 2015;Manero, Torrente, Serrano, Martinez-Ortiz, & Fernández-Manjón, 2015;Lee, Chen, & Chang, 2014;Jamet, 2013;Baek, 2008).However, as more instructional games emerge as an increasingly popular tool in instruction, their effectiveness as an instructional tool needs to be examined.
Few research findings on game-based learning showed that instructional games do not necessarily result in desired motivational properties and instructional gains (Hays, 2005).In the report of educational use of games, Mc.Farlane, Sparrowhawk, & Heald (2002) concluded that teachers in general were skeptical towards content-related learning with video games.Despite of the negative outcomes or skepticism of the use of gaming approach in teaching and learning, studies that reported positive learning outcomes through the gaming approach still outweighed those of the contrary.
Several studies which evaluated the impact of adopting instructional games in various disciplines have revealed positive outcomes in students' learning motivation and effectiveness (Yang, 2012;Rosas et al, 2003;Virvou, Katsionis, & Manos, 2005;Hwang, Wu, Chen, 2015;).The study on the effectiveness of game-based learning in high school computer science education also indicated that the gaming approach was more effective in promoting students' knowledge of computer memory concepts and triggering students' motivation in computer science learning than the non-gaming approach (Papastergiou, 2009b).
Existing research has shown that test taking can serve as an aid for students to retain learned materials (Haynie, 2004;Haynie, 2003;March, 1984;Nungester & Duchastel, 1982).In Haynie's previous study, over 88% of the 110 college students reported that they would not study materials unless they expected the materials to be reflected on a test (1997).Similarly, in Marchant's study (2002)

1381
were investigated.The results revealed that announced quizzes gave students more incentives to study for the subject.Students who received the announcement outperformed and read the assigned article more thoroughly than those who did not receive the announcement of quizzes.
A Role Play Game (RPG) with the embedded geological contents was developed in this study to serve as a reviewing tool for 10 th grade geology.It was hoped that through the fun and enjoyment of the game, students could engage in an active learning process to review their geological content knowledge even when they were not aware of an upcoming test.Therefore, the purpose of this study was to determine the effects of game-based learning and anticipation of a test on high school students' achievements of and attitudes toward geology.The following research questions guided this study: 1. Is there any interaction effect between the use of RPG and anticipation of posttest on students' achievements and attitudes in geology? 2. Is there any main effect on the use of RPG in students' achievements and attitudes in geology? 3. Is there any main effect on the anticipation of posttest on students' achievements and attitudes in geology?

Software and Role Play Game (RPG)
The RPG employed in this study was designed using the RPG Maker XP software and embedded with 10 th grade earth science geological contents.Designed as a reviewing tool to help students refresh their knowledge of geology, the RPG highlights important concepts taught in grade 10 geology.Embedded with the topics of mudflow, rocks, landslide, geographical change, tectonics, earthquakes and field sampling, the game is formatted by the following distinctive instructional elements.

Graphic illustrations
Along with the textual explanations, pictures and photos are shown throughout the game.As shown in Figure 1, the designs in the background of the game change in accordance with the different geological sites to simulate the real settings and to trigger students' interests in learning.

Field trip simulation
The game simulates several field trip sites which provide students an opportunity to experience the process of field sampling and investigation through the fun and enjoyment within the game (as shown in Figure 2).

Content review and multiple choice questions
Throughout the game, students encounter tasks to complete and problems to solve.Multiple-choice questions are used to test students' content knowledge.Figure 3 below illustrates an example of a multiple-choice question on a fault effect of a rock formation.Based on the question asked, students are required to respond with the best possible answer to proceed with the game.In the event of an incorrect answer, students are provided with the correct answer as well as an explanation.

Video clips
The game also incorporates several short video clips as a supplement to provide players additional learning materials (as shown in Figure 4).At the end of each video clip, students are asked to answer related questions.

Instant feedbacks
Upon students' responses to the questions, detailed explanations and feedbacks are provided to help students review the learned materials.

Instrumentation
In order to measure students' earth science learning outcomes, the students were both pre-and post-tested using the Geological Achievement Test (GAT) and the Attitude toward Geology Survey (AGS).Both the test and survey were constructed and developed by the researchers.
The GAT is a 27-item multiple choice test measuring students' geological knowledge.Each item counts for 1 point and the maximum score achievable for the GAT is 27 points.A panel of specialists in the field of earth science validated the contents of GAT as an appropriate test to measure students' knowledge on the 10 th grade geology.A reliability coefficient of 0.689 was obtained which is considered an acceptable reliability of the test.
The AGS is an 11-item survey that measures students' overall confidence, selfefficacy and attitudes toward geology learning.For the purpose of this study, only three items that were representative of students' attitudes toward geological learning were used.The survey was a 4-point Likert Scale, with 1 indicating strong disagreement and 4 indicating strong agreement.Therefore, students' attitudes toward geological learning could range from 3-12.The overall reliability of the survey was 0.833 derived from KR-20.In particular, the reliability of the attitude section was 0.706.

Participants and procedures
The study was conducted in a secondary school in Taipei, Taiwan.The total participants of this study were comprised of 202 senior high school students, 103 males and 99 females, who were enrolled in grade 10 earth science classes.Students were, on average, approximately 17 years of age.
The design of the study was quasi-experimental.The pre-and post-tests were identical and each required 45 minutes to complete.First, a regular grade 10 geology instruction was given to all of the participants followed by a pretest.Next, the participants were divided into 4 groups: a).RPG group with a pre-announced posttest, b).RPG group with an unannounced posttest, c).No RPG group with a preannounced posttest, d).No RPG group with an unannounced posttest.The two RPG groups were both given the game to be played at home.One of which had a preannounced posttest while the other one was left unannounced.On the other hand, the two No RPG groups were not given the game and again one had a preannouncement of the upcoming posttest whereas the other one was unannounced.A posttest was administered to all participants two months later.
The data were analyzed using 2x2 MANCOVA and ANCOVA.Inferential statistical analyses were undertaken using SPSS 13.0 (Statistical Package for Social Sciences Version 13.0).

RESULTS
The descriptive statistics of students' pretest and posttest scores on the achievements of and attitudes toward geology are summarized in Table 1.
A 2x2 MANCOVA was computed on the vector of posttest scores with students' vector of pretest scores as the covariates to determine whether or not interaction effect between RPG and anticipation of test existed.As shown in Table 2, the results of the MANCOVA test revealed that there was no interaction effect between RPG and the anticipation of a test, F(2, 195) = 1.700, p = 0.185, h 2 = 0.017, f = 0.13.The results also indicated that no significant main effect was found for the treatment of RPG, F(2, 195) = 1.971, p = 0.142,  2 = 0.02, f = 0.14.However, significant main effect existed for the anticipation of a test, F(2, 195) = 8.676, p = 0.000,  2 = 0.082, f = 0.30.The effect size index (f) was calculated from eta square ( 2 ).According to Cohen's rough characterization (Cohen 1988, p. 284-288), f =0.1 is deemed as a small effect size, f =0.25 a medium effect size, and f = 0.4 as the large effect size (for interpreting  2 , 0.01= small, 0.059= medium, and 0.138= large effect size).
Furthermore, the study conducted a 2x2 ANCOVA on posttest scores with the pretest scores as the covariate on the achievement and attitude measures respectively.The results on Table 3 & 4 showed that there were significant main effects, both on students' achievement and attitude measures between students' who anticipated the posttest and those who did not anticipate the posttest (Achievement: F(1, 201) = 8.299, p = 0.004, h 2 = 0.041, f = 0.21; Attitude: F(1, 201) = 10.475,p = 0.001,  2 = 0.051, f = 0.23).
Table 5 summarizes the adjusted posttest means and standard deviations in the achievements and attitudes between the students who anticipated the posttest and those who did not.

DISCUSSION AND IMPLICATION
Since games have been part of growing up in many people's lives, they have become an important part of our social and cultural environment (Oblinger, 2004) and thereby precipitate changes in the field of instructional design.This study used a self-developed instructional game (RPG) and the anticipation of test to examine their impacts on students' academic achievements and attitudes in the field of geology.
The results of this study showed that there was no interaction effect of RPG and anticipation of test on the 10 th graders' achievements of and attitudes toward geology.In addition, students who played the RPG had no significant difference in their achievements and attitudes in geology than those who did not play the game.However, the students who anticipated the upcoming posttest scored significantly higher in their geology posttest and held better attitudes toward geology than those who did not anticipate the posttest.Based on the result of this finding, it can be concluded that testing has positive effects as a reinforcement to help students both retain their content knowledge and have positive attitudes towards geology.Therefore, it is suggested that students are informed in advance that they will be tested because this information may have significant effects on their learning outcomes.
As aforementioned, the aim of this study was to hope that through the fun and enjoyment of the self-developed instructional RPG, students could engage in an active and effective learning process even when they were not aware of an upcoming test.However, based on the result of this study, the effectiveness of RPG as a supplementary reviewing tool for grade 10 geological instruction fell short of researchers' expectations.Although several studies as stated in the literature suggested that students learned effectively with instructional games (Chen, Wang, & Lin, 2015;Hwang, Wu, Chen, 2015;Serrano, Martinez-Ortiz, & Fernández-Manjón, 2015;Papastergiou, 2009b;Rosas et al, 2003;Virvou, Katsionis, & Manos, 2005), the results of this study did not echo those findings.The rationale behind the reason why the RPG did not significantly lead to students' better learning outcome, both cognitively and affectively, in geology than the non-RPG approach is therefore worth exploring.Some research studies have found that electronic game attributes, such as task difficulty and interactivity, are major factors which impact learning outcomes in game-based learning environments (Belanich, Sibley & Orvis, 2004;Ahlers & Driskell, 2002).Therefore, for an instructional game to demonstrate its effectiveness in learning, the game should present an optimal level of task difficulty and interactivity to the learners.While re-examining the RPG adopted in this study, through a followed-up classroom observation of how well students learn in a RPG environment, one of the teachers indicated that the game was very interactive; however, the tasks of the game might have seemed too easy for the learners.As a result, to the researchers' speculation, it might have led to students' less positive learning outcomes in this study than those of other studies in the literature.Based on several research studies, instructional games, if not designed within the optimal level of difficulty, may result in reduced motivation (Malone et al., 1987;Paas, Tuovinen, van Merrienboer, & darabi, 2005;Provenzo, 1991) which thereby may eventually lead to diminished knowledge acquisition and retention (Colquitt, Lepine, & Noe, 2000;Mathieu, Tannenbaum, & Salas, 1992).As suggested by Vygotsky (1978), learning is most effective when it takes place in the zone of proximal development.That is, "the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance, or in collaboration with more capable peers" (Vygotsky 1978, p.86).
Therefore, to conclude, as suggestions for future instructional game designs, the optimal range of task difficulty of instructional games is within the zone that is not too easy or too difficult to the learners.When students find the games challenging yet are not beyond their capability to accomplish through the guidance of the teachers or more capable peers, they can avoid boredom and the possibility of frustration when the tasks are too difficult.Ultimately, the learning outcomes of students in a game-based environment can therefore be further enhanced.

Figure 2 .
Figure 2. Screenshots of field trip simulations

Table 1 .
Descriptive statistics of students' pretest and posttest scores on the achievement and attitude by treatment

Table 2 .
Multivariate analysis of covariance summary for the effects of RPG and anticipation of test on students' geological achievements and attitudes

Table 3 .
Effects of RPG and Informed Post-Test on students' geology achievement

Table 4 .
Effects of RPG and Informed Post-Test on students' attitudes toward geology

Table 5 .
Summary of adjusted posttest means and standard deviations of pre-announced and unannounced posttest on students' achievements and attitudes in geology