A Network Meta-Analysis on the Effects of Information and Communication Technology on Students ’ Learning Achievement in Taiwan

Computer technology has been used in classrooms for many years and yet many educators still do not understand its effectiveness. In addition, with the Internet technology becomes increasingly mature and more popular, instructors of multimediabased CAI are now facing the issue of whether or not they should upgrade their technologies under limited budget. In this study, a network meta-analysis was performed to synthesize existing research comparing the effects of different types of CAI versus the traditional instruction (TI) on students’ learning achievement in Taiwan. 34 studies obtained from Taiwan’s National Digital Library of Theses and Dissertations were reviewed systematically, and their quantitative data was transformed into effect size (ES). The results of this study demonstrate that CAI is more effective than TI in students’ learning achievement in Taiwan. Furthermore, the effect of CAI increases as information technology improves and the effect of Internet-based CAI is better than that of multimedia-based CAI. To improve the effect of CAI, this paper suggests that schools use Internet-based CAI instead of multimedia-based CAI.


INTRODUCTION
The computer plays an important role in instructional technology and learning.Computer-Assisted Instruction (CAI) is regarded as advanced development of human technology, and it is characterized by customized individualization, remedial teaching, and mastery learning.CAI is classes using computer-assisted instruction or computer-assisted learning software as replacement of or supplement to traditional instruction.Traditional instruction (TI), on the other hand, is classes using old-fashioned teaching methods.A traditional course requires face-to-face instruction that comes from lectures and activities via teacher's dictating and expressions in a real classroom, and course materials are delivered in the form of books and other written resources.Compared with TI, CAI is more capable of capturing students' attention because the programs are interactive and can increase students' motivation to compete with one another.
In recent years, because the information technology has become more mature and the Internet more popular, the application of Internet not only occurs in electronic commerce but also in schools.Some universities are even equipped with virtual classrooms or e-learning system to facilitate web-based distance learning.To sum up, the development of CAI has developed from multimedia-based CAI (or computer based CAI) to the Internet-based CAI (or web based CAI) (Liao, 2007).
Multimedia-based CAI is the combination of text, sound, video, graphics, and animation with links and tools that allow the teachers and students direct, interact, and communicate with computers (Natarajan, 2006).Multimedia applications incorporate a full range of available facilities to enhance the communication between teachers and students.

/ 10
Internet-based CAI is defined as the creation and proliferation of the personal computer, the globalization of ideas and other human acts, and the use of Internet in exchanging ideas and providing access to more people (Natarajan, 2006).Internet-based CAI is the application of a computer network to present or distribute educational content.Audio, video, computer and networking technologies (such as Internet) are often combined to create a multifaceted instructional delivery system.
However, Liu (2001), Chen (2002), andLai (2002) found no significant differences between CAI and TI.Liu (2001) found that CAI can make learning interesting, but it is not helpful for students' learning achievement.He claimed that the effect of TI is better than that of CAI in basic knowledge, because teachers' assistence is more important than making learning fun and interesting (Liu, 2001).Students' learning achievement is not increased with their interests in learning but increased with learning motivation (Chen, 2002;Lai, 2002).
As computers and the Internet grow in popularity, CAI as an educational technology integrating many disciplines has gradually become an essential method for modern instruction.Therefore, how to upgrade from multimedia-based CAI to Internet-based CAI in a cost-effective way is an important issue for many schools.
Meta-analysis is a statistical process whereby the findings of a number of studies, focusing on a common problem or topic, are pooled in an effort to draw inferences as to the meaning of a collective body of research (Hannafin, Hannafin, Hooper, Rieber, & Kini, 1996).Early meta-analysis studies of CAI were published prior to the microcomputer revolution in 1970.In an effort to provide data to this debate, this study performed meta-analysis of CAI versus TI in Taiwan's elementary schools.
Traditional (pairwise) meta-analysis pools data from multiple trials that compare only two treatments.Network meta-analysis, on the other hand, is a new statistical technique for analyzing treatment effects when there are more than two possible interventions, with trials comparing different combinations of these interventions.It uses both direct comparisons from published trials and indirect comparisons that can be inferred by the results of other comparisons.
The purpose of this study is to synthesize and analyze the research results of the effects of information and communication technology on students' learning achievement in Taiwan.Unlike other studies that compared CAI with TI, this study went further to compare different types of CAI (multimedia-based CAI and Internet-based CAI) with TI.Thus, the result of this study can show the different levels of effectiveness between multimedia-based CAI and Internet-based CAI.

MATERIALS AND METHOD
This study entered data from National Digital Library of Theses and Dissertations in Taiwan into the research data in this study, and did meta-analyses using the inverse-variance method for heterogeneity.A separate record was produced for the 34 individual two-by-two comparisons, and used for the traditional meta-analysis, metaregression, and network meta-analysis.

Research Data
34 studies that compared the information technology application, or CAI with TI on students' learning achievement were used in this study.They were all obtained from Taiwan's National Digital Library of Theses and Dissertations (Taiwan's National Central Library, 2017).

Contribution of this paper to the literature
• The results of this study suggest that the effect of CAI is better than that of the traditional instruction (TI).It can be inferred that the gap between the effect of CAI and that of TI will become wider as information technology continuously evolve.
• Internet-based CAI is more helpful than the multimedia-based CAI for students' learning achievement.
• For schools that still use TI, this paper suggests that they build CAI system to improve the effectiveness of students' learning.For schools that are using multimedia-based CAI, it is important for them to upgrade from multimedia-based CAI to Internet-based CAI in order to get more benefits.

Data Analysis
Network meta-analysis is a generalization of pairwise meta-analysis that compares all pairs of treatments within treatments for the same condition (Rücker, 2012).The network meta-analysis used in this study is the application of the program designed by Rücker (2012).
Based on Rücker's study (2012) and Schwarzer, Carpenter, and Rücker's study (2015), there is n different treatments (nodes) in the network for the same condition and let m be the number of pairwise treatment comparisons.There is at most n-1 independent treatment comparisons, but the model has a parameter (θ treat ) for each of the n treatments.Thus the matrix X is not of full rank, so its inverse does not exist, and the weighted least squares estimates of θ treat cannot be obtained directly.
The network structure of different treatments and pairwise treatment comparisons is defined by the design matrix X which is an mn matrix.If it only includes two-arm studies, each row corresponds to a study and m is the number of studies.There is a "1" in the column that corresponds to the first treatment and a "-1" in the column that belongs to the second treatment.The sum of each row of X should zero.
As previously mentioned, X T X is not of full rank, so to estimate the treatment effects is constructed the Moore-Penrose pseudoinverse matrix.The nn Laplacian matrix L is defined as where W is a diagonal matrix of dimension mm whose diagonal elements are the inverse variance study weights ( 2 ).The Laplacian matrix L, has rank n-1 and is not invertible.However, its Moore-Penrose pseudoinverse L+ is defined and can be calculated by where J is the nn matrix whose elements are all 1.
With L+, the estimation of the fitted values θ �  can be calculated as Equation ( 3) where H is known as the hat matrix in regression.
Equation (3) means that the elements of θ �  are linear combinations of the elements of θ � with coefficients coming from the rows of H.Each network estimate is constituted by the observed estimates, weighted with the elements of the corresponding row of H. Accordingly, the elements of this row of H are interpreted as generalised weights.
The hat matrix H, is a projection matrix which maps θ � onto the consistent (n-1)-dimensional subspace.This gives the fitted values θ �  , which can be interpreted as the values that minimize the quadratic form After fitting the network meta-analysis model, for any treatment comparison, the estimate of the treatment effect be calculated with the direct evidence, and each piece of indirect evidence.The variance of the resulting treatment estimate is estimated by where Vij denotes the variance of the resulting comparison of treatments i and j.
Each p-arm study contributes p-1 degrees of freedom to the total Qtotal statistic.The total degrees of freedom are given by the sum of the degrees of freedom contributed by each study minus n-1.Denoting this by df, a generalized I 2 statistic can be defined as A simple random effects model can be defined using the estimate of a common heterogeneity variance τ 2 for each pairwise treatment comparison.For multi-arm studies, the estimate τ � 2 is added to the observed variance of each comparison before reducing the weights.Network meta-analysis is applied to the same observed treatment differences, now using the enlarged standard errors, as in standard pairwise meta-analysis.The estimate of τ 2 is Lai / CAI and Learning Achievement 4 / 10

Outcome Measures
The outcome of CAI measured in these 34 theses was students' learning achievement.A meta-analysis was performed to synthesize existing research comparing the effects of information technology application and traditional instruction on students' learning achievement in Taiwan.For statistical analysis, outcomes from a variety of different studies with a variety of different instruments had to be expressed on a common scale.The transformation used for this purpose was the one recommended by Higgins, Thompson, Deeks, and Altman (2003).
To reduce measurements to a common scale, each outcome was coded as a standardized mean difference (SMD) that was the learning outcome of the information technology application group (treatment group) minus the learning outcome of the traditional instruction group (control group).
The objective of a meta-analysis study is to compare two groups, such as Treated (referenced as t) and Control (referenced as c).In a meta-analysis study, μT and μC are the true (population) means of the two groups.The population mean difference is defined as and the standardized mean difference which is usually used as the effect size as weighted mean difference (WMD).
This study applied the effect size as standardized mean difference (SMD) with Hedges' g, which is an extension of Cohen's d (Rücker, 2012).Hedges' g was adjusted slightly to give better estimates for analyses of smaller sample sizes.If Ni is large, then adjustment is relatively small and there is little difference between the two methods.The effect size (mean difference) of Hedges' g (Rücker, 2012) is: Standard error Hedges' g is as:

RESULTS
The 34 theses considered for this meta-analysis were obtained from Taiwan's National Digital Library of Theses and Dissertations.The summary of these 34 theses is shown in Table 1.They were all published after 2000.Most of them (55.88%) had a sample size between 51 and 100.24 theses (70.59%) discussed the multimedia CAI, and 10 these (29.41%) discussed web-based CAI.

Students' Learning Achievement
Based on the results of the 34 theses, Figure 1 shows the effects of CAI versus TI on students' learning achievement.The number of cases studied and the effect size in each thesis are presented.In both fixed effects analysis and random effects analysis, 31 (91.18%) of all the effect size were positive and favored the CAI group, while 3 (8.82%) of them were negative and favored the TI group.The range of the effect size (Hedges'g) was from -1.43 to 7.85.The overall effect size for all the 34 study was 1.71 (p < 0.001), suggesting that the learning achievement of the CAI group was better than that of the TI group.The great heterogeneity across studies (I 2 ) is 95.8%, and τ 2 is 1.56.This means that the learning achievement of the CAI group was better than that of the TI group.

Year of Publication and Students' Learning Achievement
The publication years of these 34 theses were from 2000 to 2013 (Table 1).With 2000 being the base year, the publication years were coded from 0 to 13.The relationship between students' learning achievement and the year of publication was analyzed by meta-regression, and the result is presented in Table 2.It can be found that starting from the base year, the difference between learning achievement of CAI and that of TI increased.It increased by 0.52 (z-value=2.17;p-value <0.05) each year.

Type of CAI and Students' Learning Achievement
This study also compares TI with multimedia CAI and Internet-based CAI using network meta-analysis.The network of the comparisons is in Figure 2. Of all the 34 studies that discussed the effects of information technology on CAI, 24 (70.59%) of them compared the multimedia CAI with TI, and 10 (29.41%) compared the Internet-based CAI with TI.  Figure 3 shows the network meta-analysis of learning achievement that is based on the comparison with the traditional instruction.The τ 2 is 918.64, and Q is 66462.15 (p-value<.01).It can be found that the random effect of Internet based CAI was the best (SMD=23.21,95%-CI is 4.30~42.12),which was followed by multimedia-based CAI (SMD=19.37,95%-CI is 7.13~31.60).

DISCUSSION
With the meta-analysis, this study found a significant overall effect size (Hedges'g = 1.71) in favor of CAI in students' learning achievement.Therefore, it is apparent that CAI can better increase students' learning achievement than TI in Taiwan.One of the major findings of this study is that CAI undoubtedly can enhance students' learning achievement despite some previous researchers' claim that CAI could only boost students' interest in learning, but not motivation.
The year of publication variable in the meta-analysis allows an assessment of the effect of CAI over time.All of the 34 researches reviewed in this study were published after 2000.This suggests that studies on CAI became popular in Taiwan after 2000.From these studies, it can be found that students' learning achievement (effect size) with CAI grows as time passes.One of the reasons for this might be that teachers' ability in using information increases as time passes.The more experience that teachers have in using information technology, the better students' learning achievement is (Mumtaz, 2006).Another reason might be that new technologies are more helpful for learning than the old ones because they provide more functions for CAI (Utts, Sommer, Acredolo, Maher, & Matthews, 2017).Therefore, the findings of this study suggest that to improve the effects of CAI, schools should provide trainings for teachers to improve their skills in using information technology.
While both multimedia-based CAI and Internet-based CAI are better than TI, Internet-based CAI outperforms multimedia-based CAI in enchancing students' learning achievement.The reason for this is that Internet-based CAI provides more interactive services than multimedia-based CAI (Natarajan, 2006).In addition, students can get access to Internet from anywhere at any time without the restriction of the classroom (Mojtahedzadeh, Mohammadi, Emami, & Rahmani, 2014).

CONCLUSION
The results of this study suggest that the advantage of CAI is greater than that of TI in Taiwan.While many educators are devoting tremendous efforts with great expectation that technology will significantly increase students' learning achievement, this study is able to provide those teachers with an accumulated research-based evidence for positive outcomes of using technology in instruction.
Studies claiming that CAI had no benefits were all published before 2002, and they all discussed multimediabased CAI.This study has shown that the benefits of CAI grows over time as information technology becomes more and more mature and teachers get more and more familiar with it.In other words, the effect of newer CAI, such as Internet-based CAI is better than the older CAI, such as Internet-based CAI.Based on the results of this study, schools that are still using TI are suggested to build the CAI system, and schools that are using multimedia-based CAI should upgrade to Internet-based CAI.CAI has been proved to have positive effects on students' learning achievement in western countries in many previous studies.Despite of cultural differences between western and Chinese classrooms, this present study has shown that CAI also brings benefits to students of Chinese culture countries, such as Taiwan.The findings of this study has provided evidence to suggest that schools should consider not only implementing CAI but also upgrading the functions of CAI to improve students' learning achievement.

Figure 1 .
Figure 1.The forest plot of the 34 theses in this study

Figure 2 .
Figure 2. Network of the comparisons of learning achievement

Table 1 .
Summary of the 34 theses in this study

Table 2 .
The result of meta-regression for the year of publication *: P-value<0.05EURASIA J Math Sci and Tech Ed 7 / 10