An Empirical Study on the Effectiveness and Configuration of Multimodal College English Listening Teaching System

Based on the theories of the multimodal pedagogy, this article focuses on the evaluation of the teaching effectiveness of multimodal college English listening teaching and the configuration principle of its multimodal system. Considering the lack of empirical research on multimodal college English listening teaching in China, the research applies empirical study to compare the differences between the experimental group and the control group to find out the effectiveness of the multimodal college English listening teaching. After the one-year experiment, the subjects of experimental group and control group, who were basically similar in their English achievement and English listening ability before the experiment, manifest significant differences in both English test and English listening test. Superior characteristics are reflected by the experimental group received multimodal college English listening teaching in two test points compared with the control group, which proves the feasibility and effectiveness of the multimodal college English listening teaching system. Besides, the following interviews on the subject of the experimental group, combined with the multimodal pedagogy theories, provide the possible configuration to ensure an effective, dynamic and three-dimensional multimodal college English listening teaching system.


INTRODUCTION
In the context of rapid development and popularization of multimedia technology, it is widely acknowledged that ICT is important in the teaching and learning process (Bingimlas, 2009) and the multimodal discourse has significantly attracted more and more attention in the process of language communication.The dynamic, threedimensional and interactive multimodal texts in realistic human language are constructed by means of different modal symbols embodied in various media means.The phenomenon of multi-modality in language communication is concomitant with the languages, and more and more realistic features of various semiotic patterns interweave with each other.The concept of multimodality in linguistics originates from the paper "Rhetoric of the Image" by R. Barthes in 1977, which discusses the role of visual images in communicative discourse.Multimodality refers to "a combination of different semiotic modes -for example, language and music -in a communicative artifact orevent" (van Leeuwen, 2005), or is defined as "the diverse ways in which a number of distinct semiotic resource systems are both co-deployed and co-contextualized in the making of a text-specific meaning" (Baldry Thibault, 2006).In the field of applied linguistics, New London Group first proposed the concept of "multimodal teaching" in 1996 based on the theory of multimodality.Accordingly, many empirical studies were carried out on the multimodal teaching in the circle of western linguistics.Kress & van Leeuwen (2001) analyzes the role of nonverbal symbols in language teaching, and summarizes the feasible teaching methods of multimodal teaching.By discussing the design of whiteboard in teaching, Jewitt (2006) analyzes the relationship between multimodal teaching and modern media technology.Royce, T. (1998) focuses on the relationship between images and text in multimodal discourse, and analyzes the coordination relations of various symbolic modes in language teaching.Baldry & Thibault (2006) further explore the auxiliary role of multiple symbolic modes in language teaching.Affected by the concepts of multimodality and multimodal teaching in western linguistics, the Chinese linguists expand their research on multimodality, such as Yueguo Gu, Zhuanglin Hu and Delu Zhang.They discuss the discourse features of multimodality and the methods of multimodal teaching, especially multimodal English teaching from various angles.
At present, College English courses, which are included in the curriculum as a compulsory course by most of the Chinese colleges, have adopted multimedia information technology as the teaching method.Some Chinese colleges have applied the multimodal college English teaching in form for more than ten years.But in the practical college English teaching, teachers still work as the center and primary subject in teaching activities and students receive the contents passively and mainly in single modal form.Generally, the multimodal factors are not fully utilized in Chinese college English teaching activities, and the multimodal English teaching manifests the trend of formalization under the condition of multimedia technology.There is no doubt that multimodal college English teaching ensures the authenticity of the data, context and interaction in a certain extent, and simulates language communication in real life.Therefore, existing multimedia resources should be fully made use of, and the interaction, complementarity and mutual promotion between the multimodal symbols be fully mobilized in teaching to achieve "the ecological integration of the real network, three-dimensional, multimodality between the content and the media in the College English teaching" (Liu, Ming. & Hu, Jiasheng, 2011).The traditional teaching modes influence college English listening teaching greatly, and naturally reject the multimodal teaching method, which explains the reason why there is few empirical research on multimodal college English listening teaching in China.This paper will use empirical research methods to explore the implementation ways, effectiveness and multimodal configuration of multimodal college English listening teaching in China, and find out the possible teaching methods of multimodal college English listening teaching.

LITERATURES REVIEWING
According to the basic principles of multimodal teaching approach combined with the means of human communication, classroom teaching should include "semiotic spaces in which human beings who are the agents of their own meaning making produce multimodal texts -visual, written, spoken, performative, sonic, and gestural" (Stein, 2000).In the process of college English classroom teaching, a variety of modes cooperate with each other to achieve the goal of teaching.Among them, the main mode is spoken, and other modes supplement, strengthen and optimize the former (Zhang, Delu. & Wang, Lu, 2010).But in college English listening class, either from the perspective of textual length or written discourse length of the listening materials, the sonic modality overweighs the teacher or the peers spoken modality in the auditory modal discourse received by the students.Thus, it comes to the conclusion that multimodal college English listening teaching is independent of multimodal college English teaching and has its unique modal factors constructive system.And the proportion and collaborative structure of visual, written, spoken, performative, sonic, and gestural modalities should vary according to the different teaching contents and teaching goals.
Under the condition of multimedia technology, all modal forms except the performative modality are generally presented in college English listening teaching.In combination with teaching participants, textbooks and contexts, the modal system of multimodal college English listening teaching is as follows:

Contribution of this paper to the literature
• The present study applies empirical study to compare the differences between the experimental group and the control group, while it lacks of empirical research on multimodal college English listening teaching in China.
• Experiments conducted in this study demonstrate the feasibility of multimodal college English listening teaching and its effectiveness in improving college students' listening ability and overall English proficiency.
• With the tentative allocation of the multimodal factors in English listening teaching class, the possible configuration of the teaching system is provided in the study to ensure an effective, dynamic and threedimensional multimodal college English listening teaching system.
In the multimodal context, teachers and students act as the participants in college English listening teaching activities.In this process, the modalities involved mainly come from teachers, students, peers and textbooks.In view of the actual operation forms of multimodal college English listening teaching, the synergism of sonic, spoken, written and gestural modalities show in an unbalanced way, which play different roles in the listening class.The sonic modality from the audio materials and textbooks and the spoken modality from the teaching participants' discourse regardless of listening teaching paradigm and teaching methods are regarded as an important part of listening teaching activities.In comparison, the visual modality derived from video materials and textbooks is usually placed in secondary or complementary position in traditional listening teaching.The position and function of the written modality derived from textbooks and slides, and the gestural modality derived from the participants in the teaching activities are greatly influenced by the directors during the teaching process, because of their indirect relationship with the overall teaching objectives.The performative modality stemmed from the receivers is limited by the teaching time and rarely appears in the classroom of college English listening.The traditional listening teaching relies more on textbooks.The teachers play the audio materials, and guide students to do the listening exercises orally, which mainly relate to the written, spoken and sonic modalities, and seldom or do not utilize the sonic and the gestural modalities, while the multimodal listening teaching advocates the use of textbooks, audio materials, video materials, slides, teacher talk, teacher postures and other modal resources to promote the improvement of students' listening ability.This research attempts to explore the effectiveness of college English listening teaching, and the way to configure the various modal resources to achieve the optimal allocation when different modalities coexist, and the various semiotic modes are fully mobilized in college English listening class.

RESEARCH DESIGN
The research subjects were selected from the undergraduate freshmen who entered the college in 2016.Before they began their study in the course of college English, they were put into two levels according to their English score in the College Entrance Examination.About 20% of the top students entered the advanced class, and other students took the college English course in a class divided by their majors.The subjects of this experiment are two regular classes randomly selected.One class is selected as the experimental group at the beginning of the teaching activities, and the other is chosen as the control group.The experimental group consists of 53 subjects, and the control group consists of 56 subjects.The experimental group and the control group are similar in the natural features of the subjects.They are both made up of four natural classes from two colleges, two of which come from the major of arts and the other two of which belong to the circle of science.In order to ensure the scientificity and rationality of the subject selection, all the subjects' English scores in their College Entrance Examination are first collected and input in the software SPSS 19.0 for statistical analysis, and the analysis results show that the research subjects do not have a statistically significant difference in their English scores in College Entrance Examination.Considering that there are differences in form and content of the English College Entrance Examination in different regions of China, the listening ability achievement test were carried out on the preselected subjects at the beginning of the experiment.The materials of the listening test are from the audio materials of listening part in the past CET 2 (College English Test Band Two).The listening test takes the form of multiple-choice questions, and the materials include short dialogues, long dialogues and long passages.After obtaining the scores of two subject groups, the results were input to SPSS 19, and the independent sample T test was carried out.The results were as follows: The results show that there is no significant difference between the experimental group and the control group in their listening performance.Therefore, the members of the two classes are identified as the subjects of this experiment.
After the research subjects were selected, the college English listening teaching activities were conducted in the two groups of subjects.The experiment lasted for one year, two semesters.The teaching time of listening was 24 class hours per semester.The teaching activities for the two groups of subjects were carried out in the multimedia classrooms.The listening teaching for the control group strengthens the traditional modal symbols, and gave great emphasis to the sonic modal materials in the textbooks assisted by the necessary spoken modality from the teacher.At the same time, interactive symbolic system of three-dimensional dynamic multimodal listening teaching was applied to the experimental group in the teaching process.The selection of textbooks follows the principles of combining the video and audio materials, strengthening the spoken modality and stressing the synchronic presentation effect of gestural modality to the spoken modality, in which the written modality was introduced when it was necessary.So a variety of potential modal forms has been fully applied in multimodal listening teaching to explore different modal functions of the various modal symbols in the whole teaching system.
After the teaching stage, the overall performance test was carried out for the two groups of subjects to compare the achievement differences among the two groups.The test papers adopted are the past CET 4 (College English Test Band Four).Then the scores of the listening parts were extracted, and the subjects' differences in listening performance were compared.In view of the CET 4 relating only to the audio materials, this experiment has added video materials to examine the two groups' video materials analysis ability.The subjects took the test near the end of the experiment in class.The video material in the paper lasted for 5 minutes, selected from the video materials of the past Computer-based College English Test Band Four paper.Students were required to watch the video twice, and to complete 5 multiple choices based on the understanding of the video, and then finish 15 cloze tests without watching the video.The results were input into SPSS19.0for statistical analysis and the reasons for the specific statistical differences were explored under the consideration of the multimodal teaching theories.
Finally, in-depth interviews were conducted for 10 subjects randomly selected from the experimental group, and the interviews were taken face to face.The interviewees were required to evaluate in a five-point scale from four aspects, which include the overall evaluation of multimodal college English listening teaching, evaluation on the modalities from the teachers, video materials and slides, and the selection and the combination of video and audio materials (each subdivided into 5 specific issues) according to the established plan and unified design contents.And each interviewee is interviewed for each specific issue to find out their unique opinions as illustrations to their endowing marks in the questionnaires.From the results of the interviews, the experimental group's attitude towards multimodal college English listening teaching was obtained.

Experimental Results
Compared with the pre-tests, the number of subjects in the posttest has changed.In the course of the experiment, due to the reasons of dropping out of school, transferring the major and taking out of school, the data from 49 subjects of the experimental group were received, and 55 subjects from the control group contributed the experimental data.In addition, as for the data acquisition of video material understanding has taken the form of classroom tests, 1 subject in the experimental group did not complete the test due to the personal leave, and 48 samples were obtained eventually.After the experiment was completed, the independent sample T test was conducted for the two groups of subjects in terms of overall English scores, listening scores and video listening scores.The results of the comparative analysis were as follows: The statistical results in Table 2 show P=0.000<0.05,indicating that the overall score of the two groups of experimental subjects appear statistical difference.Table 2 also shows the comparison results of the two groups of subjects in the listening part.P=0.015<0.05,indicates the statistical difference between the experimental group and the control group in their listening performance.But the statistical results of Table 2 show that the comparative analysis result of the video listening part P=0.053>0.05,so it can be inferred that the data in this aspect are not statistically different.When the independent sample T test was carried out, the sampling number was increased to 1000 in the method of Bootstrap simulation sampling statistical estimation.The results are omitted, since they are consistent with the previous independent samples T test results.
The results show that the subjects of experimental group and control group, who were basically similar in their English achievement and English listening ability before the experiment, manifest significant differences in both English test and English listening test.Superior characteristics are reflected by the experimental group received multimodal college English listening teaching in two test points compared with the control group, and the preliminary conclusion can be reached that the strengthening of coordination, complementarity and interaction of multimodal symbol system in college English listening teaching is of help to the improvement of college English learners' overall English ability and English listening ability.The data of video listening in the posttest show that despite one-year multimodal college English listening teaching, the experimental group did not perform better than the control group in analyzing the materials combining the videos and audios, but it cannot undermine the fact that the experimental group did better than the control group in their overall performance.

Interview Results
Besides, the following in-depth interviews of 10 subjects from the experimental group further reveal the learners' attitude towards the multimodal college English listening teaching.The results are shown in Table 3.Since the interviewees are limited in number and the interview is carried out face to face with the possible feature of subjectivity, the mean score of three main interviewed items are listed and the specific issues are omitted as a supplementary explanation to the results of the experiment.
All the results of the three main items of the interview are above 3 point (the full score is five points), that is, beyond the passing line.But the specific scores vary with respective item, which indicates the different attitudes of the interviewees towards the different factors in the classroom of multimodal college English listening teaching.The results of the experiment and the interviews will be discussed in detail and the explanation of the differences will be explored.

Experimental and Empirical Findings
The results of the experimental data verify the feasibility of multimodal college English listening teaching.P of statistical result for the total score is 0.000 and P of statistical result of listening score is 0.015, both of which are less than 0.05.At least we can conclude that the multimodal college English listening teaching is effective in two measured items.In the multimodal teaching process, the teachers' reasonable allocation of multimodal factors in the classroom can effectively promote college English learners' English learning.This is not only reflected in the improvement of learners' listening ability, but also can effectively improve their overall English ability.In order to further clarify college English learners' English learning experience in the multimodal listening teaching, and find out the causes of difficulties in the analysis of video materials, the in-depth interviews of the experimental group subjects were analyzed and collated after the experiment.The interviews showed that all the interviewees held that multimodal college English listening teaching is helpful to improve their English listening ability, especially in providing authentic language input, creating a good learning environment and cultivating listening skills.When the modal factors involved in the process of teaching were inquired, the interviewees believed that teachers cannot be ignored in the multimodal college English teaching.According to the interviews, the teacher's spoken moral discourse which can effectively guide the listening learning process was noticed.Among them, they especially mentioned the importance of explanation and guidance relevant to the content of listening materials, and the appropriate change of tone and intonation.Secondly, the teachers' gestures and body language also received some attention.Comparatively speaking, college English learners paid little attention to teachers' language pacing, facial expression and eye gaze, and thought that this series of factors related little to their learning experience of college English listening class.
In the written modality, college English learners widely accepted textbooks and slides as the two main modes in college English listening class, and placed the understanding and generating of written discourse relevant to the textbooks in a more important position.The interviewees unanimously agreed that it is necessary to involve the college English textbooks for listening training in the process of multimodal college English listening learning.In addition, interviewees believed that the written modality of slides should be strengthened in listening class to match the contents of the teaching of college English listening training, such as the diverse presentation of the slides.In contrast to the pure written modal text, the interviewees indicated that the illustration of written modalities like the pictures related to the content of the listening material contributed to their understanding of the relevant listening materials.Comparing with the above factors, interviewees believed that slide layouts, background pictures, and background music were not conducive to their understanding of listening materials.At the same time, some of the interviewees also mentioned their expectation for the introduction of video materials in slides.And the basic principle for the introduction of the video materials should go beyond the scope of interest, and combine with the content of listening training to promote English listening learning.
All interviewees expressed that the multimodal listening materials combining both audio and video materials can improve their listening ability, compared with the single modal audio materials.It is especially effective when it is helpful to predict the background of listening materials and being real-time.The multimodal listening materials are more acceptable for college English learners since they have obvious advantages than the single modal listening materials.But meanwhile the interviewees also suggested that in some cases visual modality would interfere with the understanding of sonic modal utterances.This partly explains why there is no significant difference between the experimental group and the control group in the video listening part of the posttest.Some of the experimental group subjects interviewed thought they "mainly depended on listening without paying attention to the content of the video because of the tension in the test", or "paid too much attention to the video content, and missed some sentences", resulting in the wrong answers.The results of this experiment and the accounts from the subject of the experiment also corroborate the experiment by Gruba (1993), in which there were no significant differences between the two groups of subjects when they were tested with audio materials and audio-video materials about academic lectures.But the total score of the experimental group is higher than that of the control group, which confirms that the experimental group is better than the control group in the analysis of the video listening materials, but the advantage is not obvious.

Possible Explanations and Suggestions
In view of the experimental results and the results of in-depth interviews with the experimental group subjects, it is feasible and effective to adopt multimodal teaching mode in college English listening teaching, and it can effectively improve college English learners' listening ability and English level.In the aspect of auditory modal configuration, the teachers should adhere to the central principle of auditory modality.From the perspective of educational process implementer, the auditory modality in college English listening class includes sonic modality and teachers' spoken modality.In the auditory modal configuration, the real-time, diversity and difficulty of listening materials should be considered combined with the college English learners requirements.At the same time, teachers need to fully mobilize their spoken modality in the three teaching stages of before-listening, inlistening and after-listening respectively, offer necessary guidance for students in the related background of the listening materials, key points and standard answers, and ensure the perfect complementary combination of spoken modality and sonic modality.For example, before listening, the teachers can introduce the background knowledge about audio materials to the students with no direct hints involved.At the in-listening stage, the key points and difficulties of the audio materials can be mentioned to the students, the key words will be explained in detail, and the students are guided to make clear the specific content of audio materials.After listening, a proper summary of the material can be made.The relevant content can be appropriately expanded to deepen students' understanding of audio materials after the answers are clarified to the students.Teachers should pay attention to the changeable factors such as pitch, intonation and other spoken modes in the spoken modal invocation, so that it can be more conducive to the teachers' active spoken modal guidance to students.While applying the spoken modality, teachers should give students more opportunity for their spoken modal output to expand their memory capacity for more knowledge and durability and to realize the interactive multidimensional auditory transmission.
According to psychologist Treieher's experiment, human beings obtain a variety of sensory information, 83% from visual stimulation, 11% from auditory stimulation, while the amount of information combining both auditory and visual stimuli can reach 94%.From the persistence of memory, people can memorize 10% of reading information, 20% of hearing, 30% of seeing, 50% of hearing and seeing, and 70% of personal spoken information in the process of communicative interaction (He, Kekang. & Li, Wenguang, 2009).Thus, it is necessary to introduce video materials into multimodal college English listening teaching.Firstly the selection of the video materials should adopt the principle of the maximum correlation with the audio materials.Also, the audio-video materials can be used to replace the original audio materials of the similar content, length and difficulty, or the video materials work as the supplements to the audio materials to strengthen the students' understanding to the background information of the audio materials.The selection of video materials should be detached from the category of interest, and in correlation with the audio materials.The amount of information should be focused on, and the audiences' full range of sensory stimulation and experience should be stressed in the process of combining the audio and video materials.But at the same time, it is a key to balance the amount of video materials, and adhere to the center principle of the auditory materials in multimodal college English listening class.
When the sonic modality, the spoken modality and the video modality are reasonably coordinated, it is also appropriate to use the written, gestural and other modal symbols in the multimodal college English listening class.According to the feedbacks from the interviewees, their recognition of written, gestural and other symbolic modes is relatively low in the multimodal college English listening learning.As the necessary modal symbol forms of college English listening teaching system, the use of written and gestural modalities in the whole modal system should follow the principle of necessity.They can effectively assist the sonic and visual modalities under the condition of proper use.When they are optional, the use of them should be limited to reach the minimization.The choice of modal symbols is based on the teaching contents and teaching goals as the foundation, in order to achieve the best teaching effect.Teachers tend to use a variety of semiotic modes in order to achieve the best teaching effect, but from the economic point of view, the selection of modalities should be simpler (Zhang, Delu, 2010).For example, the written modality should combine with the audio-video content related to the teaching.The relevant pictures can be inserted to increase the diversity, but other factors such as the slide layout, background picture or background music that are useless for the learners' listening comprehension of materials can be abandoned altogether.

CONCLUSION
Experiments conducted in this paper demonstrate the feasibility of multimodal college English listening teaching and its effectiveness in improving college students' listening ability and overall English proficiency.Under the premise that multimedia technology is widely adopted in college English teaching at present, it is possible to launch multimodal college English listening teaching.College English teachers should concern the actual classrooms, and avoid the superficiality of the potential forms of multimodal factors in college English listening teaching.The various multimodal symbols should be reasonably allocated in college English listening course, and the possible modal configuration of college English listening class can be explored to realize the multidimensional discourse parsing of the single modal discourse and the dynamic multimodal discourse coordination.Although the experiment has achieved better effects, the participants of the experiment are limited by the number of the classes instructed by the author, and the experiment does not cover all the college English learners of all levels.Besides, the analysis of multimodal college English listening teaching system still has the potential of not being exhausted.
Although the sampling number was increased to 1,000 in the method of Bootstrap simulation sampling statistical estimation to estimate the total index in the statistical analysis of experimental data, the samples are still insufficient according to the method of maximum likelihood estimation in statistics.Therefore, in the premise of the effectiveness of multimodal college English listening teaching, the experimental results will be further spread.After the scope of the experimental subjects are expanded, the similarities and differences between the college English learners of different levels reflected in multimodal college English listening teaching will be further compared, in order to obtain the optimal and specific modal configuration of different modal symbols in the multimodal college English listening system.

Figure 1 .
Figure 1.Modal system construction of multimodal listening teaching

Table 1 .
Independent sample test results of listening performance of the experimental group and the control group before the experiment

Table 2 .
Independent sample test results of experimental group and control group in posttest

Table 3 .
Results of in-depth interview (measured in five-point scale)