Design of teaching aids in STEAM education and fuzzy hierarchical analysis of their educational effect

Focusing on problem-based learning (PBL) and the multi-disciplinary teaching of science, technology, engineering, arts, and mathematics (STEAM), we developed the evaluation tool of teaching aids in STEAM education to help students practice creative thinking and solution-finding. The fuzzy theory and the analytical hierarchy process were integrated for the evaluation of the acceptance of using robots in STEAM education. The fuzzy analytical hierarchy process (FAHP) was developed with the help of experts in education and industry who defined the goal, criteria, and alternatives (factors) of using teaching aids in STEAM education. FAHP was applied to the evaluation of three robots designed for STEAM education in this study. The result of the evaluation showed that the use of the designed robots in STEAM education was important in five criteria (structure, function, economy, aesthetic, and creativity). Among the alternatives, logic, creativity, simpleness, Avant-garde, and innovation were important for robots to be used for STEAM education. This indicated that artistic elements were important in the integration of them in STEAM education. The developed FAHP and questionnaire were useful in evaluating the acceptance of teaching aids in STEAM education. For the evaluation of the use of other teaching aids, the result of this study provides a basis and a reference for how teaching aids can be used for STEAM education.


INTRODUCTION
The development of technology has led science education to focus more on practical training than before.Thus, the design and development of teaching aids such as robots have attracted much research interest in pedagogical research to improve the professional abilities of students and train practical skills in curriculums.Countries in the world are paying more attention to the development of national scientific literacy and the cultivation of scientific and technological talents.In particular, with the integration and implementation of science, technology, engineering, art, and mathematics (STEAM), students are trained to have key competencies and qualities in communication and cooperation, information application, creativity, problem-solving, and self-awareness and regulation.
Students also learn and practice self-reflection and social participation.
To understand the key points and elements of STEAM education, the latest information is necessary for academic achievement, which also draws the attention of the public to the importance of STEAM education.STEAM education has not been fully characterized in terms of its functionality.Therefore, it is required to determine the characteristics of STEAM education to attract more attention from students.Robots can be used for students to learn and integrate technology into their knowledge and to improve learning motivation.Thus, it is important to use robots as teaching aids for science education and practices, especially in teaching product design.Then, students can enhance their professional ability, skills, and competencies for the development of future careers.In education using robots as teaching 2 / 9 aids, the design and production of hands-on products in science education can be learned, which promotes STEAM education more effectively.At present, countries are paying attention to the development of national scientific literacy and the cultivation of scientific and technological talents.They integrate and implement STEAM for cross-disciplinary education to equip students with competencies and qualities in communication and cooperation, information application, creativity, problem-solving, self-awareness and regulation, self-reflection, and social participation.
Problem-based learning (PBL) is regarded as an effective teaching and learning method, especially in STEAM education (Sigit et al., 2022).In PBL, interdisciplinary teaching of STEAM is designed using various activities in related curriculums.The knowledge in the five domains of STEAM is combined among different disciplines to fill the gap between different disciplines.Then, students can try to solve problems based on multiple sources of knowledge, and at the same time, learn science and technology based on logical thinking.In the interdisciplinary framework, students can think about specific topics using different knowledge and viewpoints of each domain.
This study was carried out to evaluate the acceptance of using robots as teaching aids in PBL, especially in P5BL, which stands for project, problem, product, process, and people-based learning.P5BL was proposed by the PBL lab at Stanford University.In P5BL, project planning, problem-solving, product development, product design and manufacturing, and teamwork for self-learning and integration of new knowledge and technologies are important (Han & Yim, 2018).In-depth integration of robotics and mechatronics is emphasized to enhance the cross-disciplinary learning of students and improve their motivation and interest in learning.By increasing students' overall learning effectiveness, the program of P5BL is designed to develop students' capability effectively (Sun et al., 2009).With the rapid development of information technology, various tools and systems for STEAM education are available and used to cultivate innovators who will contribute to enhancing a country's competitiveness.Therefore, it becomes imperative to design and develop teaching aids with intelligent functions for STEAM education.As such aids, robots are useful in PBL to stimulate the logical thinking and creativity of students as they can practice coding a program and assembling and operating a robot.Thus, we evaluated the effectiveness of using robots in STEAM education using a fuzzy analytical hierarchy process (FAHP).With FAHP, we expected to understand the preferences, needs, and subjective judgments of experts in education and in the industry (Sun et al., 2009).The result of this study provides a reference and foundation for developing and using teaching aids in STEAM education.

THEORETICAL BACKGROUND
In this study, we employed PBL and FAHP to evaluate the effectiveness of using robots as teaching aids in teaching product design.

Problem-Based Learning in STEAM Education
Nowadays, the need for cross-disciplinary education for further development of technology is emphasized.However, technical manpower and capacity are lacking, which needs to increase the number of students majoring in science and technology.Students are required to learn and practice professional skills and enhance their interest and motivation in learning.In STEAM education, students are provided with opportunities to experience a variety of disciplines and practice decisionmaking for given tasks.The integration of interdisciplinary learning in STEAM is accomplished through PBL activities with well-designed teaching aids (LaForce et al., 2017).The effectiveness of teaching and learning in STEAM education with PBL needs to be assessed.The integration of PBL into STEAM education was more effective for underperforming students than for performing students by mitigating the difference in the overall performance of the students (Han et al., 2015;Stearns et al., 2012).
It is important to make students understand the problem and apply learned concepts to find viable solutions through interdisciplinary education (Asghar et al., 2012).Mong and Ertmer (2013) stated that a studentcentered curriculum with PBL in STEAM education was effective in improving and equalizing the overall learning performance of students.They also suggested that the student-centered curriculum was the key to national scientific and economic development with great potential by learning problem-solving skills.Therefore, teaching aids need to assist students in developing such traits.

Contribution to the literature
• The main focus of this is the integration and implementation of Science, Technology, Engineering, Arts, and Mathematics (STEAM).• The contribution of this article focuses on the design and evaluation of teaching aids in STEAM education.
• STEAMH and FAHP questionnaires were combined and used in the development of teaching aids.

Fuzzy Theory
The fuzzy theory is widely used in the research of natural and social science.The fuzzy theory adopts precise mathematics to find solutions.The fuzzy theory was proposed to explore the inaccuracies and uncertainties of fuzzy data and make appropriate decision-making in fuzzy environments with fuzzy phenomena using mathematical methods (Zadeh, 1965(Zadeh, , 1975)).For example, shape, color, decoration, habitability, safety, and processability are assessed for the evaluation of a design, which is challenging with conventional quantitative analysis.Therefore, it is necessary to introduce linguistic variables to describe and solve for such evaluation.The fuzzy theory is used to quantify such fuzzy information for quantitative evaluation.
In design, computer-assisted systems are used to shorten the time for development and processing.Thus, decision support systems for optimal product design were proposed (Temponi et al., 1999), and an inference framework to meet customer requirements (CRs) was developed to relate CRs to design requirements (DRs) (Kim et al., 1998).However, professional knowledge and experience are required to construct the rules in these systems but whether the system works well or not cannot be assured.Using fuzzy sets, Kim et al. (1998) proposed a fuzzy multi-objective model by applying data for the competitive analysis of vendors to construct a function that explained the relationship between CRs and DRs.However, the construction of the function is arduous, especially for a new product (Chuen, 2001).In this case, fuzzy sets, fuzzy arithmetic, or defuzzification methods are used for defining complex and imprecise quality functions.However, the relational nature of DRs is not considered in these methods.Thus, the implementation of DRs is determined using quality function deployment for an optimal decision that is economical and satisfactory to the customers based on customer satisfaction by considering the organizational conditions such as cost factors and technical difficulties.Objectives in design may be fuzzy.Thus, it is necessary to determine the related factors to objectives and evaluate them.This is called the fuzzy comprehensive evaluation.In the fuzzy comprehensive evaluation, factors and the weights of the factors are determined through the evaluation of parameters and the construction of a single-factor evaluation matrix.The procedure of fuzzy evaluation is described, as follows (Saaty, 1980).

Influencing factors
In fuzzy evaluation, the factors that affect the value of the evaluation parameters are determined first.If the known influence factors are u1, u2, …, um, then the factor set is defined as U={u1, u2, …, um}, which is an ordinary set.

Factor weights
The influence or importance of each factor is different, that is, the weight of each factor is not the same.The degree of influence of each factor is presented by a weight of the factor.The set of the weight is defined as A={a1, a2, …, am}.ai denotes the weight of the i th factor, which satisfies Eq. (1).The weight set is a fuzzy subset and is shown in Eq. ( 2).
∑    =1 = 1,   ≥ 0 ( = 1, 2, … ,   ). (1) The weights of the factors are determined using weighting coefficients.In the hierarchical analysis, paired comparison on the Likert scale is used.Regardless of how to determine the weights, all the weights reflect factors, and their credibility is different.Therefore, depending on the data, the weight of a factor may vary.A judgment list is created to compare the importance of the evaluation targets and calculate the scores of the importance.The weighting coefficient is calculated using Eq.(3) and Eq. ( 4).
where ki is the total score of each evaluation objective and n is its number.

Decision parameters
The evaluation set shows the possible evaluation results of the subject, denoted by V={v1, v2, …, vn}, vi (i=1, 2, …, n).vi represents the total possible evaluation result.The purpose of fuzzy evaluation is to derive an optimal evaluation result from the evaluation set by considering all influencing factors.The relationship of vi with V needs to be ordinary to make the evaluation an ordinary set.

Evaluation matrix
Single-factor fuzzy evaluation is used to assess each factor.The result is used to determine the degree of affiliation of the evaluation object to the elements in the evaluation set.The object is evaluated according to the i th factor Ui in the factor set, and its degree of affiliation to the j th element of Vi in the evaluation set is rij.The result of the evaluation with Ui is expressed by the following fuzzy set function.
where Ri is a fuzzy subset of the evaluation set expressed as Ri=(ri1,ri2,...,rin) and similarly, the one-factor evaluation set corresponding to each factor can be obtained as Eq. ( 6).
The fuzzy matrix of the affiliation of each singlefactor evaluation set, R, is called the single-factor evaluation matrix (Eq.[7]).R is a fuzzy matrix and is regarded as a fuzzy relation matrix or fuzzy transformation from U to V. In this study, since there were many factors to consider, and each factor had a different weight, it was difficult to solve the problem by using a single-factor fuzzy evaluation and obtain reasonable results.Therefore, a multi-factor fuzzy evaluation matrix was used to divide the set of factors into several levels according to their characteristics.If the fuzzy evaluation matrix of the objective is given by Eq. ( 7), the weighted composite fuzzy evaluation and the product of the fuzzy matrices are conducted using Eq. ( 8).
where the symbol 「•」represents the fuzzy synthesis operation.There are various synthesis methods using the weighted fuzzy matrix A for constructing the factor judgment matrix R. In this study, four different synthesis models were used to evaluate the results of analysis and comparison.
Model 1: For the M(  ,  ) algorithm, where「  」and「  」stand for large and small values.
Model 2: For the M(  • , ) algorithm, Model 3: For the M( ) algorithm, Model 4: For the M( ) algorithm, Models used the weighted average and were characterized with ai and rewritten as M( + •, ) to define Eq. ( 13).
The models were created to explain the effects of factors and retain the information of the single-factor evaluation.There was no upper limit of ai and rij (i=1, 2, …, m; j=1, 2, …, n) but ai needed to be normalized.These were significant features and advantages of the models, which were used in the fuzzy evaluation of the parameters in optimal design to obtain better results.

Hierarchical Analysis
Analytical hierarchical analysis (AHP) is a decisionmaking method applied to problems with uncertainty and multi-criteria (Liu & Hui, 2005).In AHP, a simple hierarchical structure is used to systematize complex multi-criteria problems, and the decision maker compares the relative importance of two criteria in the same hierarchy.Thus, a pairwise comparison matrix is established to obtain the relative importance of the criteria, and then the total number of criteria in the hierarchy is calculated after linking them.Afterward, the total priority vector of the entire hierarchy is constructed as the weight of each evaluation criterion.The quantitative result of AHP helps the decision maker evaluate alternatives and determine the priority of the alternatives to reduce the risk of errors.Liu et al. proposed a weighting method to replace the pairwise comparison of AHP with a voting analytic hierarchy process using a ballot selection algorithm (Liu & Hui, 2005).The weights of AHP are calculated and compared in the following five steps.

Step 1. Defining decision problems
Step 2. Establishing a hierarchical structure Step 3. Constructing a pairwise comparison matrix Step 4. Calculating eigenvalues Step 5. Testing consistency Table 1 shows the scales and evaluations used in this study.
Several algorithms are used to calculate the eigenvectors in step 4 for the following methods: (1) Theoretical analysis method for calculating eigenvalues and eigenvectors.
(2) Approximation methods for the normalization of row average and the average of column vectors.The average of normalized columns is used to normalize the average of row vectors, and the normalized geometric mean is used to normalize the inverse mean of the row vectors.
The consistency obtained in step 5 is tested using the consistency index (CI) and consistency ratio (CR) to confirm the consistency of the evaluation in pairwise comparisons.If CI and CR are less than 0.1, the paired comparison matrices are consistent.CI and CR are calculated using Eq. ( 14) and Eq. ( 15).
where max is the maximum eigenvalue of the matrix, and n is the matrix order of the number of parameters.
where CR<0.1➔OK,CR is consistency ratio, CI is consistency index, and RI is random index.RI is a random indicator whose value increases as the number of criteria increases, as shown in Table 2.

Fuzzy Analytical Hierarchy Process
FAHP combines fuzzy theory and AHP to find solutions for problems with fuzzy nature.The fuzzy theory is integrated into AHP using TFNs to express the relative importance of two elements and then calculate the fuzzy weights of each criterion for decision (Grann, 1980).Buckley (1985) improved the original AHP and developed the fuzzy hierarchical analysis by introducing the triangular fuzzy numbers (TFNs) to the matrix of pairwise comparisons to manage the ambiguities from the process of criterion measurement and judgment.Buckley used trapezoidal fuzzy numbers to represent the relative importance of two elements to form a fuzzy inverse value matrix.The fuzzy weights of each fuzzy matrix were calculated by using the geometric mean method.Finally, the fuzzy weights of the alternatives were prioritized by using the graph of the affiliation function of the alternatives (Buckley, 1985).Chang (1996) used the triangular affiliation function to represent the ratio between each criterion and the extent analysis method to calculate the fuzzy synthetic extent weight of each element and the likelihood ratio of two elements in each unit.The degree of probability of each criterion was compared, the criterion with lower probability was chosen, and then the non-fuzzy weight of each element was calculated using regularization (Chang, 1996).Laarhoven and Pedrycz (1983) used a TFN to replace the explicit weight generated by pairwise comparison in AHP and then calculated the fuzzy weight of each criterion by the logarithmic least square method (Laarhoven & Pedrycz, 1983).It was used humans as a criterion for supplier selection.In that case, the expressed semantics could not be quantified, so fuzzy logic was used to transform the semantics into numerical values and determine the order of supplier selection.
The process of FAHP is similar to that of the original AHP method, but the difference is that fuzzy semantics, defuzzification, and normalization are required in FAHP.The original AHP is modified appropriately, and its shortcomings are solved using FAHP, which allows final decisions to be more realistic.In FAHP, criteria and alternatives in AHP are regarded as factors in the fuzzy method.

METHODS
We designed and showed three different robots for the experts to evaluate the acceptance of using the robots as teaching aids in STEAM education (Figure 1).Robot 1 had a clown look and swung its body and hands.Robot 2 looked like a fish and swung its tail.Robot 3 was a device that moved a car.The robots were evaluated for "structure", "function", "economy", "aesthetic", and "creativity".The appearances of the robots were compiled through morphological analysis.
A questionnaire survey was conducted to obtain data for the evaluation using FAHP.The weights and preferences of users (teachers and students) were considered in FAHP.The five evaluation criteria were included in the questionnaire survey.The process of this study is described in Figure 2.
Interviews were carried out following the predesigned content, and the result was compiled for analysis.Using the result, the selection process of the robots and the evaluation criteria were determined.Then, the evaluation goal was established for the design and use of the robots at the first level.At the second level, the evaluation criteria were defined, and at the third level, 15 alternatives for criteria were selected.As there were many factors affecting the evaluation, it was necessary to quantify and combine the degree of influence of the factors to obtain quantitative results.The evaluation criteria and alternatives were, as follows.
Structure referred to the structure of the robots and the alternatives for the structure included "logic" and "creativity".For function, the alternatives were defined as " simplicity ", "practicality", and "convenience".For economy, "assemble method", "cost", and "appearance" were chosen, as its alternatives.For aesthetics, the alternatives were "texture", "color", "avant-garde", and "preference".
For creativity, "innovation", "uniqueness", and "personal style" were selected as the alternatives.
We invited 15 experts in STEAM art and design and six experts in the industry.During the interview, questionnaires were filled out by the interviewees to define the goal, criteria, and alternatives.To reflect the importance of each factor, the relative weights were calculated to determine the proportion of weights.For each item of the questionnaire, scores of one, three, five, seven, and nine were assigned to 'important', 'slightly important', 'quite important', 'extremely important', and 'absolutely important', respectively.Scores two, four, six, and eight were assigned between the five scales (Table 1).

RESULTS & DISCUSSION
The result of FAHP is presented in Table 3 and Table 4 showing the weights of criteria and alternatives.
CR of the criteria was 0.01, while that of the alternatives was between 0.01 and 0.02, respectively.Thus, the evaluation method showed satisfactory consistency, which indicated that the weights were acceptable.The overall consistency of FAHP was determined by the consistency ratio hierarchy (CRH).CRH was 0.023, which indicated that the evaluation of the whole hierarchical structure was acceptable.
From the results in Table 4 and Table 5, and Eq ̃= [0.264,0.304,0.267,0.063,0.00]. (16) In addition, the evaluation set V={equally important, slightly important, quite important, extremely important, and absolutely important} was defined.Based on the weight set of criteria and the evaluation set, a questionnaire for the subsequent survey was designed.
The invited experts responded to the revised questionnaire, and the data were analyzed to construct the fuzzy evaluation matrix of each criterion.By creating the fuzzy evaluation matrix and using model 4, the evaluation was performed.The fuzzy evaluation results were normalized, and the final fuzzy evaluation sets of weights were calculated, as follows: The maximum attachment method and the weighted average method (WAM) were used to defuzzify the weights and convert them into quantitative and representative values for comparison and ranking.We used grade assignment using WAM to calculate the defuzzified value of the evaluation results by assigning {1, 0.75, 0.50, 0.25, 0} to V. The results are shown in Table 5 and Table 6.Table 5 shows the defuzzified weights of the criteria, while Table 6 presents the result of the evaluation of the use of robots in STEAM education.The defuzzified weights of the structure, function, economy, aesthetics, and creativity were 0.672, 0.683, 0.702, 0.691, and 0.722, which implied that the experts considered the criteria to be significant.The defuzzified weights were not different significantly, which indicated a high degree of commonality of the.The overall defuzzified weight of using robots in STEAM education as teaching aids was 0.723.The result also showed that the use of robots in STEAM education was satisfactory.
The defuzzified weights of the use of the robots as teaching aids in STEAM education were 0.698, 0.671, and 0.736, which using robots in STEAM education was satisfactory.Robot 3 was most preferred teaching aid owing to its functions and educational effects.The results of this study showed that the developed FAHP can used for the evaluation of other teaching aids in STEAM education, and the result of this study could be a reference for such studies (Table 7).
technology, students need to be trained in scientific literacy for competence.Thus, the importance of STEAM education has been emphasized, and effective educational methods for STEAM education have been proposed.In STEAM education, many teaching aids such as robots are used.Thus, we developed an evaluation method to evaluate the acceptance of teaching aids in STEAM education using the fuzzy theory and FAHP focusing on PBL.The experts were interviewed and surveyed to define the goal, criteria, and alternatives (factors) for evaluation.With the goal, criteria, and alternatives that the experts defined, five criteria (structure, function, economy, aesthetic, and creativity) and 15 alternatives (logic, creativity, simplicity, practicality, convenience, assemble method, cost, appearance color, avant-garde, preference, innovation, uniqueness, and personal style) were determined using FAHP.The weights of the criteria and alternatives were calculated and defuzzified to obtain the final result.The results showed that the defuzzified weights of the criteria were 0.672-0.722,and that of using robots as teaching aids in STEAM education was 0.733.Such weights indicated that the experts thought the use of robots in STEAM education to be important, and the defined criteria were important in the decision-making of using robots as teaching aids.The use of robots designed in this study was evaluated, and their defuzzified weights were 0.698, 0.671, and 0.736.The result implied the importance of using the robots.Robot 3, a rotating car showed the highest weights for the alternatives of logic, creativity, simpleness, Avant-garde, and innovation.Robot 1 and robot 2 imitated a human and fish with simple motions and did not have a sophisticated appearance, and experts thought that students could be less keen on using them in classes.The developed FAHP allowed for the effective evaluation of using teaching aids in STEAM education.Therefore, it can be applied to the evaluation of other teaching aids.However, further research is required to refine the proposed for the evaluation of the use of teaching aids. [0.377,0.213,0.147],

Table 1 .
Scales & evaluations in this study

Table 3 .
Weight of criteria in evaluation of robots in STEAM