Mathematics for Computer Science: A Flipped Classroom with an Optional Project

The Mathematics for Computer Science mandatory course was conducted in a flipped classroom (FC) setting with an optional, voluntary, project-based learning (PBL) component. The objective of this study was to examine the effect of studying in an FC setting, with and without PBL, on students ’ problem-solving performance, conceptual understanding, and affective perceptions. Participants were 374 undergraduate freshmen and sophomores, of whom 20% elected to participate in an optional probability PBL. The most significant finding was the reinforcement of collaboration, as reflected by teamwork on problem solving during the FC class time. The PBL students demonstrated an advantage in their performance over their peers in probability — the topic they explored, and in their positive responses regarding studying in the FC setting, with an emphasis on the collaborative learning component. The research demonstrates the importance of active engagement in a technology-based environment in which STEM undergraduate students solve problems and implement projects.


INTRODUCTION
Today's institutions of higher education are being challenged to meet the needs of 21 st century learners, who are expected to integrate high-quality technological online teaching and learning tools, together with social, in-person, learning environments (O'Flaherty & Phillips, 2015). Active learning, which engages students in productive collaboration, is particularly important for Science, Technology, Engineering, and Mathematics (STEM) undergraduate students in order to enter the 21 st century workforce (Dori & Belcher, 2005). Today's STEM students will be tomorrow's makers of things that serve society. Therefore, universities should engage these students in active learning, and provide them with the knowledge and skills of high-achieving engineers (ABET, 2019;Freeman et al., 2014;National Science and Technology Council, 2013). These skills, which go beyond factual knowledge and are considered expert competencies, involve high-level problem-solving abilities and conceptual understanding of the learning material (ABET, 2019; Blumenfeld et al., 1991).
There is growing realizing that scientific and mathematical knowledge and cognitive skills alone are not enough for an engineer to become successfully integrated into 21 st century careers. Students entering the workforce are expected to be equipped with 'soft skills', such as collaboration and communication (Seman, Hausmann & Bezerra, 2018). Indeed, active learning is often achieved through meaningful collaboration and communication, as students work together to achieve a common goal and have more time to construct knowledge with the assistance of their peers (Wiggins et al., 2017). Research by Molinillo, Aguilar-Illescas, Anaya-Sánchez, and Vallespín-Arán (2018) showed that collaborative learning methodologies that combine online and in-person interactions, increase students' emotional engagement in learning activities and ingrain skills that will be useful in their professional lives.
Another common method for collaboration through active learning is Project-Based Learning (PBL 1 ), in which groups of learners work together to create a collaborative artifact that demonstrates their understanding of core learning concepts (Chang & Lee, 2010). Active learning in the context of PBL is more intense, since students must learn autonomously or in teams to solve the smaller problems that are necessary for solving the final problem (Dori, 2003;Huang, Chiu, & Hong, 2016), while the instructor serves as a facilitator for students' self-directed learning and teamwork efforts.
This study responds to calls from numerous researchers for more rigorous investigations into learning environments that combine online learning platforms with face-to-face instruction for STEM undergraduate students (Bernard, Borokhovski, Schmid, Tamim, & Abrami, 2014;Chen, Breslow & DeBoer, 2018;Reis, et. al., 2018). It contributes to ALE literature by investigating the implementation of ALEs in Computer Sciences (CS). We focus on an undergraduate mathematics course that combined two ALEs: flipped classroom, and a project-based learning approach. The following sections expand upon active learning in higher 1 Please note that in this paper PBL refers to Project-Based Learning rather than Problem-Based Learning education, specifically for STEM students, and then describe two modes of active learning-the flipped classroom (FC), and project-based learning (PBL).

THEORETICAL BACKGROUND Active Learning in STEM Higher Education
ALEs comprise a variety of teaching methods that emphasize guided, learner-centered activities. Active learning refers to social, in-person, and dynamic processes that occur between instructors and students, as they work together to achieve mutual learning objectives in a supportive and respectful environment (Eberlein et al., 2008). Research on STEM courses has demonstrated that courses with an ALE nature help students achieve meaningful learning outcomes as reflected by higher order cognitive skills, retention of material, and the ability to transfer knowledge to other disciplines and applications (Bishop & Verleger, 2013;National Research Council, 2012). In several higher education programs, STEM students engage in active learning by working in teams, typically to design a final product or other artifact. Such learning incorporates real-world concepts and equips STEM students with the skills necessary to enter a workforce that requires and makes use of their ability to solve problems in novel ways (Opdecam et al., 2014). ALEs have been implemented successfully most notably for physics (Dori & Belcher, 2005;Dori, Hult, Breslow, & Belcher, 2007), chemistry (Golde, McCreary, & Koeske, 2006), and engineering (Crawley, Brodeur, & Soderholm, 2008;Dori & Silva, 2010).

The Flipped-Classroom ALE
The term Flipped Classroom (FC) was first defined in the literature by Lage, Platt, and Treglia (2000) as a learning environment that aims to engage students in active learning by restructuring the time and tasks performed in and out of the classroom. Course content is effectively relayed to students outside class time using methods such as online tutorials, clips of video lectures, textbooks, and pre-class exercises. Classes are actually Contribution to the literature • The flipped classroom (FC) with and without the project-based learning (PBL) component improved students' conceptual understanding of the Mathematics for Computer Science course. • Probability was the focus of the PBL. In this topic, the FC & PBL students gained a higher level of conceptual understanding than their FC peers. • Students' affective responses regarding studying in the FC setting were mostly positive, with an emphasis on the collaborative learning aspect. These findings were most significant in the group of students who volunteered to engage in the PBL component. • The research demonstrates the importance of active engagement in a technology-based environment in which undergraduate students solve problems and implement projects.
3 / 20 collaborative problem-solving sessions, and students are required to come prepared by independently reading the required material and reviewing the relevant concepts (Muir & Geiger, 2016). Bishop and Verleger (2013) emphasized that the most important aspect of the FC is its ability to enable active learning during class time through teamwork or collaborative learning, which draws on the cohesiveness of students learning together in groups. Class time is dedicated to problem solving activities, which would traditionally be assigned as homework. In collaborative learning, students lead discussions and ask for help from the course staff when needed. Team members must be able to rely on and motivate each other to achieve individual and group success (Dym, Agogino, Eris, Frey, & Leifer, 2006). This requires that students be accountable for preparing for the day's task. During class time, course instructors are available to interact with students by observing, guiding, helping, and coaching them. Thus, the responsibility for learning and understanding the material is shifted away from the instructors and the course becomes more student-centric (Blumenfeld et al., 1991;O'Flaherty & Phillips, 2015). Akçayır and Akçayır (2018) discussed the advantages of the FC especially with respect to students' learning gains and challenges related to insufficient preparation before the class sessions.

Project-Based Learning
In project-based learning (PBL), students perform a series of activities needed to solve a problem and apply appropriate strategies. PBL often involves exploration of a real-world problem that does not have one single solution (Dori & Silva, 2010;Huang, Chiu, & Hong, 2016). This process, which must be a product of collaborative work of teams of students, involves analyzing and searching for possible solutions, choosing an optimal solution, defending and explaining their choice, and creating a substantial or intellectual product or artifact with a real-life aspect (Crawley et al., 2008).
A key benefit of PBL is that the students themselves must actively produce all the knowledge they must acquire regarding the concepts covered in class, rather than simply summarizing the class material (DeLozier & Rhodes, 2017). The principles at the basis of engagement in PBL assignments involve collaborative-based learning, such as team development and peer evaluation (Muir, Beswick, Callingham, & Jade, 2016). In order to contribute to the success of the group's final project, students must come to group meetings prepared. This requires students to be active learners, collaborate with their teammates, and then actively contribute to creating a quality artifact (Webber, 2012).
The PBL method has been integrated successfully into many STEM undergraduate courses (Barak, Watted, & Haick, 2016;Crawley et al., 2008). PBL "is driven by the premise that basic science concepts will be understood and remembered longer when they are learned, discussed and applied in a practical, real world context" (Eberlein et al., 2008, p. 236). Adding projectbased elements to traditional courses represents another shift from teacher-centered learning to active learning that mimics real-life settings (Opdecam, et al., 2014), especially in companies and industries. Significant project elements are included in the Conceive Design Implement Operate (CDIO) approach (Crawley et al., 2008;Dori & Silva, 2010), which involves exposing STEM students to experiences that they are likely to encounter during their professional lives. Seman and colleagues (2018) found that electrical engineering students who participated in an optional real-world PBL project showed marked improvement in their ability to independently seek out information and develop solutions to novel problems. Their results suggest that "humanization is an important part of the education process of a new electrical engineer" (p. 26). They explain this claim by noting that while engineers entering today's workforce are expected to have at least baseline technical knowledge, it is awareness of the human element that sets exemplary engineers apart from the rest. Today's engineers must be aware of how their emotions affect their creativity and teamwork, to ultimately produce novel, "out of the box" results.
When new pedagogical techniques are introduced, it is important to ensure that the students are engaged in learning and are motivated to study (Kohen, 2019). This is especially true for learning in practical settings, i.e., in actual classrooms, and for evaluating learner-centered activities. To ensure the efficiency of ALEs such as the FC, studies should investigate both cognitive and affective aspects of learning experiences in these environments (Mayer, 2008;Webber, 2012). Our study investigated various aspects of STEM undergraduate learning experiences, including problem solving, conceptual understanding, and affective perceptions. Studies on the combination of cognitive and affective aspects of teaching and learning in a flipped classroom are infrequent. Studies on this topic that focus on PBL are even less prevalent, particularly in higher education (Lundin et al., 2018). In this study, we investigate both cognitive and affective learning outcomes of CS undergraduate students who study in an FC environment. Due to the unique characteristics of PBL for promoting active learning among students, we also investigated these aspects among students who enrolled both in an FC course and completed an optional PBL program on probability, one of the topics covered in the FC course. The cognitive learning outcomes of these students, particularly those that were aligned with probability studies, served as a measurement of the students' success.
The questions we set out to investigate in this research were as follows: How did studying in an FC 4 / 20 setting affect CS undergraduate students' problem solving, conceptual understanding, and affective perceptions? What were the differences, if any, between students who studied in the FC setting with and without PBL in the three learning outcomes?

RESEARCH METHOD
The study was conducted during the academic year 2014 (fall and spring semesters) on a population of undergraduate students who enrolled in the Mathematics for CS course (OCW.MIT.EDU.6.042, 2015), designed using an FC setting. The course topics included fundamental concepts of discrete mathematics such as mathematical proofs, number theory, induction, and probability. These included discrete mathematical structures, such as numbers, graphs, trees, and counting, as well as discrete probability theory. In the fall semester, students also had the option of voluntarily performing an additional PBL assignment that involved producing a final artifact on the topic of probability. Hereinafter, we refer to the group of students who participated only in the FC environment in both semesters as the FC group, and the group of students who also volunteered to participate in the PBL assignment as the FC&PBL group.

Participants
The participants were 374 students majoring in Electrical Engineering and Computer Sciences (EECS) at a top-tier science and engineering university, who attended the mandatory Mathematics for CS course. Of these students, 71 also volunteered to carry out an optional PBL assignment on topics related to probability. Enrollment data regarding gender and achievements on college entrance exams, as collected from the university admissions department, confirmed that the backgrounds of the two groups were similar. Most of the students (about 70%) were familiar with the active learning method, as they had previously participated in two studio physics courses, which offer a similar learning setting (Dori & Belcher, 2005). At the beginning of the semester, the course staff formed the teams in a way that maximized diversity in each team with respect to gender, year of study, major, and nationality. This diversity contributed to the success of the FC implementation (Reis, Isotani, Rodriguez, Lyra, Jaques, & Bittencourt, 2018).
Although less than one-third of the students did not experience ALE before attending this mathematics course, when dividing the students into groups, the instructor made sure that each group included at least one student with experience with a learning environment of this kind, in order to make sure that the lack of previous experience would not influence students' learning experience and outcomes in this course. The study received the approval of the University Committee on Use of Humans No. 1308005860.

The Learning Activities
The FC learning environment included two main settings: an online setting using the online university platform before and after class, and a classroom setting at the physical university campus. The FC activities used in this learning environment were aligned with the literature about FC (Bishop & Verleger, 2013;Lage, Platt, and Treglia, 2000). Appendix A presents the list of all the activities used in the FC learning environment in the CS course, including both the online activities that took place before and after class, the course textbook (Lehman, Leighton, & Meyer, 2017) and the collaborative problem-solving activities that took place in the physical classroom setting. Appendix A also presents descriptive statistics for each activity listed in the table to demonstrate students' perceptions of the importance of these activities (on a scale ranging from 1=not helpful, to 5=very helpful).

FC online setting
The online course website provided the students with access to the complete course content, including video lectures (approximately 15 minutes each, covering every topic in the course curriculum), lecture notes, syllabus, solutions to all the problem sets and in-class problems addressed, and links to resources, such as the Piazza online problem-solving forum (a Q&A web service) and the AI-based online tutor, which provides students with instant feedback using a "checkable answer feature" developed for other university online courses. Appendix B illustrates the course homepage. An example of an online problem is shown in Appendix C.

FC in the physical classroom setting
FC course meetings were two-hour sessions held three times per week (for a total of six hours weekly) at the university physical campus in a specially-designed classroom that facilitated in-class teamwork. Figure 1 shows a panoramic view of the customized classroom used for the course meetings.
As recommended in the literature (Bishop & Verleger, 2013;Lage et al., 2000), the FC physical learning environment was designed with round tables that accommodate six to eight students, to facilitate a variety of active learning methods, including cooperative and peer-assisted learning techniques. Problem solving, which would traditionally be assigned as homework, was done in teams during class time.
In this FC, most of the interaction during class time was between students, who were supported and guided by Teaching Assistants (TAs) and Learning Assistants (LAs). The main difference between TAs and LAs was seniority -LAs were generally undergraduates who had successfully completed the course beforehand, while TAs were generally EECS graduate students. Teaching and Learning Assistants were students who had done very well in the same course in prior semesters, and volunteered to 'coach' their peers in subsequent semesters. Every group was assigned one coach, who's main responsibility was to be available to answer students' questions and provide guidance upon request. Coaches did not give lectures or individual lessons. Rather, their role was to serve as additional resources for students' in-class activities. The course professor guided the TAs and LAs before the beginning of the course, listened to their mentoring experiences, and moved from one group to another during class, while sometimes asking the students questions or commenting on their responses.

PBL assignment
In line with the literature on PBL (Dori & Silva, 2010;Huang, Chiu, & Hong, 2016;Muir, Beswick, Callingham, & Jade, 2016), the main principle underling engagement in the PBL assignment was collaborative-based learning as related to the concepts learned in class. PBL students were grouped into teams of six to eight students and were presented with a complex, open-ended, real-world problem in probability. The groups met outside the regularly scheduled course for a few hours, once or twice a week, starting immediately after the first midterm until the end of the semester. In order to simulate a typical collaborative work environment used by engineers, in which each employee is assigned a unique role (Seman, Hausmann & Bezerra, 2018), the group members chose a specific role for each one of them. The roles included: (a) leader, who was in charge of project oversight, management, and team communication; (b) content expert; (c) pedagogy expert; (d) visualization & technology expert; and (e) documenter. The final project involved the creation of an artifact of the team's choosing that was related to probability in a real-world context, such as the gambler's ruin, random walks on graphs, the birthday principle, or Simpson's paradox.
The teams selected their target audience and how to present their artifact in a way that would demonstrate to the course staff and to their peers that they had attained deep understanding of the problem they had chosen and of probability in general. Several types of artifacts were suggested: (1) Compose a quiz with four to five questions, including their solutions, and play the role of an external assessor; (2) identify, design, and prepare enrichment material for the course (slides, video, poster, or a new proof), and use them while playing the role of an instructor; (3) perform a critical review of two recent papers while playing the role of a journal reviewer (e.g., find the papers' strengths and weaknesses); and (4) flesh out and explain a phenomenon or concept, and adapt it for a hypothetical audience other than undergraduate students while playing the role of a popular science journalist. Appendix D presents an example of a final artifact.

Data Collection Tools
Students' problem-solving performance was evaluated using a pre-test, traditional midterms, and a final exam, which were administrated to students before, during, and after the course (respectively). These tests were administrated to all the students as part of the FC course requirements. The midterms and final exam included problem solving tasks on the course topic, namely discrete mathematics, while one section was devoted to probability. The main goal of the pre-test was to measure students' prior knowledge in general and in probability in particular. Scores for this test ranged from 0 to 100, and students' scores were given by the lecturer and course assistants. As an example, Figure 2 presents a probability question used in the final exam and one student's solution. The question required students to identify the appropriate method for solving the problem, based on the methods taught in the course, while considering various solutions and rejecting any solution that did not represent an integer number of dice. The student's solution demonstrates the use of a tree diagram to check the probability of rolling two dice and getting the specific number 2 on both. A full solution would require continuing the tree diagram and finding that the maximum number of dice that the student should roll should be three, otherwise the number of dice will not be an integer, or the answer will be smaller than 0.084. Figure 1. Panoramic picture of the classroom used for the Mathematics for Computer Science FC learning environment Students' conceptual understanding was measured using pre-and post-conceptual questionnaires that focused on a general concept addressed in the course and an additional concept related to probability that might not have been covered in detail in class. The students were asked to illustrate the concept in a few sentences and briefly explain or give an example of its application. Some of the probability concepts might have been covered by students' voluntary PBL assignments. The general concepts included well-ordering, relations, recursive data, digraphs, counting, conditional probability, and proofs, and the probability concepts were strange dice, the birthday principle, Simpson's paradox, mutual independence, confidence vs. probability, gambler's ruin, and random walks on graphs. Scores for conceptual understanding ranged from 0 to 3 as follows: 0 = no response, 1= low, 2 = intermediate, and 3 = high. Before evaluating the concepts, content validity, and inter-judge reliability were assessed by four experts, of whom three were mathematics experts and one was a mathematics educator expert. Each of the concepts was reviewed by the four experts, who met to reconcile their agreements and disagreements regarding each of the scores ascribed until standardized rules for coding for each concept were formulated. Then, a recursive process of reliability was repeated by coding about 20% of the concepts, until the coders reached a 90% agreement. An example that reflects the rules for coding a high-level response to the well ordering principle is presented in Appendix E.
Finally, we used an open-ended affective survey to measure students' affective perceptions. In the survey, we asked the students to reflect on (a) the FC setting and components of the course, and (b) their feelings regarding leaning in an FC setting. The students who also participated in the optional PBL assignment were asked to respond to this survey not just in relation to FC, but also with respect to their experiences in the PBL program while working on a project and creating an artifact. These questions allowed us to identify the elements perceived by students as the most significant for ALEs, and then categorize their perceptions toward each of these elements.

Data Analysis
All the data, except for the open-ended affective survey, were analyzed quantitatively using Analysis of Variance statistical methods. The analysis of the Affective Rubric was conducted in two phases. The first phase was thematic analysis (Creswell, 2014) in which recurring categories or themes related to the research goals were identified within the students' responses to the affective survey, and assigned a category name. The second phase involved quantification of the affective responses. Directed content analysis (Hsieh & Shannon, 2005) was applied to sum the frequencies of the responses in each category that appeared in the written responses. This two-phase process yielded four main categories: teamwork (e.g., group cohesiveness), motivational orientation (e.g., the workload and effort required to succeed), individual learning (preferred learning style), and evaluation (e.g., perception of weekly out-of-class problem sets). For each full statement, several affective values were coded for four main categories: (+) Positive: The student expressed a positive emotion, such as enjoyment, happiness, fulfillment, or confidence, towards the item described; (=) Ambivalent: The student did not attribute an emotion to the item described (neutral overall affect) or the student affectively described the category as both positive and negative; and (-) Negative: The student attributed a negative emotion (anger, unfairness, etc.) to the statement described. Examples of students' statements are presented in the sub-section entitled Affective responses to the FC active learning environment in the Results section.
To ensure validity and reliability, the four expert coders who were familiar with the course, each rated approximately 30 statements. The coded values of these Figure 2. An example of a problem-solving question and one student's solution Problem 1: We flip a fair coin. If heads comes up, then we roll one die and note the result. If tails comes up, we roll X dice and note the sum of the results. If we want to get the same number in all the dice, how many should we roll to get a probability that is greater than 0.084? Please demonstrate the solution process.

RESULTS
In this section we describe the results of the quantitative comparison between the FC and FC&PBL students' problem-solving performance and conceptual understanding. We then present the results of a quantified analysis of students' affective perceptions of studying in the FC, and compare them to perceptions of students who took part in the optional PBL assignment.

Quantitative Analysis of Students' Problem-Solving Performance and Conceptual Understanding
We conducted t-tests for independence of the FC group and the FC&PBL group based on all the problem solving tests, namely (a) the pre-test (general grade and grade in the probability section), (b) first midterm, (c) second midterm, and (d) post-test. We revealed significant differences only for the pre-test, t(302)=2.21, p<.05; and the first midterm, t(296)=-4.43, p<.001. However, while the FC&PBL students outperformed the FC students in the pre-test (M= 56.56, SD=12.26; M=52.78, SD=15.74; respectively), the FC students outperformed the FC&PBL students in the first midterm (M= 57.55, SD=15.63; M=66.49, SD=16.03). No differences between the two groups were found for the second midterm and the post-test, neither for the general grade nor for the grade in the probability section. Since the pre-test was designed to measure the students' previous knowledge and was not related to the material taught in the CS math course, the subsequent findings do not include the pre-test, other than for controlling purposes to account for the differences between the two investigated groups based on this test.
To investigate the impact of studying in the FC versus FC&PBL settings on students' problem-solving performance, we conducted two repeated measures tests, each measuring the change during the course as reflected by the general final score and the final score in the probability section in the post-test. We first conducted a two-way ANCOVA repeated measures test with Type of Measure (first midterm, second midterm, and general post-test scores) and Group participation (FC vs. FC&PBL) as independent variables, and Score as a dependent variable, accounting for the effect of the pretest. Findings indicated a significant interaction between Type of Measure and Group participation, F(2,289)=9.00, p<.001, η 2 =.059. Simple main effect tests conducted separately for each group indicated that time had a significant effect on both groups (p<.001), so that improvement in students' scores was found from the first midterm to the second midterm, and a decrease in students' scores was found from the second midterm to the final exam. A larger decrease was found among the FC group, which demonstrated a relatively low score at the end of the course compared with the FC&PBL group, whose final score matched their score on the first midterm.
We then conducted a two-way ANCOVA repeated measures test with Type of Measure (first midterm, second midterm, and probability post-test) and Group participation (FC vs. FC&PBL) as independent variables, and Score as a dependent variable, accounting for the effect of the pre-test. Findings indicate a significant interaction between Type of Measure and Group participation, F(2,286)=10.71, p<.001, η 2 =.070. Simple main effect tests, conducted separately for each group, indicated that time had a significant effect on both groups (p<.05), as students' scores were shown to improve from the first midterm to the second one, and from the second midterm to the final probability score. See Table 1 for means and standard deviations.
To investigate the impact of studying in the FC versus FC&PBL settings on students' conceptual understanding, we first conducted an independent t-test to determine whether the groups' scores differed for the conceptual understanding pre-test. Findings revealed no significant differences between the two groups (t(268)=1.50, p>.05) for the mean score of conceptual understanding as demonstrated by the pre-test for the FC and the FC&PBL groups (M=1.72, SD=.77; M=1.87, SD=70, respectively). This allowed us to conduct two repeated measures tests, each measuring the change during the course as reflected by the post-score for conceptual understanding of a general concept (selected by students) and by the post-score for conceptual understanding of a probability concept.
First, a two-way ANOVA repeated measures test was conducted with Time of Measure (conceptual understanding pre-vs. post-test for the general concept) and Group participation (FC vs. FC&PBL) as independent variables, and Score as a dependent variable. We found a significant effect for time, F(1,239)=25.87, p<.001, η 2 =.098, but no significant interaction, F(1,239)=.02, p>.05, η 2 =.000, indicating that Note: Scores for the problem-solving tests ranged from 0 to 100 the conceptual understanding of concepts taught in the FC course progressed equally for both groups.
Next, a two-way ANOVA repeated measures test was conducted with Time of Measure (conceptual understanding pre-vs. post-test for a probability concept) and Group participation (FC vs. FC&PBL) as independent variables, and Score as a dependent variable. No significant effect for time was found, F(1,208)=.29, p>.05, η 2 =.001, but there was a significant interaction, F(1,208)=12.31, p<.01, η 2 =.056. Simple main effect tests, conducted separately for each group, indicated that time had a significant effect on both groups (p<.05). However, while the FC group demonstrated a decrease in their conceptual understanding of probability, the FC&PBL group demonstrated an improvement (see Figure 3). Figure 4 shows the distribution of responses per category, as well as the distribution of participants' positive (+), ambivalent (=), or negative (-) responses to each category. The figure (presented as concentric circles with slices) should be read in a clockwise manner: each slice is sized proportionally to the number of responses received in that category overall, and then to each affective response ranging from the highest to the lowest frequency. Red segments indicate a negative affective value; yellow-an ambivalent value; and green-a positive value for the category. For example, the category mentioned most commonly was individual learning, and therefore its innermost slice is the largest. Within this slice, the largest segment corresponds proportionally to the number of ambivalent responses received (36.0%). Figure 4, we received a total of 1505 affective responses regarding students' learning experiences in the Flipped Classroom environment. Overall, students commented most frequently on the individual learning category (34.4%), followed by the teamwork category (31.6%), then the motivational orientation category (22.3%), and finally the least common categoryevaluation (11.4%).

As shown in
Within the individual learning category, students were mostly ambivalent (36.0%), followed by positive (34.9%). The remainder of the responses were negative (29.1%) in nature. An example of a student's ambivalent approach  [FC] concept has diminished as the term has gone on.

In the beginning, I refused to utilize all of the available resources, but I have gotten better about that as time has gone on. The [course's] subject material is conducive to this approach, and I like the class more now, I'm just mad I didn't 'buy-in' earlier" (S_117).
The second most frequent category was teamwork, in which most students expressed positive responses (57.7%), followed by negative (24.3%), then ambivalent responses (18.0%). Regarding the teamwork category, one student stated that "group problem solving was a great resource for understanding the material. I was not too confident going into this class ... but I really ended up liking the group style as the rest of my team has been very willing to help me ... I've learned a lot and become more confident in this subject as a result." (S_24).
Like the teamwork category, most students (44.1%) expressed positive responses to the motivational orientation category, followed by negative (34.5%), then ambivalent responses (21.4%). Finally, the Evaluation category was the least common one, and when students mentioned it, their comments were mostly negative (45.7%), followed by ambivalent (29.5%), and finally positive (24.8%).

Affective responses toward PBL
As shown in Figure 5, FC&PBL students referenced very frequently to the teamwork aspects (59.8%) of the PBL learning environment. In this group, 24.6% mentioned motivational orientation, 14.5% mentioned individual learning, and only 1.1% mentioned evaluation.
The positive effect of teamwork is clearly visible in the FC&PBL group, where the majority of students (64.8%) reported a positive effect in this category. Students wrote that they had been positively motivated by their peers to actively contribute to the success of their team's project. For example, one student (S_97) wrote that some of the most important skills required for working in a successful group were to "listen to everyone's voices and be open to new things. Have accountability and make sure that you get your work done." Another student (S_12) reported: "During the course, we realized that the team, as a whole, needs to be able to collaborate in a respectful manner. There must be some understanding that everybody's input is respected and valued to make the atmosphere more conducive to learning...". Yet another student (S_109) noted that one of her most significant experiences of working in a group was that "every time I could contribute to someone else's understanding of the topic, it gave me a sense of accomplishment and it helped me have more confidence in my own abilities".
Though mentioned less frequently, the FC&PBL group was mostly positive regarding motivational orientation (71.9%), and equally positive and ambivalent with respect to individual learning (43.3%). The lowest response rate of the FC&PBL students was in the evaluation category (1.1%), where they were mostly ambivalent (42.8%).

DISCUSSION AND CONTRIBUTION
Our study investigated a flipped classroom (FC) environment in a mathematics course for undergraduate Computer Science students. Our primary goal was to gain insights into learning in an FC at the undergraduate level, with and without students' participation in an optional PBL assignment.
For both groups, we found an increase in students' scores for the various problem-solving tests, and no differences between the groups for the second midterm and the post-test. This was true for both the general score and the probability score. These results are encouraging, especially with respect to the FC&PBL students, because the voluntary decision of about 20% of the students to join the PBL program occurred after the first midterm, in which the performance of the FC&PBL students was low compared to the FC students. The FC&PBL students saw the optional project as a way to improve their final course grade. The analysis indicated that toward the end of the course, the FC&PBL students succeeded in improving their problem solving performance and closing the gap with the other students.
The contribution of the additional PBL component was specifically manifested in students' conceptual understanding category, which increased toward the end of the course and reached the same level as the FC students. Moreover, with respect to the probability conceptual understanding category, which was the topic Figure 5. Students' affective responses toward participating in the PBL in addition to studying in the FC, arranged by category and affective value: negative (-), ambivalent value (=), or positive (+) selected for the final PBL artifact, the FC&PBL students demonstrated a higher gain in conceptual understanding than their FC peers, whose postconceptual questionnaire outcomes decreased. This decrease in the FC students' scores is likely due to the fact that the pre-conceptual questionnaire focused on probability and other topics that they had studied in high school. The post-conceptual questionnaire included additional topics as well, and all the FC students had to study in order to respond to questions on all the topics covered in the course, not only probability.
The higher gains of the FC&PBL students in their conceptual understanding of the probability topic reinforces previous studies that point to the importance of promoting active learning skills such as conceptual understanding, which go beyond mere factual knowledge (Blumenfeld et al., 1991;Dori & Belcher, 2005;. The nature of PBL is a more intense form of active learning than FC without a PBL component, since students must construct new knowledge for their project, yielding higher learning benefits (DeLozier & Rhodes, 2017). Our findings underscore the importance of FC with optional PBL as key for undergraduate students' active learning, which can be offered to students interested in a more active, hands-on experience. This coincides with a study conducted by Chang, Song, and Fang (2018). Moreover, assigning students to small heterogonous groups that are required to tackle a complex, open-ended challenge, such as a joint project, enhances learning for both highability students and less skillful students, who must actively cooperate to successfully complete the project. Indeed, the PBL active learning method can be implemented in large courses as well, as suggested by Opdecam and colleagues (2014).
In the open-ended affective survey, we found four main categories that students described as significant in ALEs: (1) teamwork, (2) individual learning, (3) motivational orientation, and (4) evaluation. The first two categories were found to be the most significant to students. Through participation in the FC, students gained the skills needed to master the material as they engaged in both self-guided learning and collaborative work (Akçayır, & Akçayır, 2018;Muir & Geiger, 2016). These active learning skills are essential for entering and thriving in the 21 st century workforce. Jensen and colleagues (2015) found that successful learning in the FC environment requires a good balance between the technological component of this environment, e.g., the quality of the online videos, and the active learning component of the FC. Our study adds that beyond the technological and active learning aspects of the FC, collaboration is also a key component.
Studies have demonstrated that blended learning environments require collaboration skills for achieving emotionally positive engagement in active learning (Dziuban et al., 2018;Molinillo, Aguilar-Illescas, Anaya-Sánchez, & Vallespín-Arán, 2018). Indeed, when investigating students' perceptions of these components, an important finding of this study is that a very large majority of the students expressed positive attitudes towards teamwork. Teamwork is at the core of an FC and the fact that it affected the students positively in most cases is an indication of the successful implementation of teamwork to enhance active learning during class time. This important finding underscores the centrality that the students attribute to working in teams as a way to achieve meaningful learning outcomes.
Students' highly positive views on teamwork were also manifested in their motivational orientation. This finding supports previous studies (e.g., Seman et al., 2018) that demonstrate how teamwork enhances students' enjoyment of the learning experience, promotes their interest in and engagement with the course material, enhances their emotional belonging to the task, motivates them to achieve their goals, and gives them the important soft-skills required for the 21 st century workforce.
The importance of teamwork was more significant for the FC&PBL students, who were more positive about their experiences while working in teams. This supports the assertion that students who participated in learning of their own choosing successfully regulated their independent learning via team activities (Opdecam et al., 2014). Teamwork provides students with greater learning autonomy, especially in an FC setting -a technology-rich environment that fosters active learning. Such engagement is made possible by restructuring the time and tasks performed in-and out-of-class (Lage et al., 2000).
The most significant finding of this study was the reinforcement of collaboration as reflected by teamwork on problem solving assignments during FC class time. The PBL students demonstrated better performance than their peers in probability, the topic they explored in depth, and more positive responses regarding studying in the FC setting, with an emphasis on the collaborative learning component. We saw that teamwork during PBL empowered the students to take an active role in the learning process, as they needed to share their relevant knowledge with the group in order to complete the final project successfully. Meaningful teamwork occurs when team members rely on and motivate each other in order to successfully complete individual and group tasks (Dym et al., 2006;Molinillo, et. al., 2018;O'Flaherty & Phillips, 2015).

Limitations and Directions for Further Research
This study did not include a control group of students who did not participate in ALE at all, and therefore we cannot claim causality. However, we did explore the effects of participating in an optional PBL assignment within the FC learning environment. According to DeLozier and Rhodes (2017), research is needed to explore the combination of multiple methods of active learning within flipped classrooms, and the suitability of these methods to FC. Our study addresses this issue, as it compares the effects of studying in two active learning settings -FC with and without the PBL component.
Another limitation lies in our participants, who were students in a top tier university. Although they come from diverse backgrounds, these students are considered some of the highest achieving students worldwide. Moreover, the excellent physical and digital infrastructure available to the university, which enabled high-level implementation of a course format, and blending active engagement pedagogies and learning environments in novel ways, cannot be reproduced in many other settings. Thus, it is possible that the success of this research was impacted by our ability to work in a supportive environment, with excellent, motivated, students and faculty. Despite this limitation, we maintain that the combination of FC and PBL can be implemented successfully in other settings as well. We think that other institutions that choose to implement this approach in their teaching and learning environments, especially in mandatory, undergraduate STEM courses, will experience students' cognitive and affective gains that are higher than what can be achieved in traditional settings.

Conclusion and Contributions
Our research demonstrates the importance of active engagement in a technology-based environment in which STEM undergraduate students solve problems and conduct projects. As the number of courses taught in blended learning environments increases, researchers are calling for more investigations into how to best combine technology and face-to-face class time. They identify the evidence-based instruction practices that result in better student learning outcomes (Chen, Breslow, & DeBoer, 2018;Shi et al., 2019). Our study contributes to the body of knowledge on the impact of engaging in ALEs and blended learning environments that combine online learning technological platforms with face-to-face instruction (Lundin, et al., 2018). Our study is unique in its investigation of multiple, combined technological ALE facets, which have been explored and reported separately in the literature. These include (a) engagement in technological ALEs that emphasize in-class collaboration among STEM undergraduate students (Bernard et al., 2014;National Science and Technology Council, 2013); (b) engagement in technological ALEs that incorporate two settings, specifically the FC and the PBL approaches for active learning (DeLozier & Rhodes, 2017;National Research Council, 2012;Reis et. al., 2018); and (c) investigation of various cognitive and affective aspects of STEM undergraduate learning experiences, including performance and perceptions Wang & Zhu, 2019).
Previous research on how the FC is applied in higher education has focused mainly on students' perceptions and satisfaction (Aşıksoy, & Özdamlı, 2016). Higher education is now under pressure to transition from traditional to more active and flexible student-centered teaching and learning, and the FC approach can effectively support this transition provided that policy makers and instructors realize the advantages and challenges involved in adopting it (Akçayır, & Akçayır, 2018). Our study has revealed the ALEs elements that students perceived as the most significant contributors. These findings may provide insight into how to design successful FC courses, which can be even more effective if the course include a PBL element. Learning how to work as a team was found to be a core desired element in the FC setting.
The study also points to the significance of engaging in a project that mimics a real-life industrial setting, which these undergraduate students may encounter in their future professions in STEM companies and industries (Dori & Silva, 2010). Actively engaging in constructive teamwork and real-world projects provides students with the skills necessary for the 21 st century workforce, particularly 'soft skills' such as collaboration and communication (Seman, Hausmann, & Bezerra, 2018). Teamwork lends itself naturally to science and engineering education, as the students will eventually enter a workforce where they will be expected to plan and be involved in projects. Therefore, there is a need to expose them to learning experiences of this kind during their undergraduate studies (Barak, Watted, & Haick, 2016;Crawley et al., 2008). The finding that PBL students expressed positive attitudes towards working in teams is of great importance, as good teamwork skills in general extend beyond the classroom and into professional world. Upon entering the workforce, science and engineering students with excellent training, who also have had positive teamwork experiences, will be in high demand, especially in professions that require interdisciplinary teamwork (Seman, Hausmann, & Bezerra, 2018;Wiggins et al., 2017).
Finally, the methodology developed and presented in this research provides a new framework for evaluating the implementation of a flipped classroom. The evaluation is based on an affective rubric, which we had designed specifically to assess the needs and desires of the students who participate in an unorthodox, novel learning environment, such as FC or FC&PBL. Analyzing the students' statements for both cognitive and affective aspects allowed us to examine the salient features of the FC environment as well as how the students perceived these features. FC designers can benefit from using our affective rubric to identify their students' perceptions.

AUTHOR NOTE
Prof. Yehudit Judy Dori is a faculty member at the Technion -Israel Institute of Technology since 1991. She served as the Dean of Continuing Education and External Studies and later as the Dean of the Faculty of Education in Science and Technology at the Technion. Prof. Dori has also intermittently been a visiting professor and research scholar at the Massachusetts Institute of Technology (MIT). She received the NARST 2020 Distinguished Contributions to Science Education through Research Award (DCRA) as a recognition of her many significant, lasting contributions to the field of science education. Her research on teaching, learning and curricula has impacted how science is taught in the K-12 and higher education. Coinciding with her well-regarded research on visualization, assessment, and metacognition, she has pioneered the field of science education through her notable leadership, which has produced science education researchers and leaders and resulted in international connections that advance the field. Prof. Dori co-edited two books on cognition and metacognition in science, technology, engineering, and mathematics (STEM) education, published numerous journal papers, and mentored about seventy graduate students.
Dr. Zehavit Kohen is Assistant Professor at the Technion -Israel Institute of Technology. Her research focuses on promoting professional development and metacognition of high school mathematics teachers, assessment of science, technology, engineering and mathematics (STEM) education, emphasizing quantitative and qualitative analysis and integrating educational technology into teaching and learning of mathematics. During the academic year 2015-2016, she was a Visiting Scholar at the Center to Support Excellence in Teaching (CSET) at Stanford University, where she investigated the professional development (PD) of early-career mathematics teachers who participated in the Hollyhock Fellowship Program. During 2012-2017, she had been a researcher at the Technion Research and Development Foundation and at the Neaman Institute, Israel.
Brian Rizowy was a M.Sc. student at the Technion -Israel Institute of Technology. His research involved the affective aspects of the flipped classroom at MIT. Figure 6 shows the course homepage, where students can access all the necessary learning materials for the course. Using the left navigation bar, students can transition from one section to another. In total, there were seven sections: 1. Course homepage; 2. Syllabus: course expectations, requirements, grades, links, etc.; 3. Calendar: due dates for the entire 14 weeks of the course; 4. Readings: the Mathematics for Computer Science Textbook ; 5. Lecture notes: PDF versions of the lecture slides that were presented in the videos, the problem set for the next class session, and solutions to the in-class problems, which were posted after the relevant class; 6. Assignments: related to the weekly at-home problem sets, as well as solutions that were posted after the due-date; and 7. Exams: "mini-quizzes" that were given at the beginning of each class and were uploaded along with the solutions. Samples of final exams from previous semesters were also posted. Figure 6. An illustration of the online course homepage Links to the relevant online video lectures (such as the one presented in Figure 7) were found in the "Calendar" or "Lecture Notes" section. The screen shown in Figure 7 focuses on "scheduling" and links the mathematical concept of antichains with a student's experience scheduling courses sequentially in order to fulfill all the program requirements for graduating on time. This is but one example of how the learning materials are connected to students' daily life.

APPENDIX C An Example for an Online Problem Set
The following figures illustrate an online problem set, which is part of Study Unit 1 : Proofs (Figure 8), and the solution that was posted after the due date ( Figure 9).

APPENDIX D An Example of a Final PBL Artifact on the Topic of Gambler's Ruin
The artifact prepared by the students included a real-world problem starting with the question: "Which casino would you choose to play at?" (see Figure 10). Figure 10. An example of a real-world problem created for the final PBL assignment The students then presented the probability question (see Figure 11a) and the mathematical model they used to solve it (see Figure 11b).