Keywords

Exam Wrappers, Metacognition, Student Success, Student Engagement.

Introduction

In the business school, students may have the most challenges with quantitative courses. In the XXX School of Business at XXX University, ODS 333 Operations and Logistics Management covers many quantitative topics such as linear programming. ODS 333 is considered as one of the most challenging core courses by both faculty members and students in the business school. In this course, students are typically assessed by various ways including homework, quizzes, and exams. Although similar questions are given in their exams, students often repeat their mistakes made in previous homework or quizzes. This may suggest that students do not spend sufficient time in reflecting and improving their mastering of topics previously covered. It may also suggest that students need to reconsider their study habits and exam preparations.

The ability of thinking about thinking – metacognition has been identified as a key component in teaching and learning. There are many studies emphasizing the importance of raising metacognitive awareness and using metacognitive approach in education (Djudin, 2017; Poo & Funn, 2017; Breed & Bailey, 2018; Cunningham et al., 2016; Millis, 2016; Siegesmund, 2017; Batson, 2018; Maduabuchi & Angela, 2016.). There are many ways to increase students’ metacognition at a pedagogical level. For example, homework and exam wrappers, pass/fail homework, homework abstracts, self-graded homework, and self-directed homework are five methods of encouraging metacognition and/or reflection found in engineering education literature (Roberts, 2017). Developed by Lovett, an exam wrapper provides students with a structured reflection opportunity about exam performance. It encourages students to look more closely at their returned exams by asking questions about how they prepared, where and why they lost points, and what they planned to do differently for future exams (Lovett, 2013).

Previous studies showed that exam wrappers provide students an intervention to evaluate their learning and to seek for ways for improvement. Exam wrappers address several components of metacognition including assessing strengths and weaknesses, performance evaluation, strategy identification, and creation of behavioral adjustments. They are easy to adopt across different courses and different disciplines. Furthermore, exam wrappers are teacher friendly because they do not take up much class time; they are also student friendly because they do not require much time on the part of students.

Several studies found exam wrappers to be a valued and effective post-exam reflection tool for improving students’ self-reported study habits and students in different grade groups might benefit most from using exam wrappers. For example, exam wrappers were adopted in various engineering courses and found to be highly effective (Chen, 2016). Especially for students who performed the poorest on the first exam, they improved to nearly achieving the class average on the final exam. In another study, exam wrapper assignments were offered as extra credit after the first three exams in a large introductory Food Science and Human Nutrition course. This study found a modest relationship between use of study strategies and improved exam performance particularly for students with a B exam average, suggesting that students in the middle of the grade distribution may benefit most from this type of intervention (Gezer-Templeton et al., 2017).

The benefits of using exam wrappers are not only limited to increase exam scores. Exam wrappers prompt students to review their graded assignments and exams and encourage students to reconsider their study habits and preparations thus make an adjustment to study habits. Students who proactively took the steps for adjustment tend to perceive better outcomes. Exam wrappers are also proven to provide information about students’ understanding of content and level of skills so that appropriate measures and actions can be taken to help students who are struggling in the course.

In order to teach students to be metacognitive learners and create a classroom culture grounded in metacognition therefore ultimately increase student success and help them improve their academic performance, a three-part exam wrapper was developed to use in ODS 333. The first part is to show students a personal prognosis inventory list and ask them to check the items they have used in exam preparation and write down the percentage of their time spent on them. The second part asks students to report the percentage of points they lost due to various reasons given after they look over their graded exams. The last part contains two questions to encourage students coming up with plans to better prepare for the next exam and to work with their professor to facilitate this process.

The exam wrappers were given to three sections of ODS 333 in fall 2018. They were distributed after the first midterm exam. The total number of students in these three sections was 71, out of which 70 students participated. Students’ academic performance in terms of exam scores was monitored and tracked before and after using exam wrappers. The results of exam wrappers, Exam 1 (before the exam wrapper) scores, and Exam 2 (after the exam wrapper) scores were analyzed. The preliminary results showed that majority of students improved their exam performance after implementing exam wrappers. Other metrics of assessing the effectiveness of exam wrappers were explored.

Methodology

Implementation of Exam Wrappers

The XXX School of Business at XXX University has three academic departments and 11 concentrations including Operations and Decision Sciences (ODS). ODS 333 Operations and Logistics Management is a core course in the Bertolon School of Business curriculum. The class size of ODS 333 is small with a cap of 25 students. The majority of the students enrolled in this course are juniors and seniors. In ODS 333, a small proportion of students are concentrated in ODS, while the remaining of the student body is in other concentrations. Traditionally, there are eight sections of ODS 333 each semester taught by both full-time and adjunct faculty members. Students attend two 75-minute lectures per week. ODS 333 is delivered primarily by lectures with some discussion and a group project.

Since ODS 333 is one of the most challenging courses among all business core courses with high DFW rates, faculty has implemented various evidence-based pedagogy to increase student success. For example, MyLab Operations Management – an online teaching and learning platform by Pearson was adopted in 2017 to engage students with immersive content, tools, and experiences. Additionally, the Bertolon School of Business provides students with various learning resources including the peer tutoring service.

In ODS 333, all classes are structured into two segments. There are two 75-minute exams, in which students are only responsible for materials covered in that specific segment. Each exam is worth a total of 100 points and is composed of five problem-solving questions with multiple sub questions. The first segment in ODS 333 contains various topics including operations management, project management, forecasting, and decision making tools, and waiting line models. The second segment contains topics of inventory management, supply chain management, and linear programming models. Linear programming models in the second segment and assessed in the second exam historically has been the most challenging topic in this course. Both exams take place during the regularly scheduled class time. Prior to each exam, a comprehensive exam review class was offered to help students prepare for the exam.

Based on faculty members’ observation, students often repeat their mistakes made in previous homework and quizzes although similar questions are given in their exams. A reflection activity is needed especially evidence-based and pedagogical approach. Exam wrappers is an effective tool to increase metacognition and they ensure students to reflect on three important components of learning: what they know, how well they know it, and subsequently make choices about future learning strategies. As a result, an exam wrapper was developed for ODS 333 to increase students’ metacognitive awareness and self-regulated learning. It was used in the fall semester of 2018 in three sections of ODS 333. There were 71 students in three sections combined. Students were given exam wrappers in the following class after taking the first exam. Consent forms releasing the data from both exam wrappers and exam grades were distributed along with the exam wrappers. Students completed exam wrappers in class and understood that returning exam wrappers was voluntary. A total of 70 exam wrappers were collected at the end of the class.

Content of Exam Wrappers

The exam wrapper developed in this study includes three parts. The first part is personal prognosis inventory providing students with a list of behaviors that they should exhibit in order to succeed in ODS 333. Students were asked to check items they had used and write down the percentage of their time spent on them. This part of the exam wrapper gives students a chance to reflect on the efficiency of their exam preparation. Most importantly, the list serves as a guideline of potential learning strategies students could adopt in the future.

The second part of exam wrapper is to ask students to look over their graded exam and categorize what kind of mistakes they made and estimate the respective percentages. Mistakes include trouble with applying the definitions, lack of understanding of the concepts, not knowing how to approach the problem, careless mistakes and so on. By going over a list of mistake types, students got to reflect their exam performance and it was hoped to avoid these mistakes in the future. Additionally, students were asked to enter how many hours they had spent on preparing for the exam.

The last part of exam wrapper contains two open-ended questions. The first question asks students to name at least three things they would do differently in preparing for the next exam. For instance, spending more time, changing a specific study habit, or going to office hours and so on. The second question asks students to make a suggestion to their professor on how to help them learn better and prepare better for the next exam. Both questions were designed to provide students and faculty a path forward making collaborated efforts to achieve academic and teaching excellence.

Special consideration was made when designing the first two parts of the exam wrapper. In the first and second parts, students not only need to check off the listed items, they also need to estimate a percentage of time spent in part one and points lost in part two. These two parts in the exam wrapper prompt students to provide feedback on their exam preparation as well as the corresponding exam performance. Students were expected to learn from these two parts that there was a great selection of resources available and they should avoid making the same mistakes occurred in the past. With the addition of the last part of exam wrapper, a leaning strategy was made to facilitate exam preparation therefore increase student success.

Data Collection and Analysis

In ODS 333, students take two non-cumulative exams during a typical 15-week semester. Data collection was conducted in fall 2018. Students were informed about this research project which uses classroom surveys and their exam performance data via consent approved by IRB. About a week after the first exam, exam questions were gone over in class and students got to look at their exam performance including their scores. On the same day, exam wrappers were distributed and collected. It was well communicated to students that returning exam wrappers was voluntary and it had no impact to their exam scores. Student responses and their exam scores were analyzed. Table 1 shows an overview of the analysis. It was noted that the score of Exam 2 did increase slightly compared to.

Exam 1

Results and Insights Gained

Effect of Study Habits on Exam Performance

In Part One of the exam wrapper, students reflected how they had prepared for the exam by checking recourses they used and providing the percentage of time spent on them. The result is summarized in (Table 2) below. The relationship of the top three recourses and Exam 1 performance was explored. The coefficients of correlation are r = −0.11 (Use Pearson MyLab “Question Help” button), r =0.18 (Attend the exam review class), and r =0.1 (Re-do all/some of the homework problems before the exam) respectively. “Attend the exam review class” and “Re-do all/some of the homework problems before the exam” undoubtedly increased exam performance, however “Use Pearson MyLab ‘Question Help’ button” had a negative impact on exam performance. This might be because students depended heavily on the “Help” function built in Pearson MyLab Operations Management. Without “Help” function during the exam, students lost both proficiency and confidence.

Effect of Exam Preparation on Exam Performance

In Part two of the exam wrapper, students entered how many hours they had spent on preparing for Exam 1. The average was 3.72 hours. The correlation of total hours spent on exam preparation and the exam score is r =0.15, which indicates that the amount of study time spent on exam preparation increased exam performance.

Exploring Mistakes Made

Additionally, in Part two of the exam wrapper, students categorized mistakes they made in the exam based on various reasons given. The result is summarized in Table 3 below. Optimistically, students who made “Careless mistakes” or “Did not write notes down on the cheat sheet” were expected to avoid these mistakes in future exams. Or at least those students were alerted after completing this part of the exam wrapper. Because various factors could contribute to “Did not have time to work on/complete the problem”, many learning strategies could improve or even eliminate repeating this mistake in the future.

Exploring the Impact on Different Grade Groups

Exams count for 40% of the total grade in ODS 333. There are other grading items including Homework (20%), Attendance, Participation, and Professionalism (20%), and Course Project (20%). Further insights were gained from exploring other metrics to assess the effectiveness of exam wrappers.

Table 4 suggests that students across all grade levels increased their scores on Exam 2 ranging from 1.26% to 45.90%. It was learned that students with lower letter grades tended to benefit more from using exam wrappers. Students earned D- to D+ increased their scores by approximately 10%. And students who failed this course increased their scores even more by over 45%, although the increases were not sufficient enough for them to pass this course. Using other intervention approaches to help students pass this course might be promising.

Effect of Improvement on Other Assessment Performance

Exam wrappers were distributed in three sections of ODS 333. Individual section data is shown in Table 5. In this analysis, “Improvement” is defined as the difference of Exam 2 score minus Exam 1 score greater than zero. Students were classified into those who made an improvement in Exam 2 (“Improvement”) and those whose score did not change or did not increase (“No improvement”). Overall, there were more students made an improvement, but the margin was minor. One interesting observation was that the majority of students with “No improvement” scored higher in exams than students with “Improvement”. A reasonable explanation is that Exam 2 was challenging in nature and it was difficult to score higher than Exam 1 even for students with better academic performance.

To summarize the results of various analysis, exam wrappers did improve the average score of Exam 2. The increase was marginal, but Exam 2 was a more difficult exam compared to Exam 1. The same conclusion was given by ANOVA that there was no significant difference between Exam 1 and Exam 2 grades, p = 0.37.

We also gained some insights from the following observations:

1. The top three study habits students showed in exam preparation were: Use Pearson MyLab “Question Help” button, Attend the exam review class, and Re-do all/some of the homework problems before the exam. While the other two study habits correlated to exam performance in a positive way, using Pearson MyLab “Question Help” button too much resulted in dependence and negatively impacted exam performance (Chew et al., 2016).

2. The top three mistakes students made in Exam 1 were: Careless mistakes, Did not have time to work on/complete the problem, and Did not write notes down on the cheat sheet. Students were believed to avoid repeating the same mistakes if they learned a lesson. For example, “Careless mistakes” could be easily avoided by changing the attitude and paying attention to details. Increasing proficiency could keep away from making mistakes such as “Did not have time to work on/complete the problem”. Finally, “Did not write notes down on the cheat sheet” could be eliminated by attending the exam review class.

3. The total hours spent on exam preparation and exam score correlated positively although the correlation was not significantly strong. It suggested that spending more time preparing for the exam might be helpful but it might also require a robust study strategy.

4. It was noted that students in the lower grade group tended to benefit more from using exam wrappers. Similarly, students with lower grades in each category (exams, homework, attendance, and course project) benefited more compared to students scored lower in each category. Most likely students with better grades had already practiced reflection and improved self-regulated learning.

5. In each of the three sections of ODS 333 in this study, the majority of students improved their exam performance after implementing exam wrappers. This proved the effectiveness of exam wrappers. But after taking a closer look at the improvement, it was learned that the improvement was marginal. In the literature, similar finding was found showing the use of exam wrappers did not increase exam scores, final grades, or Metacognitive Awareness Inventory (MAI) scores (Soicher & Gurung, 2017). Another study also reached the same conclusion that exam wrappers did not have a significant impact on overall exam performance (Liao et al., 2018). In Lovett’s early study of exam wrappers, it was suggested that an increase in metacognition ratings only improved for students using exam wrappers in more than one course during a semester (Lovett, 2013).

Conclusion

In an effort to help students observe their study habits, exam preparation, and self-intervention other than simply looking at their earned scores, a metacognitive activity called exam wrapper was implemented in a junior/senior level core business course in the quantitative area.

In this paper, the relationship between exam wrappers and exam scores was explored. Exam wrappers were given to three sections of ODS 333 in fall 2018. A total of 70 exam wrappers were collected, out of which 67 were valid data points in this analysis. In each of the three sections, the majority of students improved their exam performance after implementing exam wrappers. This proved the effectiveness of exam wrappers although the improvement was marginal. An interesting observation was that students in the lower grade group tended to benefit more from using exam wrappers. Also, students with lower grades in each category (exams, homework, attendance, and course project) benefited more. Similar conclusion was found in the literature. Although exam wrappers did not seem to increase exam scores significantly, filling out exam wrappers did provide students an opportunity of reflection and adjustment in general. It was noted that exam wrapper was a good exercise to assist students in developing metacognitive skills in this quantitative business course.

There are some limitations of this study. Exam wrappers were only given to students after Exam 1. Further analysis could be done if exam wrappers were given to students after the second exam and additional data was collected. Exam score was the only measurement used in this study to evaluate the effectives of exam wrappers. Other measurements could be used to further assess the effectiveness. For example, metacognition information using Metacognitive Awareness Inventory (MAI) scores might be a good option. This paper was also limited by conducting quantitative analysis only. The effectiveness of exam wrappers could be assessed qualitatively by anonymously surveying students their perceptions at the end of the semester. For example, an exit survey with both likert-scale and open ended comments could be used to collect student satisfaction information. Additionally, this study was a one-course and one-semester effort. Extending the study to a multiple-course and multiple-semester scale would be a future research direction. Introducing demographics data (gender and ethnicity) to the current data collection effort would be another future research direction.

References

Batson, B. (2018). Introducing metacognition to sophomores and juniors and its effect on academic performance. In 2018 ASEE Annual Conference & Exposition.

Indexed at, Google Scholar, Cross Ref

Breed, B., & Bailey, R. (2018). The influence of a metacognitive approach to cooperative pair problem-solving on self-direction in learning. TD: The Journal for Transdisciplinary Research in Southern Africa, 14(1), 1-11.

Indexed at, Google Scholar, Cross Ref

Chen, J. (2016). Effective and adoptable metacognitive tools. In 2016 ASEE Annual Conference & Exposition.

Indexed at, Google Scholar, Cross Ref

Chew, K.J., Sheppard, S., Chen, H.L., Rieken, B., & Turpin, A. (2016). Improving students’ learning in statics skills: Using homework and exam wrappers to strengthen self-regulated learning. In 2016 ASEE Annual Conference & Exposition.

Cross Ref

Cunningham, P., Matusovich, H.M., Morelock, J.R., & Hunter, D.A.N. (2016). Beginning to understand and promote engineering students' metacognitive development.

Indexed at, Google Scholar, Cross Ref

Djudin, T. (2017). Using metacognitive strategies to improve reading comprehension and solve a word problem. Journal of Education, Teaching and Learning, 2(1), 124-129.

Indexed at, Google Scholar, Cross Ref

Gezer-Templeton, P. G., Mayhew, E. J., Korte, D. S., & Schmidt, S. J. (2017). Use of exam wrappers to enhance students’ metacognitive skills in a large introductory food science and human nutrition course. Journal of Food Science Education, 16(1), 28-36.

Indexed at, Google Scholar, Cross Ref

Liao, J., Kunberger, T., Papkov, G.I., Badir, A., O'Neill, R., & Nguyen, L.D. (2018). Exam wrappers, reflection, and student performance in engineering mechanics. In 2018 ASEE Annual Conference & Exposition.

Indexed at

Lovett, M.C. (2013). Make exams worth more than the grade: Using exam wrappers to promote metacognition. Using reflection and metacognition to improve student learning: Across the disciplines, across the academy, 18-52.

Maduabuchi, C.H  & Angela, I.O.I. (2016). Teaching metacognitive skills for the promotion of self-regulated learning among secondary school students in nigeria.  British Journal of Education, 4(10).

Millis, B.J. (2016). Using metacognition to promote learning. IDEA Paper# 63. IDEA Center, Inc.

Indexed at, Google Scholar

Poo, S.M., & Funn, P.L. (2017). Using metacognitive awareness in learning vocabulary with cloud-based immersive technology. International Journal of Information and Education Technology, 7(2), 114.

Indexed at, Google Scholar, Cross Ref

Roberts, M.W. (2017). Fostering reflection and metacognition with engineering homework. In ASEE rocky mountain section conference, 2017, USA.

Google Scholar

Siegesmund, A. (2017). Using self-assessment to develop metacognition and self-regulated learners. FEMS microbiology letters, 364(11).

Indexed at, Google Scholar, Cross Ref

Soicher, R.N., & Gurung, R.A. (2017). Do exam wrappers increase metacognition and performance? A single course intervention. Psychology Learning & Teaching, 16(1), 64-73.

Google Scholar

Received: 09-Sep-2022, Manuscript No. AELJ-22-12535; Editor assigned: 10-Sep-2022, Pre QC No. AELJ-22-12535 (PQ); Reviewed: 23- Sep-2022, QC No. AELJ-22-12535; Revised: 27-Sep-2022, Manuscript No. AELJ-22-12535 (R); Published: 30-Sep-2022