Introduction

Recently, the use of computers and digital devices with internet connection has become popular in many aspects of the life, such as social activities, business, and education (Bedi, 2014; Colbert, Yee & George, 2016). For education, the rapid development of the information technology drastically changes teaching-learning methods based on the use of digital devices (Yousafzai, Chang, Gani & Noor, 2016). In particular, e-learning (Cidral, Oliveira, Di Felice & Aparicio, 2018; Zhang & Nunamaker, 2003) is being shifted toward mobile learning (m-learning) (Al-Emran, Mezhuyev & Kamaludin, 2018; Liu & Carlsson, 2010) by employing mobile devices (i.e., cell phones, laptops, tablets, MP3 players, PDAs, etc.) (Nedungadi & Raman, 2012; Nordin, Embi & Yunus, 2010). With the help of mobile devices, m-learning emerges as an advantage pedagogical method to improve educational outcomes (Böhm & Constantine, 2016; Crompton, Burke & Gregory, 2017). For instance, it is stated that the limits of human-computer interaction community in e-learning can be overcome by using m-learning (Gay, Stefanone, Martin & Hembrooke, 2001). M-learning is an advantageous approach that also helps teachers and learners to overcome the difficulties in e-learning method, such as the dependence on places of computer rooms, the limitation of the computer quantity, or schedule for using computers (Fotouhi et al., 2011; Klopfer, 2008). In addition, m-learning enables students to access demanded knowledge or information everywhere and anytime (Korucu & Alkan, 2011). Therefore, m-learning is strongly suggested to be enrolled into education processes.

To perform successfully m-learning in schools, together with hardware (i.e., laptops, smart phone, tablets, etc.), computer software must be taken into account. It was found that digital game programs can be integrated into m-learning to improve students’ interaction with the contents of lessons (Burguillo, 2010; Erhel & Jamet, 2013; Herodotou, 2018; Huizenga et al., 2009; Wang, 2015). In particular, Kahoot (Jones et al., 2019) emerges as a good tool for m-learning in various subjects, from social to natural sciences. Indeed, many studies (Licorish et al., 2018; Ismail et al., 2019; Plump et al., 2017; Tóth et al., 2019) stated that Kahoot much improved students’ motivation and study perception. Besides, (Muhridza et al., 2018) suggested that Kahoot can could support the initiating and fostering engagement of learners. In another work, it was observed that students were motivated to learn with Kahoot (Idowu, Nat & Kissi, 2020).

Although several works (Owen & Licorish, 2020; Wang & Tahir, 2020) have studied the effectiveness of Kahoot on m-learning of students, from keen observation, it seems that most of them have focused on the perception and satisfactory of students. The results obtained by using the survey approach of such studies strongly depend on individual opinions of participants. Hence, study methods, which can provide impersonal results, are highly demanded to overcome the limitation of the aforementioned survey approach. It was found that the quantitative analysis on exam scores of students could be applied for the purpose (Baszuk & Heath, 2020; Iwamoto et al., 2017). It should be emphasized that experimental data as evidence to clearly indicate the positive impact of Kahoot on exam scores of students are still very limited. Besides, the effectiveness of Kahoot on scores in various subjects has not been well investigated so far. Therefore, we conducted a study to answer the question that how Kahoot impacts exam scores of students in Physics at high schools. Subsequently, the effective- ness of Kahoot on learning results of students can be quantitatively and impersonally investigated in the present study.

Materials and Methods

The random sampling method (Cohen, Manion & Morrison, 2017; Taherdoost, 2016) was employed to choose four classes based on the learning results of students in previous semesters. In this method, the sample size is deemed adequate for the chosen (purposive) sampling method as the possible pool of participants is already restricted (Byrne, 2001). This study has been carried out with a sample of 138 Grade-10 (from 16 to 17 years old) for the evaluation of the use of the Kahoot tool in Physics for six lessons in the second semester of the 2019-2020 school years. The participants were separated into two groups, who studied with (39 males, 30 females) and without Kahoot (35 males, 34 females). It is noticed that all the students relatively have similarity of Physics background, ages, and results of the first semester in the same school year. None of the participants had previous experience of using Kahoot.

In this study, both groups received the same learning content, number, and substance of multiple-choice exercises used in the teaching process for six weeks. Multiple-choice exercises were used for warming-up, introducing new concepts, summary of lessons, and so on. But the ways to solve these multiple-choice exercises were different for student groups. Kahoot was employed for the first group while the traditional education method was used for the second one.

Questions related to the lessons on Kahoot have been compiled and archived in My Kahoots. To organize teaching with Kahoot, the teacher logged in, opened the question package built on My Kahoots, and selected Play to provide a PIN for students to join. Students of the group using Kahoot need to prepare mobile devices (smart phones or tablets) with internet connection. After that, they accessed the link http://kahoot.it to enter the PIN code that the teacher provided.

The contents of questions were presented to the students by Kahoot through texts, pictures, and/or videos. Teachers used Kahoot to show a contextualization of information, quiz, or poll. Depending on the lesson contents, teachers organized a game for students to answer questions in individual or team working. For instance, the organization of the lesson on the gravitational potential energy can be organized as follows. The instructors assigned the contents to students in the two groups. For the group using Kahoot, each content of the lesson was introduced via videos, drawings, or graphs, as illustrated in panels (A) and (B) of Figure 1. To evaluate the results of this activity, teachers set up Team mode for students to complete questions online using Kahoot on the mobile devices.

To attract attention and promote the activities of student in study, by using Assign challenge option in Kahoot, teachers created a competition among students. The results of students on the multiple-choice questions could be rated. As examples, Figure 2 shows the screens of the rated results for students who studied with Kahoot for the knowledge of concepts of momentum, the law of momentum conservation, work, kinetic energy, potential energy, and the law of conservation of mechanical energy.

Activities Using Kahoot

To evaluate the effectiveness of using Kahoot in implementing m-learning and teaching Physics at high schools, we employed the quantitative research method (McKim, 2017). The evaluation was based on the scores of students in three tests. These tests helped in making comparisons between two groups, and both genders. The first test was conducted after students had finished the third lesson, Kinetic energy. The second one was taken place one week later when the fourth lesson had been studied. When the students had finished all the six lessons, the last test, which was the major one, was performed. The time schedule for the tests is shown in Figure 3.

The influence of m-learning (Bicen & Kocakoyun, 2018) using Kahoot on the insights of students into the lessons was quantitatively evaluated based on the scores of students, which were classified into five levels: poor scores (0.0–3.4), weak scores (3.5–4.9), average scores (5.0 –6.4), fair scores (6.5–7.9), and good scores (8.0–10). The learning quality of students was evaluated based on the classification following in accordance with the educational standard in Vietnam (MOET, 2011). The scores of students were collected and the quality was identified using this standard. The difference in scores of students in the two groups was analyzed using t-test (David & Gunnink, 1997) under help of the SPSS software (Bryman & Cramer, 2012). Independent samples t-test was employed to compare the scores of females and males in Kahoot group and between two groups, who studied with (Kahoot) and without Kahoot (non-Kahoot). Finally, we could obtain the results as described in the next section.

Results and Discussion

The first test (being provided upon request) was conducted after the lesson of kinetic energy (Halliday, Resnick & Walker, 2013), which was the third lesson that students studied with Kahoot. This test was to evaluate the insight of students into the concept, putting formulae and kinetic theorem into practice. Figure 4 presents the distribution of scores (Xi) [left panel] and the change in the score at Xi [right panel] for the first test of students, who studied with and without Kahoot. The results in the left panel indicate that, after studying with Kahoot (solid red curve), the scores at Xi <7.0 are decreasing by up to 50% as shown in the right panel. In contrast, the scores at Xi >7.0 are increasing by up to 50%. Notice that the change in the score at Xi was estimated based on a ratio F, which is defined as

Where N_K. and NnonK. are the numbers of students, who get the score at X_i, studying with and without Kahoot, respectively? It is found that after the third lesson, students can understand well the knowledge of the lesson. The increasing number of scores Xi=8,9,10 reflects that Kahoot enhances the quality of teaching-learning process.

Similarly, another test (being provided upon request) was following the first one after the fourth lesson, which was the potential energy. Figure 5 shows the change in students’ score distribution. It can be seen in the left panel that the low scores with X_i<7.0 are continuously decreasing while the high ones are increasing. By comparing with the first test, the number of scores at X_i=9,10 are much larger than that after the third lesson. For instance, by studying with Kahoot, three students got scores at X_i=10 in the second test while only one student at this score was found in the first test. On the other hand, as can be seen in the right panel, the scores at X_i=5 (or X_i=10) are drastically reduced (or increasing) by a factor of 2 (or by 67%) when the students studied with Kahoot.

Figure 6 presents the scores obtained in the major test (being provided upon request), which was conducted after students studied six lessons with Kahoot. It clearly shows that when students studied with Kahoot there is no significant change in the scores around 7.2 (at the good level) while the low scores with X_i<7.0 (or high scores with X_i>7.5) are drastically decreasing (or increasing).

We also performed a statistical analysis using the SPSS program (Bryman & Cramer, 2012) to clearly determine the effects of Kahoot on the study outcomes of students. Figure 7 shows the results of comparisons between the scores of two groups, who studied with (Kahoot) and without Kahoot (non-Kahoot). Table 1 represents the analyzed values of the exam scores with 95% of the confidence interval. It is found that the scores of the Kahoot students distribute in the higher ranges compared to those of the other group for the first and the major tests. For the second test, although the median values are almost the same, the lower bound scores of the Kahoot group are still higher than that of the students studied without Kahoot. It is clearly that the lower bound, mean, minimum, maximum, and median score values of students are significantly improved by using Kahoot. Additionally, with the small p-values (p<0.05) of the ttest, the result of the statistical analysis also indicates that there are significant difference between the scores of the two groups, especially for the first and the major tests. This strongly reflects the impact of Kahoot on exam scores of students.

Table-1 The Mean, Lower Bound, Upper Bound, Minimum, Maximum and Median Values Determined Using The T-Test With 95% of The Confidence Interval
Test	1st test		2nd test		Major test
Parameters	nonKahoot	Kahoot	nonKahoot	Kahoot	nonKahoot	Kahoot
Mean	7.188	7.614	6.884	7.328	7.078	7.503
Lower bound	6.891	7.329	6.569	7.020	6.772	7.214
Upper bound	7.486	7.899	7.199	7.637	7.385	7.792
Min	4.0	5.0	4.0	5.0	4.3	4.7
Max	10	10	10	10	9.7	9.7
Median	7.000	8.000	7.000	7.070	7.300	7.700

By taking the scores of all the tests after learning six lessons using Kahoot, the average scores are considered to examine the effect of the software in the m-learning for students. Notice that the average scores ( i) are defined as

Where Xi1, Xi2, and Xi3 are the scores of the students in the first, second, and major tests, respectively. The average exam scores are presented in Table 2. It is found that the scores at Xi=5.5–5.9, 6.5–6.9, and 7.0–7.4 are decreasing by 100%, 36%, and 78%, respectively. The high scores with Xi=7.5–7.9, 8.0–8.4, and 9.0–9.4 are increasing with 29%, 73%, and 67%, respectively. The reduction (and increase) of the low (and high) scores with Xi<7.5 (and Xi>8.0, except for 8.5–8.9) reflects the positive effect of Kahoot on the learning results of students.

Table-2 The Number of Students, Who Studied With And Without Kahoot, At The Average Scores (X¯I) of The Three Tests. Notice That The Negative Sign in The Last Column Indicates The Reduction; Nk. and Nnonk. Denote The Number of Students Who Studied With and Without Kahoot, Respectively
Average score X¯i	NnonK.	NK.	Score change F (%)
0.0 – 4.4	0	0	N/A
4.5 – 4.9	3	0	N/A
5.0 – 5.4	1	1	0
5.5 – 5.9	8	4	-100
6.0 – 6.4	4	4	0
6.5 – 6.9	15	11	-36
7.0 – 7.4	16	9	-78
7.5 – 7.9	10	14	29
8.0 – 8.4	3	11	73
8.5 – 8.9	7	7	0
9.0 – 9.4	2	6	67
9.5  – 10	0	0	N/A

To evaluate the change in the learning results of each student, who was affected by the time of using Kahoot in the learning process, we compare the average scores of the two first tests to the scores in the major test. Figure 8 shows the scores Xi of each student, who studied with Kahoot. It is found that the trends of the score change of students are different from each other.

This difference indicates that the impact of Kahoot strongly depends on individual ability of students. This result is consistent with the study conducted by (Owen et al., 2020), stating that Kahoot enhances attention and motivation for students, and its effectiveness on the retention depends on situational and individual factors. Hence, it is important to classify students who have similar abilities into the same class for applying Kahoot in the m-learning process. Besides, most scores (about 55%) of the major test are higher than the average scores of the two first tests. Notice that the two first tests were conducted after four lessons (at the fourth school hour) while the major test was performed after six lessons (in the sixth school hour). This result reflects the dependence of efficiency of using Kahoot on the time that students study with the software. Hence, this study also gives more evidence for the effect of the previous experience with mlearning on the outcomes of students. In other words, the present work suggests that the previous experience can be one of the variables impacting on the usefulness of m-learning since it is thought to be a factor affecting the efficiency in use of e-learning (Doulik, Skoda & Simonova, 2017).

The impact of gender on the effectiveness of Kahoot was also investigated by comparing the change in the average scores in each test of the female to those of the male students. The score change was determined from the differences between the average scores of the Kahoot group and those of the nonKahoot group for females and males, which is given by

Figure 9 shows the improvement of the study outcomes by Kahoot for male and female students. It was found that the scores of the female students are increased by 8.7%, 5.5%, and 8.5% while those of the males are improved by 7.4%, 5.6%, and 5.7% for the first, second, and major tests, respectively. In general, the scores of the females are higher than those of the males, especially for the major test, when both of them employed Kahoot for their studies. The higher scores indicate that the females had better perceptions and attention on the contents of lessons than did the males. In other words, Kahoot significantly affects the learning performance of the females more than that of the males. Subsequently, the gender is an important factor impacting the effectiveness of Kahoot on studying Physics at high schools. The result in the present study is in a good agreement with those reported in previous works by (Hou, 2019; Tsai, 2017).

Conclusion

In this study we evaluated the change in the exam scores to evaluate the impacts of Kahoot on the study outcomes of high-school students in Physics. The results show that Kahoot is helpful for learners to improve their understanding in lessons. We found that the exam scores drastically shift from the low (3-5) toward the high (7-10) ranges if Kahoot is employed for teaching and learning Physics. The benefits of Kahoot are different for each student, which depend on the using time and study ability. Additionally, the gender also strongly impacts the effectiveness of Kahoot on the study outcomes of students. On the other hand, the better learning results also indicate that the usefulness of mobile devices in classrooms is enhanced by using Kahoot. This game-based learning tool supports well not only for teaching languages or subjects in social sciences but also for Physics. The present study is helpful for better understanding of the influences of Kahoot on m-learning process.

Acknowledgement

We gratefully thank the dean and students at Vo Van Kiet high school in Kien Giang province, Vietnam, for their voluntary participation. This work was supported by Scientific Research Project - Hue University under Grant number of DHH2020-03-134.

Contribution/Originality

This study is finding clearly the benefits of Kahoot on teaching Physics in high schools in Vietnam. It will provide relevant information for understanding the impacts of Kahoot on students’ learning outcomes.

References

1. Al-Emran, M., Mezhuyev, V., & Kamaludin, A. (2018). Technology acceptance model in m-learning context: A systematic review. Computers & Education, 125, 389–412.
2. Baszuk, P.A., & Heath, M.L. (2020). Using Kahoot! To increase exam scores and engagement. Journal of Education for Business, 95(8), 548-552.
3. Bedi, K. (2014). Tablet PC & smartphone uses in education (TabletTours). In 2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 940-945. IEEE.
4. Bicen, H., & Kocakoyun, S. (2018). Perceptions of students for gamification approach: Kahoot as a case study. International Journal of Emerging Technologies in Learning, 13(2), 72-93.
5. Böhm, S., & Constantine, G.P. (2016). Impact of contextuality on mobile learning acceptance. Interactive Technology and Smart Education, 13(2), 107–122.
6. Burguillo, J. (2010). Using game theory and competition-based learning to stimulate student motivation and performance. Computers & Education, 55, 566–575.
7. Bryman, A., & Cramer, D. (2012). Quantitative data analysis with IBM SPSS 17, 18 & 19: A guide for social scientists. Routledge.
8. Byrne, M. (2001). Sampling for qualitative research. AORN journal, 73(2), 494-494.
9. Cidral, W.A., Oliveira, T., Di Felice, M., & Aparicio, M. (2018). E-learning success determinants: Brazilian empirical study. Computers & Education, 122, 273–290.
10. Cidral, W.A., Oliveira, T., Di Felice, M., & Aparicio, M. (2018). E-learning success determinants: Brazilian empirical study. Computers & Education, 122, 273–290.
11. Cohen, L., Manion, L., & Morrison, K. (2017). Research methods in education. Routledge.
12. Colbert, A., Yee, N., & George, G. (2016). The digital workforce and the workplace of the future. Academy of Management Journal, 59(3), 731–739.
13. Crompton, H., Burke, D., & Gregory, K.H. (2017). The use of mobile learning in PK-12 education: A systematic review. Computers & Education, 110, 51–63.
14. David, H.A., & Gunnink, J.L. (1997). The paired t test under artificial pairing. The American Statistician, 51(1), 9-12
15. Doulik, P., Skoda, J., & Simonova, I. (2017). Learning styles in the e-learning environment: The approaches and research on longitudinal changes. International Journal of Innovation and Learning, 21(4), 417-434.
16. Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156–167.
17. Fotouhi, G,F., Earnshaw, R.A., Moeini, A., Robison, D., & Excell, P.S. (2011). From e-learning to m-learning-the use of mixed reality games as a new educational paradigm. International Journal of Interactive Mobile Technologies (iJIM), 5(2), 17-25.
18. Gay, G., Stefanone, M., Grace-Martin, M., & Hembrooke, H. (2001). The effects of wireless computing in collaborative learning environments. International Journal of Human-Computer Interaction, 13(2), 257–276.
19. Halliday, D., Resnick, R., & Walker, J. (2013). Fundamentals of physics. John Wiley & Sons.
20. Herodotou, C. (2018). Mobile games and science learning: A comparative study of 4 and 5 years old playing the game Angry Birds. British Journal of Educational Technology, 49(1), 6–16.
21. Hou, Y.J. (2019). Gender difference in language learning with technology. In International Conference on Human-Computer Interaction, Springer, Cham, 256-265.
22. Huizenga, J., Admiraal, W., Akkerman, S., & Dam, G.T. (2009). Mobile game-based learning in secondary education: Engagement, motivation and learning in a mobile city game. Journal of Computer Assisted Learning, 25(4), 332–344.
23. Idowu, A., Nat, M., & Kissi, P.S. (2020). Student perception of usefulness and ease using Kahoot, a free web-based tool in a tertiary education setting. Acta Scientiarum. Technology, 43, 47347.
24. Ismail, M.A.A., Ahmad, A., Mohammad, J.A.M., Fakri, N.M.R.M., Nor, M.Z.M., & Pa, M.N.M. (2019). Using Kahoot! as a formative assessment tool in medical education: a phenomenological study. BMC medical education, 19(1), 1-8.
25. Iwamoto, D.H., Hargis, J., Taitano, E.J., & Vuong, K. (2017). Analyzing the efficacy of the testing effect using KahootTM on student performance. Turkish Online Journal of Distance Education, 18(2), 80-93.
26. Jones, S.M., Katyal, P., Xie, X., Nicolas, M.P., Leung, E.M., Noland, D.M., & Montclare, J.K. (2019). A ‘KAHOOT!’approach: The effectiveness of game-based learning for an advanced placement biology class. Simul Gaming, 50 (6), 832-847.
27. Klopfer, E. (2008). Augmented learning: Research and design of mobile educational games. MIT press.
28. Korucu, A.T., & Alkan, A. (2011). Differences between m-learning (mobile learning) and e-learning, basic terminology and usage of m-learning in education. Procedia-Social and Behavioral Sciences, 15, 1925-1930.
29. Licorish, S.A., Owen, H.E., Daniel, B., & George, J.L. (2018). Students’ perception of Kahoot!’s influence on teaching and learning. Research and Practice in Technology Enhanced Learning, 13(1), 1-23.
30. Liu, Y., Li, H., & Carlsson, C. (2010). Factors driving the adoption of m-learning: An empirical study. Computers & Education, 55(3), 1211–1219.
31. McKim, C.A., (2017). The value of mixed methods research: A mixed methods study. Journal of Mixed Methods Research, 11(2), 202-222.
32. The ministry of education and training. (2011). Circular promulgating regulation on assessment and classification of secondary and high school students. (MOET Publication No. 58/2011/TT-BGDDT in Vietnamese).
33. Muhridza, N.H.M., Rosli, N.A.M., Sirri, A., & Samad, A.A. (2018). Using game-based technology, Kahoot! for classroom engagement. LSP International Journal, 5(2), 37-48.
34. Nedungadi, P., & Raman, R. (2012). A new approach to personalization: integrating e-learning and m-learning. Educational Technology Research and Development, 60(4), 659-678.
35. Nordin, N., Embi, M.A., & Yunus, M.M. (2010). Mobile learning framework for lifelong learning. Procedia-Social and Behavioral Sciences, 7, 130-138.
36. Owen, H.E., & Licorish, S.A. (2020). Game-based student response system: The effectiveness of kahoot! on junior and senior information science students’ learning. Journal of Information Technology Education: Research, 19, 511-553.
37. Plump, C., & LaRosa, J. (2017). Using Kahoot! in the classroom to create engagement and active learning: A game-based technology solution for e-learning novices. Management Teaching Review, 2(2), 151–158.
38. Taherdoost, H. (2016). Sampling methods in research methodology; How to choose a sampling technique for research, International Journal of Academic Research in Management, 5(2).18-27.
39. Tóth, A., Logo, P., & Logo, E. (2019). The effect of the kahoot quiz on the student's results in the exam. Periodica Polytechnica Social and Management Sciences, 27(2), 173-179.
40. Tsai, F.H. (2017). An investigation of gender differences in a game-based learning environment with different game modes. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 3209-3226.
41. Wang, A.I. (2015). The wear out effect of a game-based student response system. Computers & Education, 82, 217–227.
42. Wang, A.I., & Tahir, R. (2020). The effect of using Kahoot! For learning–A literature review. Computers & Education, 149, 103818.
43. Yousafzai, A., Chang, V., Gani, A., & Noor, R.M. (2016). Multimedia augmented m-learning: Issues, trends and open challenges. International Journal of Information Management, 36(5), 784-792.
44. Zhang, D., & Nunamaker, J.F. (2003). Powering e-learning in the new millennium: An overview of e-learning and enabling technology. Information Systems Frontiers, 5(2), 207–218.