The web-site is now in readonly mode. Login and registration are disabled. (28 June 2019)

MON: Evaluation of game-based SRS (Student Response Systems) as OER and examples of open practices in secondary education (Ognjen Vukas)

Poster in Sway Accessibility View: https://sway.office.com/GUw4yiiLflMiZx6J?accessible=true

Cloud created by:

Ognjen Vukas
20 January 2019

I intend to explore the theme of innovation in the world of edtech start-ups such as the popular quizzing applications Kahoot, Quizlet Live, Socrative and Quizizz (Rose, 2017), and assess their suitability to support open practices in secondary education. 

Education is the third most popular category in the Apple iOS App Store (Statista, 2018). This illustrates the growth of the market in edtech apps and services, which is also evident from the billions in venture capital funding that thousands of edtech start-ups have attracted (Berger, 2017). Many of these start-ups have adopted strategies which rely on freemium business models and user generated content. Game-based SRS have used gamification and the BYOD (Bring Your Own Device) trend to achieve widespread adoption, especially in secondary education. These apps build on the tradition of Electronic Voting Systems (EVS) or clickers which first appeared in the 1960s and since then have been used in both HE and in compulsory education (Deal, 2007). 

From the open education perspective, these are interesting developments as edtech companies are creating platforms for the free sharing of resources amongst teachers and other training practitioners. The vast majority of 51 million public Kahoot quizzes (Harrell, 2018) or 300 million Quizlet study sets (Quizlet, n.d.) are created and shared freely by their users, teachers and learners, although they have both started to offer resources from educational publishers (Kolodny, 2017).

I intend to evaluate these services against the five Rs of OER (Weller, 2014). I also intend to explore the impact commercial interests and market forces can have on openness, including the following issues: ownership of user-generated content; ability of users to retain their data in cases of acquisitions, ownership changes or services shutting down; interoperability and reusability of content between different apps and services; and whether these apps contribute to wider adoption of open practices.

As an exploratory part of the project I intend to create an outline of a website which will present the results of my investigations into the openness of game-based SRS applications; advice and information on interoperability and reusability of content between different apps and services; and advice for practitioners on how to protect their data and ensure they can retain it.

I will present the results of a literature review of SRS in relation to openness, the results of an online survey of practitioners using game-based SRS and the outline of the advice and information website. 

References

Berger, R. (2017) “Are Education Startups The New Dot Com?,” /Forbes/, 19th January [Online]. Available at https://www.forbes.com/sites/rodberger/2017/01/19/are-education-start-ups-the-new-dot-com/ (Accessed 25 November 2018).

Deal, A. (2007) ‘Classroom Response Systems, A Teaching with Technology White Paper’, Office of Technology for Education, Carnegie Mellon University, [Online]. Available at http://www. cmu.edu/teaching/resources/PublicationsArchives/StudiesWhitepapers/ClassroomResponse_Nov07.pdf (Accessed 2 July 2017).

Harrell, E. (2018) “Kahoot! reached 70 million unique users on its platform,” /Kahoot!/ [Online]. Available at https://kahoot.com/blog/2018/01/18/70-million-unique-users-kahoot/ (Accessed 25 November 2018).

Kolodny, L. (2017) “Popular study app Quizlet faces a moment of truth as a new school year begins,” /CNBC/, CNBC, 23rd August [Online]. Available at https://www.cnbc.com/2017/08/23/quizlet-a-popular-study-app-faces-a-moment-of-truth.html (Accessed 25 November 2018).

Quizlet (n.d.) /About Quizlet | Quizlet/ [Online]. Available at https://quizlet.com/en-gb/mission (Accessed 25 November 2018).

Rose, J. (2017) ‘Best Games Like Kahoot 2017 - Top Alternative’s for Kahoot’, /AppInformers.com/, 14 April [Blog]. Available at http://appinformers.com/four-games-like-kahoot/9100/ (Accessed 10 September 2017). 

Statista (2018) /Apple: most popular app store categories 2018 | Statista/ [Online]. Available at https://www.statista.com/statistics/270291/popular-categories-in-the-app-store/ (Accessed 25 November 2018).

Weller, M. (2014) Battle for Open: How Openness Won and Why It Doesn’t Feel Like Victory [Online], London, Ubiquity Press. Available at DOI: http://dx.doi.org/10.5334/bam (Accessed 26 October 2017).

Extra content

Hi everyone, 

I’ve created a little quiz about open education using Kahoot. If you have time for some light entertainment in expectation of your presentations, head over to kahoot.it or download the Kahoot! app on your phone and enter this PIN number:

*0206846*

Alternatively, click on this link:

Play Kahoot! - Enter game PIN here

If you haven’t used Kahoot or similar apps before, it will give an idea of what my presentation is about. If you’re familiar with Kahoot, you can still test yourself on some open education trivia. The quiz will be up for a week. 

Ognjen 

 

Ognjen Vukas
20:16 on 14 February 2019

Embedded Content

Contribute

David Appel
7:32pm 20 January 2019


Hi Ognjen, this is an interesting project and I was wondering if you will also evaluate Mentimeter. We are currently running a pilot with Mentimeter at our school and will evaluate its suitabilty for interactive classroom teaching and not related to open practices. I am looking forward to your findings and website. 
David

Ognjen Vukas
11:27pm 31 January 2019


Hi David,

Thanks for your comment. I'm not sure whether to include Mentimeter. I've used it a few times but as a polling tool, to get students reactions and opinions before or after class discussions. It has a lot of potential for adding interactivity to presentations and getting the students/audience reactions quickly. On their website they list 16 different ways of using it, including quizzing and open-ended questioning, so it could be used in a similar way as the other SRS apps, but I think the free version is more limited.

Ognjen

Kelly Williams
5:50pm 9 February 2019


Hi Ognjen,

As a classroom user of Kahoot and Quizlet for SEN FE Arts students - I find these apps useful for occasional one-off uses - mainly to check knowledge transfer in an informal manner. I tend to go to Quizlet mainly as Kahoot can, at times, feel quite pressurised time wise for students who may have processing issues and have to enter answers on assistive technology devices or have low hand function. The quantity and quality of open Quizlet material is outstanding, it's pretty much ready-made and goes into much greater depth for our purpose. I've used it in tandem with my Trello project to assess knowledge on Elements of Design recently.

All the best with your presentation

Kelly

Phill Grimes
11:50am 16 February 2019


Hi Ognjen

Much the same as Kelly. I often use Quizlet for the learning, have a game of Quizlet live for some complete chaos, then a game of Kahoot for some assessment.

I do like the amount of available stuff on each and can be really useful, but then sometimes I can spend as much time editing one than I could have spent designing one from scratch!

Phill Grimes
11:51am 16 February 2019 (Edited 11:51am 16 February 2019)


I have completed your Kahoot.

It was going really well and then I got a bit trigger happy :)

Andrew Augustine
1:21am 18 February 2019


Hello Ognjen

Very interesting project. I am looking forward to your presentation.

Dr Simon Ball
11:13am 19 February 2019


Hi Ognjen

Well done on a great presentation! Here is a summary of the comments and questions you received following your presentation (including those you may have addressed verbally). Please respond in whatever way you choose - I suspect you may wish to deal with the first few in one response!

Best wishes

Simon

  • I have used Kahoots with adult learners and they love the competative nature of it. I have found that it helps to engage students
  • Quantity and investment but is there quality?
  • How is Kahoot used in HE?
  • summative or formative
  • There is certainly scope for learners to create their own quizzes, as Robin stated earlier about learners taking ownership, wonder if Ognjen is going to look into this .
  • Will be using to check understanding but quizzes can be too pressurised for neurodivrse students in my experience
  • Working in FE, I wonder how he can use such quizzes to relate to success critiera, we tend to have them as a plenary activity.
  • I imagine designing the teaching assessment is important to ensure its a positive experience for learners?
  • Quizzes are fun, but can they assess real understanding? Have you identified any similar apps which can assess true understanding?

Ognjen Vukas
10:56pm 23 February 2019 (Edited 4:39pm 24 February 2019)


Thank you all for your comments and questions.

Ognjen Vukas
10:56pm 23 February 2019 (Edited 4:39pm 24 February 2019)


I had some issues with adding all the responses in one comment, so I also replied to the questions separately.

Ognjen Vukas
10:57pm 23 February 2019


Thank you all for your comments and questions. Please see my replies below.

  • I have used Kahoots with adult learners and they love the competitive nature of it. I have found that it helps to engage students

Engagement is one of the most cited reasons for its popularity (and of other similar apps). Learners have even asked me to stay late after their last lesson on a Friday afternoon, so that they can play a Kahoot. Whilst competitiveness isn’t always healthy, it’s definitely one of the main reasons for the popularity.

  • Quantity and investment but is there quality?

There is a sort of crowd-sourced quality control, but it’s far from foolproof. When you search for quizzes, they can be sorted by Most relevant, Most played or Highest quality. Most relevant, according to Kahoot, is what they think ’is an ideal mix of factors including popularity and quality’ (Kahoot, 2019). It seems to take into account the subject and general topic area, and also how many times they’ve been favourited, duplicated and shared. The most played option is self-explanatory, but often misses the mark as it can return results with literal search terms present, but which are irrelevant to you. Highest quality, is again a mix of many factors, including ‘average play time and how many times each kahoot was favorited’ (Kahoot, 2019). 

Still, it’s always instructive to check the quizzes before reusing them. It’s quite easy to duplicate a quiz and revise, add and delete.

Recently, Kahoot and Quizlet, have added sets prepared by Kahoot, large science organisations, (e.g. NASA) and educational publishers (e.g. the National Geographic education dept). This Explore games page showcases some of this variety.

  • How is Kahoot used in HE?

Student Response Systems such as EVS (Electronic Voting Systems) or clickers have been used in HE lecture halls for a long time, since the 1960s. Positive effects have been found in a variety of settings (e.g. see a meta-review of studies in Chien et al., 2016). Some researchers have recommended encouraging active learning through the use of SRS, as it has been found to lead to greater learning gains. Crouch and Mazur (2001) recommend doing this by using peer instruction (PI) methods. This involves alternating cycles of questions and responses with ‘meaningful student interactions and activities’. I think many HE practitioners were attracted to SRS as they had this potential for encouraging interaction in large lecture halls. Kahoot has a team mode, which can be used to facilitate forms of peer instruction and interaction. Quizlet Live is also a team game.

Apps such as Kahoot have been more popular in secondary school settings, but there are examples of use, and even research of their use, in HE. For example, Plump and LaRosa’s (2017) examination of student responses to apps such as Kahoot in graduate and undergraduate settings was quite positive as they ’add positive energy, support concept exploration, and add fun to the classroom, which seems to translate into increased comprehension and motivation’.

Here are a couple of links with some ideas, including getting the students do create quizzes:

Kahoot!

Using Kahoot! in higher education

  • summative or formative

Formative, preferably low stakes. Also, for revision and facilitating debate and discussion. Retrieval practice has positive effect on learning, it is not simply a form of assessment. SRS can also be a good way of giving task-performance feedback to students, which also leads to better performance. Competitive nature of Kahoot increases engagement but on the other hand, as the top five results are publicly displayed, it may have a negative effect on students confidence and self-evaluations. Quizziz, for example, can be used without displaying any of the results to the class.

  • There is certainly scope for learners to create their own quizzes, as Robin stated earlier about learners taking ownership, wonder if Ognjen is going to look into this.

This is one of the ways Kahoot is used. They even have a template for this.

  • Will be using to check understanding but quizzes can be too pressurised for neurodiverse students in my experience

The competitiveness of Kahoot can create a pressurised environment, and it may not be appropriate or effective in all settings. Kahoots can be set as a homework challenge - this way learners can retake the quiz as many times as they want and can be done anonymously. Other apps, Quizziz, Quizlet, Socrative, Mentimeter, etc. can also be used as alternatives in less competitive ways. Another possibility is to use the team mode to encourage collaboration and peer instruction.

  • Working in FE, I wonder how he can use such quizzes to relate to success criteria, we tend to have them as a plenary activity.

I generally use these apps as formative assessment tools. As a form of retrieval practice, especially when spaced, these quizzes can have positive effects on long term learning. 

Asking the students to design their own quizzes to showcase their understanding could be a more summative form of assessment. 

  • I imagine designing the teaching assessment is important to ensure its a positive experience for learners?

Yes, I agree, making sure that it is a positive experience is important. When formative assessments are well designed and aligned with the curriculum, feedback is more effective. Kahoot can be used to give task related feedback, and encourage self-assessment, though the competitiveness also leads to some ego-focussed feedback. On the other hand, it encourages everyone to take part in the lesson and be actively engaged. The information about learner progress that the teacher gets is more complete than, e.g. questioning one or a few students in class.

  • Quizzes are fun, but can they assess real understanding? Have you identified any similar apps which can assess true understanding?

I think that there are ways of encouraging deep as well as surface learning with these apps, though perhaps they tend to be used to test and practice surface knowledge of facts, rather than deep understanding and critical thinking.

One way would be to encourage peer instruction. Some research shows that positive effects are larger when SRS are used in conjunction with active learning methods (Chien et al., 2016). For example, asking students to discuss their answers in pairs or groups and to explain their reasoning to their peers before answering, or discussing why the answers are correct and incorrect after the results are given.

Also, asking students to create their own quizzes, as others have pointed out above, can lead to deep learning and focus on meaning.

I’ve found this paper by Draper (2009) interesting as it discusses ways of encouraging deep learning when using MCQs.

References for the studies cited in the answers

 

Chien, Y.-T., Chang, Y.-H. and Chang, C.-Y. (2016) “Do we click in the right way? A meta-analytic review of clicker-integrated instruction,” Educational Research Review, vol. 17, pp. 1–18 [Online]. DOI: 10.1016/j.edurev.2015.10.003.

Crouch, C. H. and Mazur, E. (2001) “Peer Instruction: Ten years of experience and results,” American Journal of Physics, vol. 69, no. 9, pp. 970–977 [Online]. DOI: 10.1119/1.1374249 (Accessed 23 February 2019).

Draper, S. W. (2009) “Catalytic assessment: understanding how MCQs and EVS can foster deep learning,” British Journal of Educational Technology, vol. 40, no. 2, pp. 285–293 [Online]. DOI: 10.1111/j.1467-8535.2008.00920.x (Accessed 23 February 2019).

Kahoot (2019) The Discover page [Online]. Available at https://support.kahoot.com/hc/en-us/articles/115002817607-The-Discover-page (Accessed 23 February 2019).

Plump, C. M. and LaRosa, J. (2017) “Using Kahoot! in the Classroom to Create Engagement and Active Learning: A Game-Based Technology Solution for eLearning Novices,” Management Teaching Review, vol. 2, no. 2, pp. 151–158 [Online]. DOI: 10.1177/2379298116689783.

Ognjen Vukas
11:06pm 23 February 2019 (Edited 4:41pm 24 February 2019)


Open Education Kahoot results (top three):

Open Education
Final Scores
Rank Players Total Score (points)
1 Vh 10682
2 An 10006
3 Mountain Goat 9348

 

Open Education Kahoot

Ognjen Vukas
11:15am 24 February 2019 (Edited 11:22am 24 February 2019)


Thank you all for your comments and questions.

  • I have used Kahoots with adult learners and they love the competitive nature of it. I have found that it helps to engage students

Engagement is one of the most cited reasons for its popularity (and of other similar apps). Learners have even asked me to stay late after their last lesson on a Friday afternoon, so that they can play a Kahoot. Whilst competitiveness isn’t always healthy, it’s definitely one of the main reasons for the popularity.

Ognjen Vukas
11:18am 24 February 2019 (Edited 11:21am 24 February 2019)


  • Quantity and investment but is there quality?

There is a sort of crowd-sourced quality control, but it’s far from foolproof. When you search for quizzes, they can be sorted by Most relevant, Most played or Highest quality. Most relevant, according to Kahoot, is what they think ’is an ideal mix of factors including popularity and quality’ (Kahoot, 2019). It seems to take into account the subject and general topic area, and also how many times they’ve been favourited, duplicated and shared. The most played option is self-explanatory, but often misses the mark as it can return results with literal search terms present, but which are irrelevant to you. Highest quality is again a mix of many factors, including ‘average play time and how many times each kahoot was favorited’ (Kahoot, 2019). 

Still, it’s always instructive to check the quizzes before reusing them. It’s quite easy to duplicate a quiz and revise, add and delete.

Recently, Kahoot and Quizlet, have added sets prepared by Kahoot, large science organisations, (e.g. NASA) and educational publishers (e.g. the National Geographic education dept). This Explore games page showcases some of this variety.

References

Kahoot (2019) /The Discover page/ [Online]. Available at https://support.kahoot.com/hc/en-us/articles/115002817607-The-Discover-page (Accessed 23 February 2019).

Ognjen Vukas
11:21am 24 February 2019


  • How is Kahoot used in HE?

Student Response Systems such as EVS (Electronic Voting Systems) or clickers have been used in HE lecture halls for a long time, since the 1960s. Positive effects have been found in a variety of settings (e.g. see a meta-review of studies in Chien et al., 2016). Some researchers have recommended encouraging active learning through the use of SRS, as it has been found to lead to greater learning gains. Crouch and Mazur (2001) recommend doing this by using peer instruction (PI) methods. This involves alternating cycles of questions and responses with ‘meaningful student interactions and activities’. I think many HE practitioners were attracted to SRS as they had this potential for encouraging interaction in large lecture halls. Kahoot has a team mode, which can be used to facilitate forms of peer instruction and interaction. Quizlet Live is also a team game.

Apps such as Kahoot have been more popular in secondary school settings, but there are examples of use, and even research of their use, in HE. For example, Plump and LaRosa’s (2017) examination of student responses to apps such as Kahoot in graduate and undergraduate settings was quite positive as they ’add positive energy, support concept exploration, and add fun to the classroom, which seems to translate into increased comprehension and motivation’.

Here are a couple of links with some ideas, including getting the students do create quizzes:

Kahoot!

Using Kahoot! in higher education

References:

Chien, Y.-T., Chang, Y.-H. and Chang, C.-Y. (2016) “Do we click in the right way? A meta-analytic review of clicker-integrated instruction,” /Educational Research Review/, vol. 17, pp. 1–18 [Online]. DOI: 10.1016/j.edurev.2015.10.003.

Crouch, C. H. and Mazur, E. (2001) “Peer Instruction: Ten years of experience and results,” /American Journal of Physics/, vol. 69, no. 9, pp. 970–977 [Online]. DOI: 10.1119/1.1374249 (Accessed 23 February 2019).

Plump, C. M. and LaRosa, J. (2017) “Using Kahoot! in the Classroom to Create Engagement and Active Learning: A Game-Based Technology Solution for eLearning Novices,” /Management Teaching Review/, vol. 2, no. 2, pp. 151–158 [Online]. DOI: 10.1177/2379298116689783.

Ognjen Vukas
11:24am 24 February 2019


  •  summative or formative

Formative, preferably low stakes. Also, for revision and facilitating debate and discussion. Retrieval practice has positive effect on learning, it is not simply a form of assessment. SRS can also be a good way of giving task-performance feedback to students, which also leads to better performance. Competitive nature of Kahoot increases engagement but on the other hand, as the top five results are publicly displayed, it may have a negative effect on students confidence and self-evaluations. Quizziz, for example, can be used without displaying any of the results to the class.

Ognjen Vukas
11:30am 24 February 2019


  • There is certainly scope for learners to create their own quizzes, as Robin stated earlier about learners taking ownership, wonder if Ognjen is going to look into this.

This is one of the ways Kahoot is used. They even have a template for this. Recently, the Kahoot app also got this function so learners can create quizzes in the app.

Ognjen Vukas
11:32am 24 February 2019


  • Will be using to check understanding but quizzes can be too pressurised for neurodiverse students in my experience

The competitiveness of Kahoot can create a pressurised environment, and it may not be appropriate or effective in all settings. Kahoots can be set as a homework challenge - this way learners can retake the quiz as many times as they want and can be done anonymously. Other apps, Quizziz, Quizlet, Socrative, Mentimeter, etc. can also be used as alternatives in less competitive ways.

Another possibility is to use the team mode to encourage collaboration and peer instruction.

Ognjen Vukas
11:33am 24 February 2019


  • Working in FE, I wonder how he can use such quizzes to relate to success criteria, we tend to have them as a plenary activity.

I generally use these apps as formative assessment tools. As a form of retrieval practice, especially when spaced, these quizzes can have positive effects on long term learning. 

Asking the students to design their own quizzes to showcase their understanding could be a more summative form of assessment.

Ognjen Vukas
11:34am 24 February 2019


  • I imagine designing the teaching assessment is important to ensure its a positive experience for learners?

Yes, I agree, making sure that it is a positive experience is important. When formative assessments are well designed and aligned with the curriculum, feedback is more effective. Kahoot can be used to give task related feedback, and encourage self-assessment, though the competitiveness also leads to some ego-focussed feedback. On the other hand, it encourages everyone to take part in the lesson and be actively engaged. The information about learner progress that the teacher gets is more complete than, e.g. questioning one or a few students in class.

Ognjen Vukas
11:36am 24 February 2019


  • Quizzes are fun, but can they assess real understanding? Have you identified any similar apps which can assess true understanding?

I think that there are ways of encouraging deep as well as surface learning with these apps, though perhaps they tend to be used to test and practice surface knowledge of facts, rather than deep understanding and critical thinking.

One way would be to encourage peer instruction. Some research shows that positive effects are larger when SRS are used in conjunction with active learning methods (Chien et al., 2016). For example, asking students to discuss their answers in pairs or groups and to explain their reasoning to their peers before answering, or discussing why the answers are correct and incorrect after the results are given.

Also, asking students to create their own quizzes, as others have pointed out above, can lead to deep learning and focus on meaning.

I’ve found this paper by Draper (2009) interesting as it discusses ways of encouraging deep learning when using MCQs.

References

Chien, Y.-T., Chang, Y.-H. and Chang, C.-Y. (2016) “Do we click in the right way? A meta-analytic review of clicker-integrated instruction,” /Educational Research Review/, vol. 17, pp. 1–18 [Online]. DOI: 10.1016/j.edurev.2015.10.003.

Crouch, C. H. and Mazur, E. (2001) “Peer Instruction: Ten years of experience and results,” /American Journal of Physics/, vol. 69, no. 9, pp. 970–977 [Online]. DOI: 10.1119/1.1374249 (Accessed 23 February 2019).

Draper, S. W. (2009) “Catalytic assessment: understanding how MCQs and EVS can foster deep learning,” /British Journal of Educational Technology/, vol. 40, no. 2, pp. 285–293 [Online]. DOI: 10.1111/j.1467-8535.2008.00920.x (Accessed 23 February 2019).

Contribute to the discussion

Please log in to post a comment. Register here if you haven't signed up yet.