Publication/Presentation Date

7-25-2014

Abstract

Abstract

This project was a dream for both the Division of Education and the EPIC team: to create a game that would assist all Lehigh Valley Health Network employees, and possibly medical professionals nationwide, in the switch to the EPIC systems. In order to do so, a game design process was required that would optimize both the instructional and traditional benefits of games. After design and development was complete there needed to be an evaluative testing process. This process would compare the extent of knowledge of those that played the game and those that just read the information from a standardized sheet of paper. The results would then be analyzed to see if there is any noticeable difference in the effectiveness of these learning methods.

Background

The reason for this project was due to a challenge in the process of switching the hospital systems from their current applications to the new EPIC systems. Many of the staff knew the old systems by name and function, but the EPIC systems were entirely different. So the Division of Education put out the request for research scholars specializing in game design, as a way to help smooth out the transition. The team set about creating an engaging and educational tool in the fashion of a video game. Now at first it may seem that a video game is not the best educational tool to use, but the team’s research may help to counter that belief. The team started by reading different books about basic game design principles and how to apply those principles in an educational setting. This kind of educational method has been looked at by a number of people from varying professions. Jane McGonigal, a game designer and author, has a number of TED talks and a few books on how games can make the human race better not just with education. Karl Kapp, a Professor and author currently at Bloomsburg University has several books on the specific topic of educational gaming which has come to be known as Gamification. These are just some of the authors the team examined as a basis for the research aspect of this project.

Literature Review

As mentioned above, the team took time to examine all of the prior research on Gamification and eLearning development processes as well as standard, traditional game design principles. Each team member began by reading through a book that had been suggested by the team manager. Jonathan read Gamification of Learning (Kapp, K. M.) and Matthew read Rules of Play (Salen, K., & Zimmerman, E.). After finishing their books respectively, Jonathan did research on several different TED Talks, which all were speeches comprised discussing specific topics, and Matthew went on to examine the 100 Principles of Game Design (Despain, W., & Acosta, K.).

Gamification was written from the perspective of a teacher or educator and discussed how video games can be beneficial to learning different types of information and can help process information more quickly. The concepts retrieved from Gamification were those such as how to structure reward systems, details on motivation (both intrinsic and extrinsic), and four general types of players and how they act in accordance with the game.

Rules of Play, however, discussed principles of game design and traditional ‘play’ from the perspective of a designer. The main concepts learned from Rules of Play were the concept of meaningful play, the idea of a system, the ‘Magic Circle’ and ‘lusory attitude’, the flow, emergence, and how the overarching attributes of rules, play, and culture and how they influence games.

The TED Talks influenced our design and development from several of the concepts that spanned numerous videos. The majority of the talks discussed how video games affect the brain and how ‘gamers’ have become exceedingly efficient at certain activities in the real world. Some of the topics were simple reminders that would help with making the games more enjoyable and rewarding such as measuring progress, rewards for effort, elements of uncertainty, rapid, clear and frequent feedback, and blissful productivity.

The 100 Principles of Game Design excerpt was very helpful because, even though it did not include all one hundred, it did give a lot of definitions for concepts that were never quite explained. For example, how to go about giving a player a sense of accomplishment or how to get a player into ‘the flow’. The list of principles that have been recorded and used in classic games is countless but the team kept track and made sure to use those that they thought were the most important.

After researching both online and in textbooks for over a week, the team began to design their own game development process using what they had learned in such a way that they could come up with a method for testing the learned material at hand.

Methodology

The team began by recording all of their previous thoughts about traditional and instructional game design in order to record all knowledge prior to research and maintain a clear, conscious, and open mind to the new information. Several books on relevant topics such as the educational learning and traditional game design methods were obtained and studied rigorously. After a week of intense study and diligent penmanship, the team sat down to compare notes of all that they had learned and, ultimately, to move forward and create a template-like design process that could be used for developing a game that would be both immersive and educational.

They began by listing all design elements that comprise any game, sorted them into an order that would be logical for development, and then finally broke each piece down individually to get a more precise concept of what they would be dealing with at each phase of design. After meticulously analyzing every aspect of game design, they began to test the process they had developed. They did so by walking through classic games of all different types such as Solitaire, Kingdom Hearts II, and even the general concepts that had began to blossom in their minds that could live up to this EPIC task.

The team storyboarded three different ideas that all dealt with a similar solution to how players would be exposed to the data and how they would manipulate it later on to help integrate their knowledge. The team members all agreed that in terms of actual development, they should begin with the core learning interface and gameplay and then later build up the more ‘non-essential’, non-educational aspects slowly to create a complete and holistic game that would serve the purpose it was designed for.

A playable prototype, known to the team as Version 1.0, was developed that allowed the user to drag and drop three different systems into corresponding drop boxes when shown three definitions. After submitting that answer, the player would be shown an evaluation screen that gave feedback on what the correct systems were, whether the player got the match right or wrong, a total amount correct and incorrect thus far, and both a total time played and an average time per ‘run’. This was to be the core learning mechanic of a much larger game that would have the players traversing puzzles and collecting data objects. At the end of the level the player would connect the systems correctly based on what they had learned prior by collecting the pieces of information.

It was at this phase when the team decided to refocus to a smaller game with a more accurate scope for the restrictive timespan of the eight week Research Scholar program. The team took a day to sit and research classic, educational video games and what made them unique, instructional, immersive, and most importantly entertaining. The team began brainstorming several ideas and it seemed unanimous that the best concept was a simple carnival race game that required the players to match one of four EPIC systems to one definition, known as Version 2.0. This allowed the player to focus on one system at a time as well as testing themselves against three other computer players that were answering their own questions at the same time.

After Version 2.5 was finished to such an extent that it would allow for immersive testing, a simple written matching test would be created that would be used to test the effectiveness of playing the game created versus reviewing the sheet of paper that all employees were sent to help with the transition to EPIC. The participants would be tested both before and after they experienced the intermediate learning method and the scores on both tests would be analyzed. An extra survey would be given out to get additional feedback on specific aspects of the game that would allow for future refinement.

Below are the main processes developed by the team that were used throughout development.

Meaning of Play Structure:

Description: At the bottom is the core, basic requirement for meaningful gameplay and as your game builds and the player continues the higher levels are required to keep that player interested and immersed in the game.

Development Order:

Description: This is an overall development timeline that should be used to transition smoothly from one step in design to the next.

Principles to Remember:

Description: This diagram shows the flow of concepts to double check when playtesting as a designer. You have a goal, you are conscious of the awareness of the meaning of play, and you are setting up the principles to ease the player into more dynamic gameplay and overall enjoyment.

Results

The team conducted a standard experimental and control group test that would require both groups to take a pretest to record their prior knowledge, do different activities to attempt to better learn the EPIC systems, and then take a posttest to evaluate their new, learned knowledge. The experimental group would play the “EPIC Space Race” game that the team developed between the two tests, and the control group would just review the standardized sheet of paper that was given to all employees with the system names and functions.

Eight people participated in the control group testing. The average score on the pretest was 39%, with the highest being 57.1% and lowest being 8.6%. The control group then took however long they desired to review the sheet of paper. They then took the posttest which yielded an average score of 78.2%, with the highest being 100% and the lowest score being 40%.

Ten people participated in the experimental group testing. The average score on the pretest was a 34%, with the highest being 60% and lowest being 5.7%. The testers played approximately four rounds of the “EPIC Space Race” at an average of 2 minutes and 4 seconds each. The most times played was six and the least amount of times played was twice, leading to an average total amount of time played per person being 8 minutes. They then took the posttest which yielded an average score of 52.3%, with the highest being 82.9% and the lowest score being 14.3%.

We also took a survey for the experimental group and asked them to give us specific feedback on what they thought about the game. 100% (10/10) of the players thought that the game was fun to play. 90% (9/10) of the people thought that it was intuitive on how to play, but 30% (3/10) of people said they wanted some sort of instructions on the main screen. Lastly, 40% (4/10) of the people went out of their way to say that they did feel like they had learned from the game.

Conclusion

The team took all of the constructive criticism that was given from the surveys and made numerous improvements to the game. They changed the color of the text to be a neutral color such that on evaluation it would become apparent which was correct and which was incorrect. They also added a dynamic ranking system in the top left corner that would display what rank all of the players are at every moment during the race, to add to the intensity of the game play. Hints were also added underneath the system description to add an extra feedback system. Finally, the team changed the correct to incorrect ratio that the computers answer at to make it more realistic and based on testing data; this way it seems that the player is actually competing against average players.

In terms of the results from the testing, there was an increase in knowledge of approximately 18.3% in the experimental group and 38.9% in the control group. The team believes that this large difference in favor of the control group and the standardized paper were because of numerous unforeseeable reasons. The first being the encouraged norm of memorization for short term success that is fostered in the modern school system and such a hindrance between the correlation of school testing scores and actual real world knowledge of subjects. Secondly, the team believes that, although a significantly short amount of time was spent playing the Epic Space Race, the retention of the knowledge would be significantly greater than that of the control group because of the previous factor as well as enjoyment. The final piece of conjecture that the team discussed dealt with the standard flaw in matching tests: process of elimination. If the control group came in with a higher percentage of information on the pretest and only learned a few more terms during the time in which they were studying, then if they guess on the last few questions they will end up getting more of those right because there are less terms to choose from, which skews their final score. Our team believes that because of that flaw, it was only natural to see more improvement in the control group because they did have higher scores on the pretest.

After earnest analysis and review of the feedback proclaiming positive results for the “EPIC Space Race”, the immense amount of fun the testers had from playing it, the fact that it gets employees to manipulate the information and make mental connections between the departmental functions and the systems that they will use, and the expected retention rate means that this program is a viable tool and a optimum replacement of the standardized sheets of paper for teaching the EPIC applications that will be coming to the Lehigh Valley Health Network.

References

Despain, W., & Acosta, K. (2013). 100 principles of game design. Berkeley, CA: New Riders.

High-Level Description of the Sixteen Personality Types. (n.d.). Retrieved July 21, 2014, from http://www.personalitypage.com/high-level.html

Platinum Butterfly. (2013) Artificial Gravity. Retrieved July 21, 2014, from http://dig.ccmixter.org/files/F_Fact/41039

Cinematrik. (2005) Revolve. Retrieved July 21, 2014, from http://ccmixter.org/files/hisboyelroy/430

Kapp, K. M., Blair, L., & Mesch, R. (n.d.). The gamification of learning and instruction fieldbook: Ideas into practice.

Kapp, K. M. (2012). The gamification of learning and instruction: Game-based methods and strategies for training and education. San Francisco, CA: Pfeiffer.

NGUI 3.6.0 Tutorial 1: The Environment. (n.d.). Retrieved July 21, 2014, from https://www.youtube.com/watch?v=dM44h752LyA

Salen, K., & Zimmerman, E. (2004). Rules of play: Game design fundamentals. Cambridge, MA: MIT Press.

TED: Ideas worth spreading. (n.d.). Retrieved July 19, 2014, from http://www.ted.com/

As3sfxr. (n.d.). Retrieved July 21, 2014, from http://www.superflashbro

Comments

Mentor: Jason Paden

Department(s)

Research Scholars, Research Scholars - Posters

Document Type

Poster

Additional Files
Kastner and Rossi - Manuscript.pdf (216 kB)
Download

Share

COinS