A qualitative descriptive study of SimWars as a meaningful instructional tool

Objectives: To investigate faculty and residents’ perceptions of whether SimWars can serve as a meaningful instructional format for Emergency Medicine residents and to identify strategies to implement SimWars effectively in a residency training program. Methods: In this descriptive interview study, 5 facilitators, 2 contestants, and 8 observers were recruited from an Emergency Medicine residency program at a large, urban, university-based, level-I trauma center. Interview questions were created with the guidance of the social constructivism theory. Participants were interviewed individually. Themes were identified associated with responses to these questions. Results: (1) SimWars can be a meaningful instructional tool; (2) Debriefings and a well-structured rubric are essential; (3) Competition motivates participants to do their best, but can have a negative impact on them; (4) Residents expect to use the clinical skills, patient management and teamwork skills practiced in SimWars; and (5) Residents need more training on teamwork, including Team Strategies and Tools to Enhance Performance and Patient SafetyTM (TeamSTEPPS). Conclusions: With its focus on competition, SimWars can be effective in residents’ training when debriefings are guided by well-structured rubrics.


Introduction
In January 2008, three emergency medicine physicians in the United States, Yasuharu Okuda, Scott Weingart, and Steven A. Godwin, had a vision for assessing teamwork, communication, and patient care using a fun, interactive, and meaningful medium (personal communication, 2011).As a result, SimWars, an onstage competition came into being.For participants and audience alike, SimWars shares similarities with popular television shows produced worldwide where musical contestants perform on stage and are judged by a panel of experts.Following the critique and debriefing, either the panel or an audience vote determines which contestants move on to the next round of competition.One such example in the United States is American Idol.In SimWars, clinical teams demonstrate their communication, teamwork, clinical management, and problem solving skills.Upon completion of each simulated clinical encounter, judges assess and discuss participants' clinical actions and team dynamics.Most often, the debriefing occurs on stage and in public and the audiences vote on a winner via an audience response system.Since its inception, SimWars have been staged at national meetings such as the annual International Meeting on Simulation in Healthcare (IMSH), the Society for Academic Emergency Medicine (SAEM), the American College of Emergency Physicians (ACEP) as well as locally in the form of citywide competitions.
The objectives of the study are to investigate whether a SimWars event held locally at a large, urban Emergency Medicine residency program is an effective educational program and how to best use features of SimWars, beyond entertainment, for educational purposes.No published research demonstrates the impact of SimWars on residents' learning.Therefore there are more questions than answers regarding SimWars for educational purposes.It is not clear how curriculum leaders should use SimWars to train residents or what curriculum design strategies should be used to enhance the residents' experience.How does SimWars' competitive nature affect residents' performance and learning outcomes?In simulation education, creating and preserving a psychologically safe learning environment is critical and can be challenging. 1This can be amplified in SimWars local events as inexperienced facilitators may create harsh conditions for participants, which could have detrimental impacts on current and future learning.Debriefing time constraints can limit opportunities to provide sufficient feedback for participants, especially at an individual level.For example, in a SimWars debriefing there is no opportunity to demonstrate the correct way of performing a procedure; as a result, incorrect skills and procedures can go uncorrected or superficially discussed.This qualitative research study is intended to elicit and analyze feedback from both facilitators and participants to help to assess the value of SimWars as well as areas for improvement.Specifically, the research questions include: 1. What are residents', facilitators', and observers' perception of the key elements of a SimWars session, including: teamwork, competition, debriefing, reflection on their own action, and social construction of knowledge?2. What are strategies used to conduct a SimWars session?

Theoretical framework
The theory of social constructivism, research on competition, and psychological safety guided the design of our interview questions (see Table 1 for key elements of interview questions).The theory of social constructivism emphasizes that individuals create their own knowledge based on how they relate new information to what they have previously experienced, 2 and additionally recognizes the value of social interactions in the learning process, meaning that people clarify personal conceptions as they interact with the understandings of others. 2 When we encounter something new, we reconcile it with our previous ideas and experience, changing what we believe, or discarding the new information as irrelevant.In a SimWars session, residents collaborate and interact with one another while providing care for simulated patients.The goal is for participants to demonstrate and gain clinical knowledge, teamwork and communication, and patient management skills.The constructive process involves not only residents' prior knowledge and experience in the clinical scenarios, 3,4 but also the psychosocial factors such as motivation, anxiety, and stress, which have been shown to impacts learners' performance. 5,6The learning outcome is optimal when learners are free from threat, least defensive, and want to learn. 1,3The SimWars experience theoretically combines collaboration with peers and competition which requires both psychological safety and simultaneously induces stress.With regards to collaboration in healthcare settings, the Agency for Healthcare Research and Quality (AHRQ) and the Joint Commission (TJC) have identified four competencies for teamwork for clinical education: leadership, communication, situational monitoring, and mutual support. 7ese competencies are based on research on the crew resource management and human factors principles that improve patient safety and clinical performance. 8Effective clinical teamwork and collaboration also requires psychological safety and a culture of learning. 1These key features were the foundation for our interview questions.

Methods
A descriptive interview design was selected due to the exploratory nature of the research questions.A descriptive research design is a widely used qualitative research method used to gather information about a particular situation. 9nterview is one of the three main types of data collection methods in descriptive research design. 10The focus of this study was on the experience of individual participants, observers and facilitators in an Emergency Medicine Residency SimWars competition.Predetermined interview questions were used, and participants were encouraged to expand upon their answers in an investigative manner, which often lead to more detailed, in-depth information.

Participants and data collection
This study was approved by the Institutional Review Board (IRB) at the New York University (NYU) School of Medicine, a large, urban, university with a level-I trauma center.
Participants were residents and faculty members at the Department of Emergency Medicine in the NYU School of Medicine.They were briefed on the purpose of the study and the potential risks and benefits.They were informed that their decision to participate in the study would have no impact on their academic or professional standing at the institution and that their responses would be de-identified.
Informed consent was obtained and interviews were conducted in private with each lasting 30 to 45 minutes.Fifteen interviews, including 5 facilitators, 2 participants, and 8 observers were completed following a SimWars session in the Emergency Medicine residency training program at NYU School of Medicine.The facilitators included faculty members and senior residents.The participants included those competing in and observing the SimWars competition.

Interview questions
The interview questions are listed in Appendix 1.The interview questions were tailored to the person's role in the SimWars competition.For example, questions for facilitators focused on case scenario design, debriefing, and serving as judges.Questions for faculty panelists focused on how the team cared for the patients and their clinical accuracy.Questions for SimWars participants focused on teamwork, attitudes during SimWars, levels of motivation, and responses to the facilitators' feedback.Questions for the observers had the same focus as that for the participants, except that they were told to answer from an observer's perspective.

Data analysis
Qualitative data analysis methods were used to identify themes associated with the responses to these questions.The data were coded using Atlas.ti 11software, a secure, PCbased software tool, which enabled researchers to discern the themes in the audio data (from interviews) to explore the relationships among the themes.Two researchers independently reviewed the interview data and made note of specific comments to discern themes and develop a coding scheme (CD and TC), followed by the complete research team analyzing the findings.Dominant themes were identified by iteratively comparing the data to ascertain recurrent themes using a bottom up coding strategy and quantifying their significance. 12Comments were categorized into a branching tree of thematic labels.Analysis was done until we obtained a state when additional layering of themes and stakeholder analysis yielded no new information from additional data. 13Although the sample size is adequate to answer the research questions (5 facilitators, 2 participants, and 8 observers), we are unable to categorize the results based on specific roles.

Results
The interview questions focused on the key elements such as teamwork, debriefing, psychological factors, issues related to SimWars, and future improvements.The results regarding each key element were presented as following.

Teamwork
Teamwork is the most frequently discussed theme among all the interviewees.Teamwork was addressed in the following four aspects: leadership, mutual support, communica-tion, and situational awareness.Table 2 summarized key codes and associated quotations.
 Leadership.All participants agreed that there was a clear leader during the SimWars session, and the leader received support from other members. Mutual Support.Mutual support means that the team members support one another during the SimWars.Among the 11 who addressed this issue, 9 said mutual support was obvious among team members, which entailed clear roles and responsibilities of team members and support of one another. Communication.Among the 14 who addressed communication issues, 8 said communication was great among the teammates; while 6 said they could do better.However, when asked to provide specific examples, including closed-loop communication and check-backs, few were able to do so. Situational Awareness.Situational awareness involves each team member scanning the situation to understand the environment and the patient status.Often, it is a prelude to mutual support to the leader or another team member.Situational awareness empowers each team member to speak up and call out solicited and unsolicited information that can influence patient care outcome.This competency raised the most controversies.Five had different interpretations of situational awareness, such as the participants' understanding of the environment.Among the seven who interpreted situational awareness as the understanding of the team dynamics, 5 said the team did a great job and 2 thought they did an average job.

Key Codes Quotations
Leadership "I think that, uh, there was a clear leader, um, that identified themselves and took charge, and, uh, was…um… really delegating responsibilities.So, I don't know, pretty good leadership in those regards." Mutual support "I think the leader was supporting the group but there could be more communication from the other team members." Communication "I think mostly the team members need to be more assertive in getting their point across but also the leader needs to be more receptive to that.""I didn't hear enough bottom-up communication.

Situational awareness
"Yeah I think they-they hadn't really grasped the capacity for them to-to really know the things were being done because it's so artificial.

Debriefing
The participants commented on debriefing strategies and the key elements in the debriefing sessions. The debriefing sessions should include positive feedback and also offer suggestions for future improvement. 14 The debriefing session should be structured to include comments on clinical skills as well as patient management and organizational skills. A rubric based on the learning objectives should be used to guide the debriefing. Debriefing should include input from audience members, rigid grading scales, and take-home sheets to highlight the key points.
Table 3. Key codes and quotations related to psychological factors

Psychological factors
Table 3 presented the key codes on psychological factors and associated quotations.Eight participants indicated that psychological safety was preserved during SimWars, while two said it was not.Psychological safety was best preserved when the debriefers focused on group, rather than individual performance.Regarding the impact of stress on participants' performance, 5 out of 6 who answered the question said that the participants were more anxious in the competitive environment but could do better in a real clinical environment.One participant indicated that the residents were well-trained, and the pressure did not affect their performance.Participants also commented on simulation fidelity or the physical realism associated with the simulator or the simulation environment.All the interviewees said that the simulation is an artificial environment that resembles but does not replicate a real clinical scenario.Three interviewees noted that although the scenarios were not realistic, simulation and SimWars seemed to be an effective teaching tool.They offered suggestions for making the scenario and participants' actions more realistic, for example using make-up and other moulages to make the simulated patient look ill and more human, or situating participants in "ifthen" scenarios where participants' actions are clinically consequential through making the simulation conceptually and emotionally relevant. 15egarding the question of whether residents can transfer what they learn from SimWars into patient care, all participants perceived that these skills would be transferred to the clinical setting.Depending on the case scenario, residents are faced with a situation that rehearses rare clinical events, requires complicated clinical skills, reviews a familiar case, and/or exercises patient management and leadership skills.

Future improvement
Future improvement focuses on realism of simulation, participants' involvement, debriefing, and the structure of the SimWars.Table 5 includes details for these aspects.

SimWar as an instructional format
This study indicates that SimWars can serve as a meaningful instructional format to train residents' clinical knowledge, communication, and teamwork skills when used as part of a comprehensive education program.In this study, SimWars was used as an alternative to the weekly simulation sessions in the Emergency Medicine residency program.However, there was still a need to review the clinical topics as well as communication and teamwork skills as indicated in the learning objectives of the SimWars session.
Previous research indicates that educational goals should match the format of simulation technology for teaching. 16In this study, the simulated patient has lifelike full-body features that display symptoms such as heart rate and blood pressure, and responds to physical interventions such as chest compressions.The decision to use a hi-fidelity simulator was based on the objectives of the SimWars scenario to allow the residents to accomplish specific clinical interventions.However all interviewees commented that the simulated patient is not a real patient, which made it difficult to treat him or her as a real patient.For example, intubation, a frequently mistaken procedure that requires extensive practice, was not performed by any of the teams, although it was a key step in the particular trauma scenarios.To improve realism of SimWars, one suggestion is to use make-up and other moulages to make simulated patients look ill and believable.When the realism of the simulator or the environment is augmented, we refer to this as increasing the fidelity. 17In the absence of realistic features, leaders of SimWars could appeal to an informal contract as suspension of disbelief, including specifying the limitations of the simulator and the expectation of the participants to treat the simulated clinical situation as a real event.This could help manage learners' expectations and may encourage higher quality performances.

Teamwork training
Teamwork training has been recognized as an important educational goal by federal quality and regulatory agencies as well as academic societies, medical boards, and residency review committees.In our study, participants were able to apply some of the four TeamSTEPPS competencies, including leadership, mutual support, communication, and situational awareness.However, additional training is indicated.In addition, some residents participated in roles outside of their profession, including portraying the role of nurses.It will be interesting to see how residents perform as part of an interprofessional team in SimWars competitions.

Debriefing
Our results demonstrated the critical role of debriefing in SimWars, in accordance with previous research studies. 18he results also point out the key elements of effective debriefing, which is consistent with the best practice discussed in prior studies 19,20 Debriefing should include formative feedback with the goal of improving participants' performance.A well-constructed debriefing rubric could assist with this task, in particular at the level of identifying the performance gap. 21

Psychological factors
][24] In simulated patient care, these factors have similar impact Stress has similar impact on performance as competition.The impact can be best described by the Yerkes-Dodson curve that performance increases with physiological or mental arousal, but only up to a point; when levels of arousal become too high, performance decreases. 25In SimWars, stress could be induced from peers' observation and/ or evaluation by attending physicians.All interviewees agreed that the participants in SimWars experienced observational pressure and individuals may use different strategies to address these feelings.It is possible that competition and stress might contribute to knowledge transfer since practicing under pressure seems to improve performance under similar circumstances. 26

Knowledge transfer
No matter what format of simulation technology is used, the ultimate goal is to improve patient care outcomes, which is the highest level of the Kirkpatrick hierarchy, a widely used classification for training outcomes. 27Like any other formats of medical simulation, SimWars may not occur in an actual patient care environment and some learners may not make the connections or transfer what has been learned in the simulation environment to clinical practice.Whether the knowledge gain through SimWars can be transferred to patient care has to be addressed through longitudinal follow-up research.Studies of this type are very difficult to design and to execute. 28However, without reservation all interviewees perceived that residents will use these skills in their future practice, including clinical skills, patient management, and teamwork skills.Further studies are required to investigate residents' immediate knowledge gain and skill acquisition as well as skill transfer to clinical settings.

Implications, limitations and future work
This study showed that faculty and resident perceived SimWars as an effective instructional tool when combined with lectures.To maximize SimWars' instructional value, faculty should consider teaching teamwork and communication using TeamSTEPPS as a guide.The use of rubrics to guide the objectives and the debriefing process also adds value to the SimWars competition.SimWars in our study consisted of clinical content in a single topic area -gastro-intestinal bleeding.A few of the interview questions are scenario-dependent.It is possible that a different clinical topic area might yield different answers for the same questions.This problem can be solved by interviewing a different group of residents who will go through the same clinical scenario, and by interviewing the same group of residents after they go through a different case scenario.It will be interesting to compare the data to see whether different scenarios yield different outcomes.This was a local competition.The effects of a national setting or performing outside the home institution may increase stress or otherwise affect the experience for both organizers and learners.The larger sample size and the more extensive data will enrich our understanding of SimWars as an instructional format in medical simulation.

Table 1 .
Key elements of the interview questions

Table 4
summarizes the key codes and associated quotations related to SimWars.All the interviewees agreed that Sim-Wars was helpful for the following reasons:

Table 4 .
Key codes and quotations related to SimWars "They'll remember certain things, reinforce learning, and the competition will maybe make the audience more focused.For the participant, I'm not clear, but I guess, I would expect that perhaps the participant would try to do a little better than the other team.But um-it's unclear to me whether it would be better without competition.For the audience, it's probably a little better because it's more repetitive and people try, people try to maintain attention and make it more entertaining."Howis SimWar useful?"Because the best learning is from patient cases, at least for me and most people in emergency medicine.I think were kind of like learners by doers, and this way you get to do something, sort of learn on a patient but it's not actually going to impact the patient, real patients care.""Andthat's good for the boards as well.I think because those, it's even more of an artificial situation than SimWars.And I like having fun…um…and it's entertaining, and it gets you involved in trying to see who is the best."leadership roles are the things that you can practice more here and you don't feel, people don't usually get a chance to experiment on how they would do those things."

Table 5 .
Future improvement to use SimWars in residents training