Developing and evaluating an e-learning package for medical students on genocide and public health

Objective: This paper describes the development and evaluation of an e-learning package on the topic of genocide and public health aimed at undergraduate medical students. Methods: An e-learning package was designed based on pedagogical principles and was evaluated in a sample of second-year medical students. Students were asked to rate the usability of the e-learning package (design, interactivity, pace, suitability as a learning mode for the topic). The effectiveness of the e-learning package in relation to achieving learning outcomes was tested via baseline and postlearning activity questionnaires. Results: Out of 253 second-year medical students, 96(38%) participated in the evaluation of which 45% gave a score of 9 out of 10 for the e-learning package design, 97% preferred it as a learning mode over traditional lecture based teaching and 95% agreed that the package increased understanding of the topic. Immediately following the learning activity 93.75% of second-years agreed that genocide was a public health issue as opposed to 23.96% at baseline. 96% agreed that health professionals had a role in genocide prevention but only 32% agreed that they had a personal role in genocide prevention. Conclusions: The evaluation of the e-learning package suggested that learners found it well-designed and a preferred learning mode as compared to traditional lecture based teaching. There was some evidence of positive changes in knowledge and attitudes relating to genocide prevention and the role of public health immediately following the learning activity.


Introduction
Genocide is defined, according to the United Nations Convention on the Prevention and Punishment of the Crime of Genocide, as "any of the following acts committed with intent to destroy, in whole or in part, a national, ethnic, racial or religious group, as such: 1. Killing members of the group 2. Causing serious bodily or mental harm to members of the group 3. Deliberately inflicting on the group conditions of life calculated to bring about its physical destruction in whole or in part 4. Imposing measures intended to prevent births within the group 5. Forcibly transferring children of the group to another group". 1 Since the twentieth century, genocide has claimed several million lives.3][4] The health outcomes of genocide extend beyond the devastating impact of the crude mortality rate.Genocide often causes shortages of food, water and shelter along with poor sanitation and living conditions.This results in long-term health sequelae which include infectious diseases and malnutrition. 2,5,6It is important to realise that health professionals can have a role in the prevention and mitigation of genocide. 7,8enocide prevention is currently not listed as a core learning outcome in the UK undergraduate medical curriculum 9 and is covered rarely within the undergraduate medical course. 10Therefore, whilst there is a clear role for health professionals in genocide prevention, they may not be aware of this and there may be a case for educating health professionals.Some of the educators within Public Health therefore decided to develop a pilot educational initiative to provide an overview of genocide as a public health issue to test the receptivity of medical students to this concept.Given that this was a pilot initiative combined with existing timetabling constraints, it was agreed that an e-learning package would be the best approach.To make it more relevant and engaging for medical students, the main role in developing the package was assigned to a third year medical student Makarious Awad (MA).Another advantage with an e-learning package designed as a standalone learning module that did not require prior knowledge, would be that it could be made available to other interested learners with a basic understanding of public health even outside of the medical course.This paper reports on a project to design, develop and evaluate an e-learning package on genocide and public health for undergraduate medical students at the University of Nottingham.The evaluation focused on the effectiveness and usability of the e-learning package as an educational intervention.

Methods
We used the framework advocated by Leeds University for the development and evaluation of the e-learning package. 11e also drew on Mayes' conceptualisation cycle, Salmon's five stage model of e-learning and Laurillard's conversational model when designing the e-learning package. 12,13A project steering committee comprising MA, Heather Roberts (HR), Puja Myles (PM), and Nicki Keating (NK) agreed on the development and evaluation strategies to be used based on a consensus approach.

Design and development of the e-learning package
Learners and the context: the e-learning package was mainly aimed at undergraduate medical students but assumed no prior knowledge on the topic.This therefore, made it suitable for anyone with a basic understanding of public health principles and health education.The package incorporated a combination of approaches to accommodate for differences in learning styles.A selection of text, visual imaging, videos, interactive tasks and quizzes throughout the e-learning package were utilized to engage the learner.

Technological aspects of design
][16][17][18][19][20] E-learning allows the user to access the content virtually anywhere and at any time, as long as an internet connection is available.The package was also designed to allow the slowest of internet connections to support use, and is therefore compatible for use with any device that allows internet access and 'Flash' use.
Xerte Online Toolkits (XOT), an open source e-learning development tool developed by the University of Nottingham and permitted for use online under the GNU Public License, 21 was used to develop this e-learning package.XOT is compatible with almost all hardware.The software allows interactive e-learning to be produced with simple tools.In terms of navigation, XOT provides a list of contents on each page that can be accessed at any time during the elearning session.XOT allows for bi-directionality and so users can navigate forwards and backwards using on-screen arrows.Users can also exit the e-learning at any stage and a help facility is available.
The font type, size and colour were consistent throughout the e-learning package to ensure that it was user-friendly without additional distractors.The software allowed the colour scheme, screen size, volume, text size and font to be changed according to the users' individual preferences.

Content and presentation of e-learning
Suitable learning outcomes were agreed in consultation with genocide experts and education specialists in the university (Table 1).Following a consultation with our e-learning specialists, it was agreed that the maximum duration of the e-learning package should be 30 minutes in keeping with the overall aim to provide an overview of the topic.Much of the textual content within the package was drawn from material from a postgraduate module on this topic. 22Genocide is not usually considered to fall within the remit of public health so it was felt that the presentation of the material should follow the typical format used to teach public health problems i.e. defining the problem, describing the problem in epidemiological terms (e.g.herd immunity, risk factors for genocide, levels of prevention etc.) and the public health impact.This was undertaken to encourage the learner to make the connection between genocide and public health concepts.To outline strategies to prevent genocide and address its consequences The e-learning package 23 was designed to accommodate a number of learning styles, through the variation in text, videos, images and interactive learning tasks that enable effective and engaging learning.Interactive tasks also facilitated deep learning rather than mere recollection of facts.

Pilot testing
The e-learning package was sent to 20 second year medical students for pilot testing and detailed feedback.Fifteen students completed the e-learning package (with the pre and post learning surveys), and the pilot testing feedback questionnaire which included free text response questions.
Key themes from the pilot evaluation are summarised in Table 2. Minor changes were made to the e-learning package such as technical issues with the interactive tasks, presentation of the material and spelling errors.Some amendments were also made to the pre and post-learning survey questionnaires.

Evaluation approach
Criteria for evaluating effectiveness and usability: 'effectiveness' was defined as positive changes in knowledge outcomes and attitudes in relation to genocide.A positive change in knowledge outcomes was indicated by an increase in the proportion of students with correct responses after completion of the e-learning package as compared to baseline knowledge scores.A positive attitude was defined as the recognition that genocide was a public health issue, that health care professionals had a role in genocide prevention and a willingness to engage in genocide prevention activities at a personal level.An increase in the proportion of students responding in the affirmative to these options as compared to baseline was considered a proxy measure of 'positive attitudinal change'.

Study design
In order to evaluate the impact the e-learning package had on knowledge outcomes and change in attitudes related to genocide, a simple pre-and post-learning comparison study was used.This was done by incorporating a pre and postlearning survey questionnaire within the e-learning package itself.The 'usability' of the e-learning package was assessed through questions asked after the e-learning activity was completed.

Study population
The e-learning package content was designed to be accessible to anyone without prior public health knowledge.For the evaluation a deliberate decision was made to test this package with second year medical students.This was because the second year students would have been exposed to e-learning during their first year and therefore any feedback on the elearning design itself would probably offer a fairer reflection on the specific e-learning package being tested.The elearning package with the embedded pre and post e-learning questionnaires was sent via an online link to all 253 second year medical students at the University of Nottingham for completion.The free version of the commercial survey software Survey Monkey was used for the questionnaires.

Structure of the questionnaire
The pre e-learning survey consisted of 7 questions: 1 on previous e-learning use, 3 to assess knowledge outcomes and 3 to assess attitudes to genocide.The post e-learning survey consisted of 10 questions: 5 questions on the design and usability of the e-learning package itself, 2 to assess knowledge outcomes and 3 to assess attitudes to genocide (Appendix 1).The questions on knowledge were presented as multiple choice questions with a single best answer.To avoid the risk of testing bias i.e. better post-test performance on a question because of the pre-test rather than the elearning package itself, the knowledge questions were reframed to test the same concepts in a different way. 26ble 2. Feedback from the pilot evaluation An un-weighted scoring system was used with each correct answer being awarded one mark.For questions assessing attitudes and eliciting feedback on usability, a range of question formats were used including: multi-option questions, dichotomous response questions, interval scale scoring questions and Likert item response type questions.For the Likert item response type questions, the 'neutral' category was eliminated to force respondents to take a position.A deliberate decision was taken based on the pilot testing to use a range of question formats to avoid mechanistic responses.
Design and presentation of e-learning package 'Used photos and videos, highly interactive and informative' 'Perhaps more interactive sessions' 'I liked how in areas where you can find out only the things you don't know unlike in a lecture where you are forced to listen to facts you may already know.Here I enjoyed the option to go at my own pace and taking more time on new concepts and whizzing through areas of genocide I have been previously exposed to (e.g.history GCSE).' 'Consistent design.Not too much text on each page.Content was divided up well into topics to make new knowledge less daunting' 'A lot of textual information maybe considers some more visual ways to explain information.To aid recall of information maybe have short 2/3 questions every couple of pages to test' 'Looked very professional' 'Too much information on most slides' 'Good use of colour scheme and layout' 'Great design, images and videos.Good quizzes and questions to test knowledge' 'All positive.The use of colours was good; not too bright, and not too bland either.The order of the presentation had a nice flow to itit was logical and flowed well' 'All aspects of the package were positive.I especially liked the drag and drop parts a lot.It was a nice way to keep me engaged in the package, rather than just reading a large mass of material and detail' Content of e-learning package 'Very good content' 'Interesting topic presented with interesting information' 'Very good!!! ' (sic) 'Good that it looked at lots of examples but a lot of information to be taken in' 'Very informative content with clear layout' 'Detailed and interesting' 'I liked it a lot.I feel that I actually learnt a good, sufficient deal about genocide.I didn't realise the relation between genocide and public health in medicine until using this package' Duration and depth of e-learning package 'I think I was losing concentration towards the end but the information was relevant and I felt it was important for it to be there so would rather sacrifice the time to learn about all the stages of genocide and their public health impact' '(Duration and depth of e-learning package) seemed about right to me' 'Too long.Too much info' 'Very detailed but maybe a bit too much text' 'Good coverage' 'The package was well balanced and contained information which was just right' 'Good length considering the amount of info covered' 'It maintained my interest throughout.I feel that it's too long and had good enough coverage of the topic; good enough breadth and depth of the issue.That's why I chose 5 (on the rating scale)-right in the middle, not too long and the topic coverage amount was just right'

Data analysis
For questions assessing knowledge, the percentage of students getting a correct response before and after the e-learning package was compared.For questions assessing positive attitudinal change in relation to genocide, the proportion agreeing with the options previously described as being positive attitude indicators, before and after the elearning package was compared.For questions assessing usability of the e-learning package, where interval scales (1-10) were used, the median scores were presented with a score above 6 indicating a favourable response.For questions assessing usability of the e-learning package, where Likert item type responses were used, the proportion of students responding 'agree/strongly agree' was compared to those responding 'disagree/strongly disagree'.Constraints on the types of data provided by the free version of Survey Monkey meant that only summary statistics were available and testing for statistical significance was not possible.

Ethics approval
The Division Education Committee deemed this evaluation exempt from full ethics approval process by the Medical School Ethics Committee as the aim of this project was to improve the student learning experience and involved minimal risk to the participants.A condition of this exemption was that the evaluation would not include randomisation of students to the e-learning package.

Results
This section presents the evaluation results.The e-learning package was sent to 253 second year students, 98 completed the pre-learning questionnaire but only 96 completed the post-learning questionnaire (total response rate of 38%) and were included in the evaluation of the e-learning package.
The results summarised in Tables 3 and 4 suggest that the e-learning package performed well in terms of user satisfaction, interactivity, design, achievement of learning outcomes, as well as influencing learner attitudes to genocide.

Summary of key findings
The evaluation results summarised in Tables 2 and 3 show that the e-learning package scored highly on 'usability' in terms of interactivity, pace and delivery of core learning material in an engaging way.The results summarised in Table 4 suggest that there were some short-term gains in genocide related knowledge and positive attitudinal expression immediately following the completion of the e-learning package.

Strengths of the Study
The e-learning package itself was designed to be studentcentred and this was demonstrated in the positive student feedback.The results suggest the effectiveness of the elearning package in improving knowledge and promoting positive attitudes at least in the short-term.Health professionals are known to have a role to play in genocide prevention, and the evaluation of this educational package has shown its potential use in promoting positive attitudes towards genocide prevention in medical students.Considering the knowledge-attitudes-practice (KAP) model of health behaviour, this would be the first step towards change in behaviour relating to genocide prevention.However this could not be assessed in the evaluation as behaviour change following qualification is a long-term outcome.Moreover, behaviour change, in this case being actions to prevent genocide, is not only dictated by knowledge, but by contextual factors.On the other hand, advocacy for genocide prevention by health professionals educated on this topic is a reasonable goal to expect.

Limitations of the Study
The response rate to the questionnaire was limited.Out of a class of 253 second year medical students, 98 initially responded to the pre e-learning survey, 2 subsequently dropped out, leaving 96 participants in the stage II evaluation testing (38% response rate).Considering that the completion of the e-learning package was voluntary, this could be considered a relatively good response rate to generalise findings to the study population.The evaluation study used a quasi-experimental approach involving pre-and post-intervention assessments of knowledge and attitudes, which may have caused respondent burden, potentially leading to respondents not taking the survey seriously.Moreover, an inherent bias with all quasiexperimental approaches is self-selection of study participants.It is likely that our study participants already had an interest in the topic of genocide therefore were more engaged with the e-learning package.This could have led us to overestimate the effectiveness of the e-learning package in achieving learning outcomes.We have already mentioned the threat to internal validity from testing bias, whereby, the pre-intervention test primes the intervention group for better performance in the post-intervention test, irrespective of the intervention.
Another threat to internal validity is the 'history threat' as described by Trochim 25 where some historical episode rather than the educational intervention could have led to the outcome observed is a common limitation. 25However, in this e-learning package there was only one hour between the pre and post e-learning surveys, it is therefore unlikely that another concurrent event could have affected the knowledge or attitude outcomes.
It is worth pointing out that any likely gains in knowledge or expressions of positive attitudes towards genocide prevention following the e-learning package could be purely short-term gains and without further consolidation or the right contextual factors, it is unlikely that long-term knowledge or attitudinal change would result.However, the results do suggest that the e-learning package has served to spark an interest in a topic that may not have been considered relevant by medical students before this point.
Finally, this study evaluated an e-learning package as a learning tool.Although approximately 97% stated that the elearning package was sufficient to give them a good understanding of the topic as opposed to a traditional lecture and ideally, the study should have evaluated the effectiveness of the e-learning package as compared to a traditional lecture and compared the two styles for their relative outcomes.However, this is rarely a pragmatic option for practitioner based research limited by timetabling and resource constraints.A parallel survey of third year medical students from the University of Nottingham showed that just over two thirds of respondents (82/121) thought that genocide should be part of the public health curriculum and 84% wanted to learn more about genocide following the survey with their preferred option being e-learning. 26

Conclusions
In conclusion, our evaluation of an e-learning package on genocide and public health found that it was well received by students and was associated with short-term knowledge gains and positive attitudes in relation to genocide prevention.We invite other educationalists to use and evaluate this resource with their own students.

Table 1 .
Learning outcomes covered in the e-learning package To define genocide and recognise actions that constitute genocide To outline the key stages in genocide, including the progression from exclusion to genocide  To recognise the extent and health impacts of genocide To recognise the relevance of genocide to public health and the medical profession 

Table 3 .
Learner feedback on the design and usability of the e-learning package (n=96)

Table 4 .
Comparison of pre-and post-learning scores to assess learning and attitude outcomes among the intervention group (n=96)