Evaluation of student-prepared portfolio of guidelines in undergraduate clinical skills using a mixed methods study

Objectives: The aim of the study was to assess the acceptance and usefulness to students of a studentdeveloped portfolio of Guidelines (GLs) in clinical skills, as a tool for competency development in the early years of the undergraduate medical program. Methods: A mixed-method design was employed to explore students’ perceptions of the general usefulness of their portfolio of GLs in focus group discussions. From the results of these discussions, a questionnaire was developed to ascertain the prevalence of the views that emerged from the qualitative data amongst a whole cohort of Year 6 students. Results: Students rated the usefulness of their portfolios very highly and the uses of the portfolios evolved as students progressed to their final year, when they were also able to perceive the importance of the acquisition skills required whilst developing their own GLs, for use in subse-


Introduction
In the last decade there has been a significant move towards competence based medical education and away from the simple acquisition of theoretical knowledge. 1,2One important challenge has been to find teaching and assessment instruments supporting competence development and assessment of whether competence has been achieved. 3,4he portfolio could be one such suitable instrument.
Portfolios have been introduced at all stages of medical education over the past two decades especially in undergraduate medical education, and have been the subject of much educational research. 4They have been widely used in health care education and have a strong potential for enhancing learning. 5They have contributed to students' professional and personal development, 6 and may stimulate reflective learning. 7However, the introduction of portfolios with the purpose of supporting competence development has met with mixed success. 6,8,9 Gelines (GLs) form an important aspect of evidencebased clinical medicine, and the development of GLs by students has the potential to support development of competence.In our early years clinical skills program (years 1 and 2 medical students), we have incorporated the development of a portfolio of GLs not as a tool for reflection, but rather as a tool that students can use to help them develop their competencies in clinical skills.To our knowledge, this is the first time that a portfolio specifically directed to clinical skills has been implemented and evaluated in very early, undergraduate medical education.

Context of the Study
The coordinator of the Years 1 and 2 MBBS clinical skills program at the University of Adelaide, South Australia, first introduced the concept of students developing their own portfolio of guidelines in clinical skills in 2006, initially for only Year 2 students and then for both Years 1 and 2 students from 2007 on.Prior to this, lecturers and tutors had developed GLs for the students and clinical skills were taught in a more didactic manner.The aim of the change was to encourage students to become more self-directed in their learning and to become more actively involved in a way that would lead to a deeper understanding and retention of clinical skills.
A Case-Based Learning (CBL) approach is used across the whole integrated curriculum in the first three preclinical years of the medical program at the University of Adelaide.The GLs in clinical skills developed by students are related to the cases they are studying as this gives students the opportunity to integrate the different areas of the curriculum (Scientific Basis of Medicine, Medical, Professional and Personal Development, Clinical Skills and CBL tutorials).One week prior to each clinical skills tutorial, the students are given the learning objectives for the tutorial, a list of GLs that they need to prepare for the tutorial and a list of suggested resources.
Since 2006, the portfolio of GLs in clinical skills has become an important aspect of the Years 1 and 2 clinical skills programs and has been assessed both in a formative and summative manner.The portfolio of GLs in clinical skills was not a "tick box competency" portfolio.Students develop themselves, through their own directed research; certain clinical skills.Hence, students take ownership of creating their own guidelines on how to perform certain clinical skills.It was therefore appropriate to investigate the acceptability of this approach to students, and the degree to which the portfolio of GLs continued to be used by students as they progressed through subsequent undergraduate years.Therefore the purpose of this study was to evaluate the acceptability and usefulness of a self-developed portfolio of GLs in clinical skills to early undergraduate medical students, and to determine whether these GLs are used and/or modified in subsequent undergraduate years after their introduction in Year 2.

Method
Students who participated in the research were Year 6 students from the 2011 MBBS graduating cohort.They had begun their portfolio of GLs in clinical skills in Year 2 of the MBBS program at the University of Adelaide in 2007.
A mixed-methods design, "the exploratory sequential design", 10 was employed to explore students' perceptions of the general usefulness of their portfolio of GLs.Focus groups were conducted amongst a subset of the cohort and from the results of the focus groups, a questionnaire was developed to ascertain the prevalence of the views that emerged from the qualitative data, amongst the whole 2011 MBBS cohort of Year 6 students.

Sampling
Students who participated in focus groups were Year 6 medical students who had chosen Medical Education as an elective placement.This was a convenience sample, where students were selected for their accessibility because they were present in the medical school whilst they undertook their elective.As tutors of Year 1 and 2 clinical skills sessions themselves, they were also able to observe the development and use of portfolios of GLs by their junior colleagues.
The entire 2011 Year 6 student cohort was invited to participate voluntarily in the questionnaire, which was made available for students to complete online.

Collection of qualitative data
A total of seventeen students participated in four focus groups, each group comprising 3-6 students and lasting for approximately one hour.Students were provided with a list of the 30 clinical skills for which they had been required to develop GLs in 2007 and they were then asked to discuss the questions summarised in Table 1.

Written comments about the portfolio of GLs
Four focus groups were deemed sufficient, as data saturation had occurred by the fourth interview.The focus groups were semi-structured in that although all questions were covered during the interview, the questions were not strictly adhered to, as new information on topics outside the range of the questions was welcomed.The focus group discussions were recorded and transcribed.Data in the transcription were coded manually, initially under headings related to the six questions that had been asked.Any data that did not fit into one of these headings were coded in a miscellaneous category.After initial coding was done by one of the researchers, another member of the research team independently coded the data to check for inter-coder agreement which was found to be high, as there was complete agreement on sorting data into the initial coding categories.Analysis of exploratory data from focus groups suggested five areas for subsequent investigation and these areas were designated as the sections in the questionnaire (see Table 1).

Development of questionnaire and collection and analysis of quantitative data
Analysis of focus group data from questions 4 and 5 (Table 1) showed that students changed their GLs in ways other than expansion or creating new GLs.Therefore in the questionnaire, these two questions were condensed into one section "modification of GLs" (Table 1).Discussions from focus groups also revealed that students perceived the aims of preparation of their own GLs differently in Year 6 than they had in Year 2. Section 5 of the questionnaire was modified to investigate whether this was a general change of perception in the whole cohort.Two final sections were added with Section 6 investigating the overall usefulness in subsequent years of the portfolio of GLs that the students had begun in Year 2 and the final section giving students the opportunity to submit written comments about the portfolio of GLs.The questionnaire was trialled with a small group of Year 6 students who verified the clarity of the questions, before it was administered in an online format to the whole Year 6 cohort.
The study was not exempt from ethical review as data were collected during the year through focus groups and from an end of year questionnaire.There was no use of existing collections of data and records.Ethics approval for the investigation was obtained from the University of Adelaide Human Ethics Committee in March 2011 for a period of one year.In accordance with ethics guidelines, all participants were assured of the confidentiality and anonymity of any data they contributed and they received information sheets explaining the secure storage of all data.

Qualitative results from focus groups
Many students who participated in focus groups reported that they did not appreciate the potential usefulness of GLs when they were introduced to them.As one student explained, "I guess probably at the time I didn't think they were as useful.... but as you approach the clinical years and you start examining patients with pathology more often, you see how useful they are."(Female 1, Focus Group 2) Additional relevant comments included: "In Year 2, I did not understand how useful the portfolio would be in later years.Some students spent only a small amount of time in preparing them.The attitude was just to get the job done as the preparation of the portfolio of GLs was seen just as a necessary assessment tool."(Male 2, Focus Group 3) And "Most students would not prepare for clinical skills unless they had to do their guidelines."(Female 3, Focus group 2) Several students felt that the self-development of GLs would have been better understood at the time if more guidance had been given "about the level of detail to include" (Male 1, Focus Group 2).They also made it clear that feedback was regarded as valuable, because although the portfolios of GLs were handed in for assessment, they were often returned with no comments or markings causing students to be concerned that errors in their portfolio of GLs would not be corrected.
In Year 6, students still valued the GLs, but importantly, also valued the skills that they had gained through selfpreparation of GLs in Year 2, reflecting that: "…the actual skill of being able to develop guidelines is important because I continue to do it."(Female 2, Focus Group 1) They considered the ability to write new GLs as "... an important skill of being able to find new resources, synthesise them and then document them in a way that is digestible and comprehensible for you to use yourself."(Male 3, Focus Group 2) Another student commented that "... students need to be made aware that the clinical skill GLs developed are an evolving document or a work in progress that may be changed after clinical skills tutorials and that they will be using the guidelines for many years to come".(Female 2, Focus Group 4) Students described how they used their GLs in many different ways but found them most useful for carrying out physical examinations and preparing for their Objective Structured Clinical Examinations (OSCEs).As one student explained, "It (portfolio of GLs) helps with every (physical) exam that you do, because all of your clinical reasoning for your exam findings is in that folder."(Female 3, Focus Group 3) For physical examinations, students also reported that "...GLs are useful to give you a framework for doing a skill ..." (Male 2, Focus Group 3) Students saw their GLs as a very important tool for preparing for OSCEs and as one student commented, directed after that.So it is actually good to have that teaching for first and second year."(Female 1, Focus Group 1) Another student explained how during preparation for OSCEs, she found GLs useful because they contained both detailed notes and summaries.
"So if I kind of needed to do my six minute OSCE there and then and I needed something to refer to, I understood it because I had the detailed notes, but then there was my quick cheat page if I needed."(Female 2, Focus Group 2) Students modified their GLs in subsequent years by summarising, expanding, rewriting and updating data.One student described how this often resulted in quite different GLs: "My GLs are nothing like they were in second year….I would have had like ten pages on the cardiovascular (CVS) exam which is now in two pages or one page….They are definitely not the same."(Female 1, Focus Group 2) These changes were often made as a student's understanding of a topic increased.New GLs were written when students were studying for their OSCEs in groups but did not have GLs to cover all possible OSCE stations.They would assign the preparation of new GLs to group members and then share these with the whole group.
When the Year 6 students were asked to reflect on what they, as Year 2 students, had considered to be the purposes for student-prepared GLs, one student described how it However, students reported that by the time they were in Year 6, they placed less importance on this purpose of student-prepared GLs and more importance on the purpose of providing them with the skills to develop their own GLs in future years.In both Years 2 and 6, students saw their GLs as extremely important in preparing them for practising clinical skills.
The overall usefulness of the portfolio of GLs was summarised by one student as follows: "The portfolio of GLs, while initially seeming overwhelming in the depth and complexity in second year, has been a lifesaver throughout my medical degree.I have continued to add to, adapt and modify my portfolio throughout my training, and am now using the same approach to develop new GLs in preparation for sitting clinical examinations for higher medical training."(Male 1, Focus Group 4)

Quantitative results from questionnaire
Quantitative data from the questionnaire provided information about how the whole cohort of Year 6 students used their portfolio of GLs.The response rate from the cohort for the online questionnaire was 76.9% (90/117 students).Table 2 shows the items in Sections 1-6 of the questionnaire, with the last column in the table summarising the positive responses for each item.
Results for sections 5a and 5b of the questionnaire are shown in Figure 1 and compare how students would have rated the aims of the student-prepared portfolio of GLs when they were in Year 2 with how they rated them now they were in Year 6.The mean student rating of each aim (out of a total of 4) was calculated and for each aim, the significance of the difference between rating as a second year student and as a sixth year student was tested using the paired t-test.
The results from the cohort were consistent with the reports in the focus groups, in that preparation for clinical skills sessions and the use of the GLs as an assessment tool were regarded as statistically significantly less (p<0.05)important aims from the Year 6 perspective, than they had been in Year 2. In contrast, the Year 6 students rated the acquisition of skills to prepare their own GLs as a statistically significantly, more important aim that they had realized in Year 2.

Discussion
The original aim of our study was to assess how acceptable and useful the portfolio of GLs was in learning undergraduate clinical skills, and whether the GLs were used and expanded on in subsequent undergraduate years after their introduction in the second year of medicine.This information would contribute to quality improvement of the clinical skills course and also enhance the credibility of this aspect of clinical skills' learning.
It was clear from the qualitative results that students had a greater appreciation of the benefits of developing their own set of clinical skills GLs in Year 6 than they had in Year 2. Students in Year 6 were able to perceive that they had acquired new skills through the process of developing their own GLs, and were also aware that they had not foreseen this outcome when they were in Year 2. The content of the GLs, particularly after modification over subsequent years, was regarded as particularly useful for carrying out physical examinations during clinical attachments and preparing for clinical examinations.
The qualitative data informed the development of the questionnaire in two important ways.Firstly, it became apparent that students had been modifying their portfolios in ways which had not been predicted, particularly in finetuning and abbreviating previously large, relatively unfocused GLs into more useful focused GLs as they gained more clinical experience.Secondly, the issue of overall usefulness of the portfolio of GLs was perceived by senior students to be as much, or more, about the development of the skills involved in preparing their own GLs as about the content of the GLs themselves, although this was also perceived as useful by many students.
The high response rate to our questionnaire/survey (76.9% of the cohort) improved the reliability of the results  to introduce students to the concept of GLs so they have the skills to prepare their own GLs in subsequent years 64.8 Section 6. Please rate the overall usefulness in subsequent years of the portfolio of GLs that you began in Year 2 on a scale of 1-6 (1=not at all useful, 6=extremely useful) % students rating usefulness 6-10 82.0 and their applicability to the whole cohort of students.It is also reasonable to extrapolate the results to other cohorts, although a follow-up study would be required to confirm this.
Our quantitative data showed that the vast majority of respondents (84.3%) perceived that the introduction to the concept of a portfolio of GLs in Year 2 had provided them with the skills to prepare their own GLs in subsequent years (Table 2, Section1).Students recognised that the foundations for preparation of new GLs were laid down in Year 2, and also recognized, from their Year 6 perspective, that they had not appreciated this benefit of GL development during Year 2.
Portfolios of GLs were found to be used often or always for physical examinations (80% of cohort) and to prepare for OSCEs (75.8%) in both early and later clinical years (Table 2, Section 2).The usefulness of the development of GLs for physical examinations was supported by the results in Table 2, Section 3, with 93.3% of students giving this use a rating of 6-10 (on a scale of 1-10).
It was interesting to note that most students (27.8% never and 35.6% sometimes), did not often use their GLs as a tool for reflecting on how they had performed a physical examination or presented a case history in the hospital or other clinical setting.The portfolio of GLs was not designed as a journal for reflection but rather for competence development.2][13] Although reflection was not the aim of these portfolios, perhaps they could have been used for this purpose if this had been made more explicit to students.A topic for future research could be whether student outcomes would improve if students used their portfolios for reflecting on physical examinations and case presentations.
Students modified their GLs in a variety of ways, with a high proportion of students expanding, summarizing and writing new GLs in later years as their knowledge and skills base increased (Table 2, Section 4).The majority of students also updated their GLs with current information.It was clear that over time, the students had developed a clearer understanding of the role and purpose of developing their own GLs as they become more skilled and experienced in the clinical milieu (Figure 1).The aims of preparation for and focus on clinical skills sessions and preparation for practising clinical skills in clinical settings were perceived as important at both time points.However, the Year 6 perspective provided a better understanding of the importance of the acquisition of the skills through developing their own GLs, for use in subsequent years.Interestingly, the significant changes in ratings of the four major aims of GL exercise from Year 2 to Year 6 resulted in the Year 6 views being closer to the views of the staff teaching the course, perhaps implying a process of increasing maturity.
Overall, the portfolio of Guidelines was perceived as continuing to be useful over Years 3 to 6 by the great majority of students in Year 6 (Table 2, Section 6).The quantitative data from the questionnaire thus strongly supported the qualitative data from the focus groups, indicating that the views obtained from the smaller sample of students who had participated in the focus groups were prevalent amongst the whole cohort of Year 6 students.
The use of portfolios in undergraduate education has met with mixed success 6,8,9 and has been more successful when not used in isolation but as a part of other educational activities, for example, in tutorial groups. 5,6,9,14,15Our student-developed portfolio of GLs is an important part of our early undergraduate clinical skills tutorials and we believe that its use in combination with other learning activities for students, such as the practice of history taking, performing physical examinations and receiving tutor and peer feedback, has contributed to its success.
6][17][18][19] This may have been the experience of some of our cohort in second year but this appears to have changed as students progressed to Year 6, due to a greater appreciation of the contribution of self-development of the portfolio to skills acquisition and learning.
Portfolio formats have also influenced the contribution made to learning.A clear and flexible structure is required for an effective portfolio.Clear instructions are also important. 5,11,20Students in our study reported a sense of uncertainty in Year 2 as to the level of detail expected, and this issue needs to be addressed in our ongoing development of the portfolio activity.Portfolios are also more highly appreciated when learners are given a certain amount of freedom to determine content, 5 and students by Year 6 had become aware of their ability to change their GLs as necessary long after they had completed the Year 2 clinical skills program.Mentoring and feedback by tutors and educational supervisors make a very important contribution to the success of portfolios. 5,7,9,17,20mproving feedback from clinical skills teachers is an area requiring improvement to maximise the benefits of the portfolio of GLs.
There were several limitations to the study.Students participating in the focus groups had chosen medical education for their elective and this could have created a bias in the results because of their special interests in this area.Students self-selected for participation in the questionnaire and this could have created bias as we did not investigate the characteristics of students who did or did not participate.However, the very high response rate makes this less likely to have influenced the results.The fact that qualitative data was collected from Year 6 students who were asked to reflect on their experiences of the previous four years, could have introduced an element of inaccuracy in their ability to recall these experiences.The results were not primarily intended for generalisation or transferability but for use within the clinical skills course investigated.However, this approach may be generalizable to other courses and contexts with appropriate local modifications.

Conclusions
The student-developed portfolio of GLs in clinical skills has been well accepted by students who have found their GLs to be useful for a variety of purposes as they progressed through their medical course.The requirement for studentdevelopment of the portfolio of GLs in the early undergraduate years also gave students the skills to prepare their own GLs in subsequent years.Further research in this area would be valuable to address several unresolved questions, including identification of which GLs were particularly useful and whether the portfolio of GLs could indeed be expanded and used as a tool for reflection.As one student had suggested "Clinical skills GLs could be used with case presentations, to reflect on what was done poorly and tips for the next time.The best portfolios are those of students who have actually reflectively learnt, looked back and doing it right when writing up the case presentations".
It would also be of considerable interest to identify any correlation between students' approaches to GLs (quality, attitude, modification and ongoing use of GLs) and their performance in OSCEs and in other clinical assessments.

"
… made me look and prepare for the clinical skills session every week, really get into focused preparation so when I came to clinical skills actually I knew what to do and what we were doing."(Male 1, Focus Group 2)

Figure 1 .
Figure 1.Comparison of mean rating of importance of aims of the portfolio of GLs as a Yr 2 student and as a Yr 6 student (paired t-test) rating of aim out of a total of 4 Aims of student prepared portfolio of guidelines Rating as Yr 2 student Rating as Yr 6

Table 1 .
Relationship between questions from focus groups and sections of the questionnaire

Table 2 .
Results of questionnaire (N=90)Section 1. Please indicate your level of agreement with statements about the usefulness of the portfolio of GLs that you began developing in Year 2 on a scale of 1-6: (1=strongly disagree, 6=strongly agree)