Teachers ’ experiences of the role and function of the mini clinical evaluation exercise in post-graduate training

Objectives: To investigate teachers’ experiences and views regarding the mini clinical evaluation exercise as an assessment tool and to evaluate its feasibility as well as its influence on teachers’ pass/fail decisions and feedback delivery. Methods: Seventeen teachers who had all used the mini clinical evaluation exercise in assessing residents at least twice during the study period were interviewed. Transcripts of the interviews were analyzed qualitatively using a phenomenographic approach. Results: All teachers considered the mini clinical evaluation exercise as a useful assessment tool that promotes direct observation and constructive feedback. The format was considered feasible because of its easy adaptability to daily practice. Uncertainty as to what should specifically be observed during encounters, interpersonal relationships, and preconceived notions of resident performance were given as reasons for teachers’ difficulties in defining a pass or fail score. Teachers generally tended to be averse to failing residents. Conclusions: This study shows that teachers perceive the mini clinical evaluation exercise as a feasible and useful formative assessment tool. Contextual factors such as interpersonal relationships, preconceived performance notions, and lack of specific guidelines and performance standards appear to explain why teachers tend to be reluctant to fail poorly performing residents.


Introduction
The mini clinical evaluation exercise (mini-CEX) is designed around both the skills most commonly performed by residents in actual patient encounters and the educational interactions that attending physicians routinely have with residents during teaching rounds.It is a performancebased evaluation method that can be used to assess selected clinical competencies (e.g.physical examination, communication and interpersonal skills, professionalism) in the clinical training context. 1 In educational practice, decisions are rarely based on research and, especially with regard to assessment, teachers, students, and institutions tend to have strong opinions that are largely based on sentiments and tradition.Teachers are usually unaware of educational research or do not consider it important. 2The extent to which an assessment procedure is accepted by those involved in its execution is a crucial element in the introduction of a new assessment method.
Several studies have focused on teachers' and students' experiences with the mini-CEX.Norcini and colleagues showed that examiners as a group were very satisfied with this method. 1Torre and colleagues showed that the mini-CEX was highly rated by students and evaluators as a valuable tool to document direct supervision of clinical skills. 3Alves de Lima and colleagues also found that residents and evaluators were satisfied with the mini-CEX format.Residents' ratings ranged from 5 to 9 on a ten point scale (mean 8.08 + 0.83) and evaluators' ratings ranged from 6 to 9 (mean 8.06 + 0.74). 4Assessment procedures that are not accepted by teachers or students are likely to be discarded eventually.Probably the best course of action in achieving acceptance is to make strategic use of information about teachers' and students' beliefs in order to gain their commitment. 5This seems particularly applicable with work-based assessment where the value of the assessment appears to be determined by the users of the instruments rather than by the instruments themselves. 6he purpose of this study was to investigate how teachers value the mini-CEX as an assessment tool, its feasibility, the influence of contextual factors on teachers' pass or fail decisions, and feedback delivery.We conducted a qualitative study in which we interviewed teachers about their experiences with the mini-CEX.In analyzing the interview data we used a phenomenographic approach in order to gain a thorough understanding of teachers' experiences.Unlike many other assessments, where the teacher is largely a passive instrument, the mini-CEX requires teachers to actively construe their judgments.We need to understand this process as well as contextual factors that influence teachers' judgments not only in terms of feedback but also in terms of summative decision making.We expect that such insights will promote the use of the mini-CEX and help to optimize the format. 7he study was conducted at the Cardiovascular Institute of Buenos Aires (ICBA), a 55-bed cardiovascular training hospital located in the federal district of Buenos Aires, the capital of Argentina.Both the institution and the cardiology residency program are affiliated with the University of Buenos Aires (UBA).

Participants
The participants were cardiology teachers delivering the postgraduate cardiology training program.The following criterion was used to select candidates: faculty members who had used the mini-CEX to assess residents on at least two occasions between May 2005 and May 2006.Out of 24 teachers, 17 met this criterion and we invited these to participate in the study.All 17 agreed to participate.Participation was voluntary.Three, eight, and six teachers had used the mini-CEX twice, three to five times, and more than five times, respectively.Three had assessed the same resident more than once.
In 2002 the mini-CEX was introduced in the cardiology residency program for formative assessment.Since 2004 it has been used for summative assessment and all the participants had used the mini-CEX during this period.In 2004 the program director met with each assessor to discuss general performance standards.The residency program of the Cardiovascular Institute of Buenos Aires lasts four years and the Institute accommodates four residents at the same time.Every year each resident takes part in five mini-CEX assessments.Residents who fail have one chance to re-do the mini-CEX, only this time not with their teacher but with the program director.For the mini-CEX, one faculty member observes and evaluates a resident who is taking a history and performing a physical examination on an inpatient, an out-patient or on a patient in the emergency department.After the encounter the resident presents a diagnosis and management plan to the faculty member, who then completes a short evaluation form and gives feedback.The mean time used for the mini-CEX was 42.77 minutes (SD=19.97).
Residents' performance is rated on the following competencies: medical interviewing skills, physical examination skills, humanistic qualities, clinical judgment, counseling skills, organization skills and efficiency, and the resident's overall clinical competence.Ratings are given on a 9-point scale: 1, 2, and 3 indicate unsatisfactory performance, 4 marginal performance, 5 and 6 satisfactory performance, and 7, 8 and 9 indicate superior performance.A rating of 1, 2 or 3 means a fail.In addition to the data on the resident's performance, the teacher also records the site of the assessment (the inpatient service, the out-patient clinic, or the emergency department), the complexity of the case (low, moderate, high), and the patient's gender, age, and main medical problems and diagnoses.
This research protocol was approved by the institutional Review Board of the Cardiovascular Institute of Buenos Aires.

The phenomenographic approach
We used an inductive method to explore teachers' experiences with the mini-CEX as an assessment tool, its feasibility, and its influence on their rating and feedback delivery strategies.Using a phenomenographic analytical approach we repeatedly read the transcriptions of the interviews in order to gain an in-depth understanding of each interview. 8

Analysis
We designed a procedure for the study aimed at minimizing interference with daily clinical routine.Thirty days before their interview participants received a written invitation.We conducted one open, semi-structured interview per participant and continued interviewing until we considered that saturation had been reached.We used interviews to collect data, because this method would allow each participant to elaborate on his/her perceptions of the mini-CEX.We used an interview guide (see Appendix) to elicit the participants' views regarding the mini-CEX with regard to its feasibility, teachers' strategies to arrive at pass/fail decisions, teachers' appraisal of the format, and feedback.One of the investigators conducted the interviews, which lasted, on average, 55 minutes (range 35-80 minutes).
We audio recorded interviews after obtaining consent from each participant.We then transcribed the recorded interviews and two investigators independently coded the transcripts.In analyzing the transcripts, the investigators related and compared the sections of each interview with the corresponding sections of the other interviews. 9The investigators categorized the similarities and dissimilarities between the interviews that were identified during the analysis. 10,11

Results
The results are presented for each of the four issues addressed during the interviews (see Appendix).The results are holistic.They represent common experiences recounted by teachers and reflect the different ways in which the teachers experienced the mini-CEX assessment format.

Feasibility issues
The participants' views are grouped into one category: integration of the method in patient care and integration of teaching activities in daily practice.

Integrating patient care and teaching activities in clinical practice
All the teachers worked in very busy clinics and struggled to combine clinical and administrative duties with their teaching responsibilities.All the teachers agreed that the mini-CEX was a very useful assessment instrument, and they were looking forward to its further implementation.For most of them, the main challenge was to incorporate direct observation of residents into practice routines, something they were definitely not used to doing.The majority of the teachers were used to assessment in which they listened to residents presenting their findings from and records of physical exams.The teachers said that observing residents during history and physical examination enabled them to detect important errors in residents' performance.The teachers used two different major implementation strategies.Some teachers scheduled appointments for the mini-CEX in their daily schedules, while others preferred on the spot observation when deemed appropriate.Some teachers agreed upon a schedule with the resident and some even asked residents to remind them of the arranged date and time of the mini-CEX.The teachers suggested that a pocket-sized form would be of great help.
…"No, there weren't any great difficulties in the implementation; the major difficulty is scheduling time…..to find some quiet time to devote to this type of protocol"… (Interviewee MJ 1.3)

Pass/fail strategies
This item relates to teachers' feelings when they had to decide whether a resident's performance merited a pass or a fail.The participants' answers could be grouped into two categories: feelings of discomfort or insecurity in making this decision and their perception that the format could induce avoidance of fail decisions.

Feelings of discomfort or insecurity in making pass/fail decision
Most of the teachers were confident of their capability to determine whether a resident was performing poorly, but they had difficulty identifying the pass/fail threshold.They did not know which type of information or which specific behaviors they should document to support a judgment of poor performance.As a consequence, they tended to avoid asking questions until the resident had reached a satisfactory level of performance during the assessment.They felt uncomfortable giving pass/fail judgments, and they all agreed that, in borderline cases, they preferred giving a resident the benefit of the doubt.This could lead to inflation of scores and false positive decisions.
…"Yes, it depends on where the pass/fail point is considered to lie...but I guess that it is much more flexible, as there are many items... someone may therefore fail one item and do very well on another one"… (Interviewee FB 2.3)

Avoidance of fail decisions as a result of using the mini-CEX
Teachers reported difficulty establishing which level a resident was expected to have attained, due to heterogeneity among residents of the same year.Some residents do well on certain aspects, but perform poorly on other aspects.As a result, an overall competence judgment requires a delicate process of weighing different pieces of information and combining them into an aggregate decision.The teachers indicated that they perceived this to be a particularly delicate and difficult task.
In addition to the intrinsic challenges of the assessment task, the teachers mentioned the impact of personal relationships between teachers and residents.The teacher generally knows the resident and, consequently, has a preconceived notion of his or her competence.Also, the relationship of residents with the Institution plays an important role: the teachers felt that it was unacceptable to fail a resident whom one had personally selected for admission to the Institution's residency program.
…"I always know something about the resident's clinical competence, if the resident shows poor performance I start asking new questions because I am sure that he will do well"… (Interviewee KP 2.5)

Format issues
The teachers' views on this topic fell into two categories: assessment in an authentic environment and positive educational experiences.

Assessment in an authentic environment
The majority of the teachers indicated that the format enabled them to assess all aspects of clinical practice.In contrast to other formats, such as MCQ or essay questions, the mini-CEX allows realistic cardiologic clinical scenarios to be addressed in the assessment.Accordingly, realism was described as an important aspect of the use of real patient cases in an authentic clinical environment.Also, teachers valued the opportunity to confront residents with cases of different complexity.

A positive educational experience
Many teachers valued their exposure to students' work and the opportunity to review students' clinical skills, such as physical examination and communication skills, which was afforded by the mini-CEX.They found it a useful teaching tool especially as their feedback and direct observation skills improved and they realized that they could do it all in less time.They reported a learning curve in their capacity to provide feedback and use the format efficiently.Teachers admitted that the format promoted more intensive interaction with residents.

Feedback strategies
The teachers' views with regard to feedback strategies related to the feasibility of constructive feedback.

Feasibility of constructive feedback delivery
All teachers identified providing feedback as a key aspect of their role as educators, but they also indicated that the feedback they gave was generally brief and non-specific.They also felt insecure because they had not been trained in feedback delivery and they worried that their feedback might cause frustration, loss of motivation, or diminution of self-esteem.Furthermore, they did not know how to respond when a resident showed a certain reaction, such as anxiety.The teachers expressed satisfaction about the opportunity to assess performance immediately after direct observation.They appreciated that the mini-CEX enabled them to give residents immediate, constructive, and structured feedback, including recommendations for improvement and action plans.It was the teachers' experience that the mini-CEX facilitated the delivery of feedback and promoted more extensive and specific feedback.As noted before, they experienced a steep learning curve with frequent use of the format.

Discussion
We conducted a qualitative study to explore clinical teachers' experiences with the mini-CEX as an assessment tool, their perceptions of its feasibility and the influence of the format on their pass/fail decision strategies and feed-back delivery.As for feasibility, the teachers thought the mini-CEX was easy to apply and integrate with daily patient care activities.The teachers' views suggest that the mini-CEX has acceptable feasibility, and this resonates with the literature.Kogan and colleagues defined feasibility as the percentage of completed forms, average completion time and satisfaction ratings. 12They analyzed data from 162 students that underwent mini-CEX evaluations and completed a total of 1,297 forms (89% completion rate), with a mean number of 7.9 forms per student (range; 2-10, median 8).Torre and colleagues considered one mini-CEX per month to be sufficient to determine its feasibility, and they achieved a 100% completion rate. 3s for strategies to reach pass/fail decisions, the teachers' main response was that such decisions made them feel uncomfortable and uncertain, because they did not know what type of information or specific behaviors they should document during the mini-CEX.They also felt that contextual factors were likely to bias their judgment, and as a result they were likely to be too lenient.Other authors have reported similar findings.Dudek and colleagues explored factors, identified by supervisors as affecting their willingness to report poor clinical performance when completing In-Training Evaluation Reports (ITERs). 13They identified four major barriers to judgments of poor performance: 1. lack of documentation, 2. lack of knowledge as to what specifically to document, 3. anticipation of an appeal process and 4. lack of remediation options.In the present study teachers' judgments appeared to be most strongly affected by the absence of well-defined standards and probably also by the high stakes for residents and the relationship between teacher and resident.The teachers in our study indicated that uncertainty surrounding pass/fail decisions could cause them to be overly lenient.Comparable problems with standards for pass/fail judgments have been reported by other authors.Litttlefield and colleagues found that, compared with their colleagues, 27% of assessors were either very lenient or strict in assigning intraining evaluating scores.
Apparently, there was considerable variation in the standards and expectations which assessors think should be met by third-year medical students. 14Turnbull and Van Barneveld pointed to the dissonance between the roles of teacher and evaluator as a potential source of error when attending physicians evaluate the students they supervise. 15he work done by Eldin and colleagues suggests that faceto-face evaluation could yield higher scores than any other type of evaluation not involving direct contact between assessor and assessee at the time of the exam. 16e findings with regard to pass/fail decisions suggest that, when a new assessment format is introduced in a residency program, directors and faculty members should consider setting specific guidelines and performance standards for each level of training, thereby facilitating longitudinal assessment of individual residents.Alterna-tively, it might be wise to use individual mini-CEX evaluations as purely formative tools, which do not require a pass or fail decision, and to use the collection of all mini-CEX judgments for summative decisions.Such a collection of judgments, together with other assessment and performance information, might be a better basis for summative decision making.In this way, the emphasis in pass-fail decisions shifts to remediation and longitudinal personal development, which results in protection of both the individual assessor and the relationship with the resident.As a result, the focus in each separate mini-CEX evaluation is the resident's clinical performance in a specific situation, and the teacher does not have to pronounce judgment as to whether the resident is a good doctor.
Another issue that deserves special consideration is feedback delivery.In the present study the teachers saw the mini-CEX as an excellent opportunity to deliver instant feedback.One of the aims of assessment of residents' performance based on observation of individual patient encounters is to promote and optimize feedback, which requires teachers to identify which aspects went well, which needed improvement, and what action should be taken.This type of feedback is likely to diminish the pressure on both assessor and learner.Moreover, the richness of the feedback in individual encounters can contribute to informed and defensible aggregate judgment across encounters.
Several publications have described the mini-CEX as a tool for feedback delivery.Holmboe and colleagues reported results of an analysis of 107 feedback sessions after application of the mini-CEX: in 80% of the sessions the supervisor gave the residents advice regarding performance improvement at least once, in 61% the supervisor asked the resident to give his or her reactions, in 34% the supervisor asked the resident for self-observation and in 8% supervisor and resident developed an action plan together. 17As far as duration is concerned, Kogan and colleagues reported that the feedback session following the patient encounter of the mini-CEX lasted eight minutes and Hauer reported similar results. 12,18Later, Alves de Lima and colleagues analyzed 253 mini-CEX encounters and found an average duration of 17 minutes . 4lthough it is important that an evaluation instrument should allow for narrative feedback, it is difficult to get teachers to provide written feedback.We therefore need strategies to facilitate written feedback through either technological support (i.e.voice recording or speech recognition, or electronic formats that aggregate scores and feedback across multiple assessments) or procedural measures (the learner writes down the verbal feedback).From the teachers' views in the present study we can derive several practical recommendations to be considered in introducing the mini-CEX into a training program: (1)  analyze with the teachers involved each of the competencies to be assessed, (2) identify what is important to ob-serve, (3) agree on minimum requirements for residents based on their levels of expertise/experience, (4) make sure that assessment forms are available in all the different locations where the mini-CEX encounter could take place (coronary care unit, emergency room) or, alternatively, design pocket-sized forms, (5) schedule the session with the resident (alternatively, residents can request the teacher to observe them), ( 6) observe residents' performance, (7)  complete the form and (8) deliver feedback immediately after observation (Table 1).

Limitations of the study
The external validity of the study is limited because we investigated the teachers' reported experiences with the mini-CEX in a single residency program.Another limitation is that this particular mini-CEX required residents to perform to a high standard, with only one fail allowed per mini-CEX, which implied re-assessment by the program director.The teachers' limited experience in using the format could be considered another limitation.

Conclusions
Teachers perceive the mini-CEX as a useful formative assessment tool that promotes direct observation and facilitates instant feedback delivery.Implementation of teacher training programs and development of specific guidelines and performance standards for each level of residency training could be helpful in setting clear pass/fail thresholds.Also, interpersonal and institutional relationships and preconceptions regarding a resident's performance profile are relevant contextual factors, which can impact pass/fail decisions.An individual mini-CEX should be used primarily as a formative assessment tool, while the combined judgments on all mini-CEX assessments, together with other assessment and performance information, could be used for summative decision making.In this way, each individual mini-CEX assesses clinical performance in this specific situation and the learner is not judged on his/her general qualities as a good doctor.Just as students and examinees adhere to understandable behavioral patterns, teachers display certain patterns of human behavior.Assessors often bring specific values to assessment, based on personal experiences, beliefs, and conceptions (or misconceptions).The central lesson is that, with instruments like the mini-CEX, assessors are not "passive 1. Analyze with the teachers involved each of the competencies that are to be assessed 2. Identify what is important for the teacher to observe 3. Agree on minimum requirements for each resident according to his/her level of expertise/experience 4. Distribute the assessment forms throughout the settings where residents are observed or provide pocket-size forms 5. Schedule the session with the resident.The resident could be allowed to request which teacher observes him or her 6.Observe performance 7. Complete the form 8. Deliver immediate feedback measurement instruments", but active judges who, implicitly or explicitly, bring interpretations and values to the assessment which can have considerable impact. 7The theoretical implication is that we need to further clarify the following: What do assessors bring to the assessment and why?How can they be supported?What contextual influences affect them in making judgments?Before we have more answers here, the practical implication is that we need to be cautious with high stakes summative mini-CEX implementations.In the meantime we should encourage its formative value and use.

Table 1 .
Recommendations for enhancing feasibility