A pilot study using Cultural Consensus Analysis to measure Systems-Based Practice performance

Objectives: This pilot study was designed to compare the performance of ‘relationship-based’ statements from a cultural consensus analysis, a standard anthropological technique for measuring value differences, with ‘gold standard’ patient and nursing satisfaction surveys often used in 360° evaluation of the systems-based practice competency. Methods: We performed a cross-sectional correlation study in a teaching clinic in the United States. A single research assistant approached a convenience sample of ten residents, ten clinic staff, and ten patients per resident (120 participants). The cumulative scores for each resident on patient and nursing satisfaction surveys were compared to the average difference in cultural consensus analysis ranking between patient or nurse and the resident for two statements using Spearman rank correlation coefficients. These statements were selected because they represent the ‘relationship-based care’ pole in a previously validated conceptual model of clinic. Results: The correlation between patient satisfaction cumulative scores and the difference in patient and resident cultural consensus analysis rankings on ‘goals’ was -0.527 (less difference between residents’ and patients’ value ranking correlates with higher satisfaction). The correlation with ‘changes’ was -0.351. The correlation between nursing satisfaction cumulative scores and the difference in nursing staff and resident cultural consensus analysis rankings on ‘goals’ was -0.086. The correlation with ‘changes’ was -0.415. Conclusions: Systems-based practice is a notoriously difficult competency to evaluate. These moderate correlations in the expected direction between commonly used 360° evaluation instruments and this cultural consensus analysis tool suggest that it can provide an alternative measure of this competency.


Introduction
The Accreditation Council on Graduate Medical Education (ACGME) is standardizing resident training in the United States by requiring performance-based assessment of each resident in six competencies: patient care; medical knowledge; practice-based learning and improvement; communication; professionalism; and systems-based practice.Systems-Based Practice (SBP) requires that "Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value". 1 Systems-based practice is a difficult competency for training programs to evaluate using the suggested 'gold standard' of 360° feedback. 2This feedback is costly and time consuming, requiring valid information from a statistical sample of all stakeholders (patients, nurses, staff, colleagues, faculty, and self-evaluation).In addition, there are potential conceptual problems with 360° feedback.Data collection instruments are typically sponsored by the executive leadership and reflect their particular goals and values.These instruments may simply not contain questions about important goals or values of other stakeholders, such as patients.Data collection instruments for 360° feedback are usually composed of Likert-style questions aimed at evaluating one element at a time.By evaluating each element in isolation, these instruments may not reflect competing value decisions faced by trainees in clinic.
What is needed is a more 'real world' method of assessing the systems-based practice competency that includes elements of major concern to all stakeholders, accounts for possible value conflict, and is relatively easy to administer.In this study, we performed an analysis of Cultural Consensus Analysis (CCA) as a method that meets these requirements.CCA is a standard anthropological technique that determines whether groups hold shared knowledge, and whether there are conflicting preferences and values between groups.CCA is not subject to the conceptual problems mentioned above.During the development of a CCA, the viewpoints of all stakeholders are included and CCA is a 'forced-choice' technique that requires participants to balance values given existing constraints.Only one form of the CCA is used for all stakeholders, and it is fairly easy to administer.CCA has been used to demonstrate to health systems the diverse patient beliefs that drive behavior in diabetes, 3 cervical cancer screening, 4 and breast cancer screening; 5 and to design culture-specific interventions that improve screening behavior. 6ur team has designed and validated a CCA tool specifically for use in resident teaching clinics.First, we performed extensive ethnographic observations to identify the beliefs and values of nurses, learners, faculty members, and patients. 7From these data, we developed sixteen CCA statements that included the key values of all groups (appendix). 8We then tested this CCA tool at five teaching clinics and found that large value differences between groups on the CCA predicted operational problems as identified by blinded inter-disciplinary focus groups. 9Data from these studies have also been used to verify a coherent and complete model of clinic operations that includes all stakeholders perspectives (Figure 1). 10 Our pilot study was designed to further validate this CCA tool as a 360° measure of resident performance.In particular, we wanted to examine the association between the two CCA statements representing the 'relationshipbased care' pole of the conceptual model mentioned above and standard measures of patient and staff satisfaction with medical housestaff commonly used for 360° performance evaluation. 11,12

Methods
We studied the relationship between the two CCA statements representing the 'relationship-based pole' in our conceptual model 10 and standard patient and staff satisfac-tion with medical housestaff measures using a crosssectional correlation study.The Human Subjects Division from the University of Washington (Seattle, Washington, USA) performed an ethical review and approved this project.It was also reviewed by the Research Committee (IRB) of the Boise VA Medical Center where the study was performed.

Figure 1. Validated conceptual model of the clinic visit
A single research assistant approached a convenience sample (available in clinic that day) of ten residents, ten patients (per resident), and ten clinic staff (total 120 participants).Sample sizes were calculated to assure 95% confidence of answering at least 90% of the questions per the group norm, assuming the prior performance of this instrument in each group. 13For instance, in our previous study the residents achieved an average cultural knowledge of 0.65 (range 0.57 -0.71). 8Calculated minimum sample size to achieve these parameters for this level of competence is 8. 13 Our CCA tool consists of a set of 16 laminated 3 × 5 cards with one statement per card about "things that could happen during a clinic visit" (appendix).We asked subjects to rank-order these cards by order of importance to them.The CCA exercise was carried out by a single trained research assistant (RA).Subjects were approached individually and asked to sort the cards in a private or semi-private area.Their responses were recorded on a standardized form, and later transferred to an Excel spread sheet for analysis.
At the same time that we administered the CCA, we had each of the patients fill out a trainee satisfaction questionnaire (the American Board of Internal Medicine-Patient Satisfaction Questionnaire, ABIM-PSQ).This form consists of 10 questions in a 5-point Likert style format.Nursing staff filled their own version of a trainee satisfaction form (the American Board of Internal Medicine-Nurse Evaluation of Medical Housestaff, ABIM-NEMH) at the same time that they performed the CCA.This form consists of 17 questions in a 5-point Likert style format.The cumulative score for each resident on the patient satisfaction instrument (ABIM-PSQ) was compared to the average difference between patient and resident CCA ranking of the two statements representing the relationshipbased pole: "Doctor and patient agree on goals" (goals) and "Doctor asks what is changing in patient's life (such as a move or major family change)" (changes) using Spearman rank correlation coefficients.The expectation was a negative correlation, meaning that when the patient and resident share several values (have a small difference in CCA ranking) the patient should be more satisfied with the resident.A similar correlation coefficient was calculated between the cumulative score for each resident on the nursing satisfaction instrument (ABIM-NEMH) and the average difference between staff and resident CCA ranking on the same two statements.

Results
All of the residents and nursing staff agreed to participate.Three per cent of the patients refused to participate in the study.The correlation between patient satisfaction cumulative scores and the difference in patient and resident rankings on the two relationship-based pole CCA statements was negative as expected.For the 'goals' statement, the correlation was -0.527 (less difference between residents and patients in CCA ranking correlates with higher satisfaction score).The difference on 'changes' was -0.351 (Figure 2).The correlation between the nursing satisfaction cumulative scores and the difference in nursing staff and resident CCA rankings on 'goals' was -0.086.The difference on 'changes' was -0.415 (Figure 3).There was no important association between the CCA statements representing the other poles of the conceptual model and the satisfaction surveys.

Discussion
SBP is a notoriously difficult competency to evaluate.These moderate correlations between commonly used 360° evaluation instruments (ABIM-PSQ and ABIM-NEMH) and the relationship-based pole of this CCA instrument suggest it can provide an alternative measure of SBP competency.CCA also has conceptual advantages over these other instruments: it is derived from elements that are of concern to all stakeholders in clinic, it forces prioritization of values and it can be presented in a simple graphical format, which makes it an ideal instrument for motivating dialogue about values in teaching clinic and their negotiation.This study has some limitations.It was conducted during a single brief time period and at a single teaching clinic, so the findings may not be generalizable.The assumption that the two CCA statements 'goals' and 'changes' reflect interpersonal relationships in the same way as the ABIM-PSQ and ABIM NEMH may be erroneous, especially for residents and nursing staff, since they are written from the doctor-patient perspective. 10t is often difficult to make meaningful improvements in teaching clinic.This has to do with the complexity of clinic systems, resistance because many top down interventions have a perceived lack of relevance to people on the 'front lines', and the difficulty in having open, honest conversations.An ecological perspective of teaching clinic has been proposed that focuses on stabilizing features, dialogue, and pragmatic observable behaviors. 14Graphical presentation of CCA data could be an important component of an ecological approach, acting as both a stabilizing feature and facilitator of meaningful dialogue about clinic values and function, and this could improve systems-based practice performance.We believe that a CCA tool should now be studied prospectively as an outcome measure for the systems-based practice competency.
Algorithmic Care (Valid algorithms that can guide the provision or measurement of care) Structured Care (Knowledge of patient, familiar clinic routines) Educational Care (Graduated supervision and responsibility of trainees) Relationship-based Care (Concern for patient as person and their medical choices, treat patients as individuals)