Comparison of medical students ’ diagnostic reasoning skills in a traditional and a problem based learning curriculum

Objectives: This cross sectional study aimed to investigate the relative effectiveness of a problem based learning curriculum (PBL) and a traditional curriculum (TC) in the development of medical students’ diagnostic reasoning skills. Methods: Junior and senior clinical students (n = 431) at a single clinical school of the University of Melbourne selfadministered Bordage et al.’s Diagnostic Thinking Inventory (DTI) to assess diagnostic reasoning skills during transition from a TC to a PBL curriculum. Mean scores for the flexibility in thinking and memory structure sub-scales were compared for students at junior and senior clinical levels under the two curricula using multivariate analysis of variance (MANOVA). Results: An effect of curriculum, Wilks’ Lambda = .95, F (2, 426) = 11.67, p < .001, ηp =.05 and level of trainee, Wilks’ Lambda = .86, F (2, 426) = 35.00, p <.001, ηp = .14 was evident on DTI scores overall. Senior students on average had higher flexibility in thinking (86.7 vs. 80.9) and memory structure scores (84.8 vs. 77.9) than junior students. Students in the TC had higher average flexibility in thinking (84.4 vs. 83.0) and memory structure scores (82.6 vs. 79.9) than students in the PBL curriculum. Conclusions: These results suggest both a developmental effect and a curriculum effect in the development of diagnostic reasoning skills. Contrary to expectations, a TC appears to favour the development of diagnostic reasoning skills compared with a PBL curriculum. This unexpected finding highlights to educators the importance of monitoring student attributes during curriculum change.


Introduction
The development of effective diagnostic reasoning skills is a critical aim of all medical training.7][8][9] In recent times, many medical schools have changed curricula from a traditional course (TC) to a problem based learning curriculum (PBL).Early proponents of PBL claimed that clinical reasoning or problemsolving skills were the most important set of abilities a doctor should possess but that these skills were not sufficiently emphasised in traditional curriculum approaches to medical education. 10PBL was seen as a way of enabling students to develop key reasoning and critical thinking skills more efficiently than traditional methods of medical education. 11The development of clinical reasoning skills characteristic of the expert clinician was cited as one of the main educational objectives of PBL. 12 Nonetheless, there is little empirical evidence of the superiority of diagnostic reasoning for students trained in PBL versus TC. 13 In 1999 the University of Melbourne implemented a new problem based learning course to replace the existing Correspondence: Barbara Goss, Austin Health and Northern Health Clinical School, The University of Melbourne, Australia.Email: b.goss@unimelb.edu.autraditional course.This transition between curricula types presented the opportunity to investigate the effect of different teaching styles on the acquisition of diagnostic reasoning skills.With curriculum change comes the need to ensure that student outcome measures are maintained or improved, specifically those skills deemed critical for medical practice.In the current study therefore, medical students' diagnostic reasoning skills were assessed by means of the Diagnostic Thinking Inventory (DTI) developed by Bordage, Grant and Marsden 14 to see how the skills of students in the PBL compared with those in the TC.Testing also afforded the opportunity to see how other student characteristics (e.g.gender) might relate to diagnostic reasoning ability and how students' clinical reasoning varied at different stages of the medical course.
Thus the purpose of our study was to examine the diagnostic reasoning skills of medical students at the beginning and the end of their clinical training to determine if a PBL curriculum is more effective in supporting the acquisition of these skills when compared to a TC.

Overview of Curricula
The two medical courses (TC and PBL) differed in content and to some degree in student selection processes.The TC students entered the course after the final year of school and were selected by a combination of final secondary school rank (Equivalent National Tertiary Entrance Rank or ENTER) and the Undergraduate Medical Admission Test (UMAT) score.Most PBL students also entered the course directly from school using ENTER and UMAT, however approximately 25% of students had completed a prior undergraduate program and were selected by their grade point average (GPA), scores in the Graduate Australian Medical School Admissions Test (GAMSAT) and a structured interview.
TC students completed a six-year course.The first three years were university-based and focused on the enabling bioscience disciplines with little integration.Years 4-6 of the course were in a fulltime clinical setting at one of four clinical schools.Students were taught by means of a comprehensive lecture programme and clinical instruction.This was broadly structured covering semesters in general medicine and general surgery and a range of sub specialties.
The PBL was six years (12 semesters) in duration for students entering the program from secondary school and 4.5 years for students with prior tertiary qualifications.In Semesters 1-5 all students were based at the university and completed a PBL-based programme with discussion of 75 cases.The problem of the week was underpinned by an integrated biomedical science curriculum.Students also participated weekly in a clinical exposure programme which included fortnightly visits to hospitals for bed-side teaching.Semesters 6 and 7 comprised an intercalated research year for the students who entered the program from school and then all students proceeded to a clinical school for 2.5 years of fulltime clinical training.In Semesters 8 and 9, weekly PBL tutorials continued and clinical training occurred in six 6-week blocks of medical and surgical specialties (e.g.Nephrology, Urology, Endocrinology and Vascular Surgery).Semesters 10, 11 and 12 covered sub-specialties and a pre-internship.

Participants
University of Melbourne medical students undertaking their clinical training at one of the university's four clinical schools participated in the study.Between 2000 and 2005 (the period of transition from TC to PBL) cohorts of students were recruited at two equivalent stages of fulltime clinical training: the end of their first 12 months -the completion of fourth year for TC and Semester 9 for PBL (junior clinical); and the end of their final year of the course -sixth year for TC and Semester 12 for PBL (senior clinical.)Participants comprised two cohorts of junior clinical and one senior clinical cohort of TC students and two cohorts of both junior and senior clinical PBL students.Table 1 shows the number of participants in each year of the study and the percentage of the total available students who participated.Response rates varied across the years of the study from a minimum of 52.4% to a maximum of 89.7%.

Demographics of participants
Of the 431 students involved in the study, 58.2% were female and 41.8% were male.55.9% were born in Asia, 38.7% were born in Australia and 5.4% were born in other locations.43.2% had been resident in Australia for more than 18 years, 29.2% for between 6 and 17 years and 27.6% for less than 6 years.Most (78.4%) spoke English as their usual language at home.The characteristics of students undertaking the TC and PBL were similar.Table 2 shows the demographic characteristics of participants.

Materials
6][17][18] The DTI consists of 41 items.Each item comprised a stem, followed by a semantic-differential response rated on a 6-point scale.The inventory assessed two different aspects of diagnostic reasoning, (a) flexibility in thinking and (b) evidence for structure in memory.Bordage et al. defined flexibility in thinking as "the use of a variety of thinking means or processes that can be applied during the diagnostic process (p.415)" and memory structure as "availability of knowledge, stored in memory, during the diagnostic process (p.416)." 14Responses to the 41 items of the DTI are scored and then totalled.Total DTI scores can range from a minimum of 41 to a maximum of 246, (flexibility in thinking 21 to 126 and structure in memory 20 to 120).

Procedure
In this cross sectional study, participants were invited to complete the DTI as part of routine evaluations conducted in a whole class lecture theatre setting at the end of clinical programs.Ethics approval was not formally sought as the data was collected as part of routine quality assurance and the measure assessed an aspect of skill acquisition similar to other assessments of students' skills.Student participation was voluntary and student responses were anonymous.

Data Analysis
The purpose of the analysis was to determine if there was variation in diagnostic reasoning ability between the different groups of students assessed in the study.A total DTI score was determined for each student as well as flexibility in thinking and memory structure sub-scales.A multivariate analysis of variance (MANOVA) was conducted with clinical level (junior or senior) and curriculum type (TC or PBL) as independent variables and the scores on the flexibility in thinking and memory structure scales of the DTI as dependent variables.A second MANOVA explored variation in scores on the diagnostic reasoning sub-scales as a function of student gender and the length of time students had been resident in Australia (18 or more years, 6-17 years, less than 6 years).All analyses were conducted using PASW Statistics for Windows Release 18.0 (SPSS Inc., Chicago, IL, USA).

Internal consistency of scales
It should be noted that three items with very low corrected item-total correlations were identified.Item 11 (Clarification before further data acquisition), Item 14 (Diagnostic error: lack of knowledge or inability to retrieve existing knowledge) and item 25 (Specific enquiry) all had close to zero corrected item-total correlations (r = .09,r = .02and r = -.01 respectively).These items were retained in the analyses for comparability with Bordage et al.'s original scale and because dropping these items improved the scale's internal consistency only marginally.However, the results suggest that these items might have limited use as discriminating items in the inventory.In particular, Bordage et al. identified Item 25 as a potentially ambiguous item, though the authors decided to retain the item in the final scale. 14he mean total DTI score for all junior clinical students including students from both curricula was 158.8 (SD = 18.5) and the mean total score for all senior clinical students was 171.5 (SD = 18.8.)The mean total score recorded for junior clinical students in this study is comparable to that reported for third-year medical students by Bordage et al. (M = 158.3)and the mean total score for senior clinical students was similar to that of Senior House Officers (M = 170.1). 14ata from students in cohorts from the PBL and TC at the same level were pooled to optimise statistical strength.The average scores were very similar for same level cohorts, (junior clinical TC: flexibility in thinking 82.

Analysis of differences in diagnostic reasoning ability by gender and years of residence in Australia
Table 4 shows the mean scores for the flexibility in thinking and memory structure scales as a function of gender and years of residence in Australia.A MANOVA was conducted with gender and years in Australia (18 or more, 6-17, less than 6) as independent variables and scores on the flexibility in thinking and memory structure scales as dependent variables.In order to constrain the number of independent variables in the analysis, country of birth was not included because it was strongly related to years of residence in Australia.That is, Australian born students had almost always been resident in Australia more than 18 years (95.2%) and students born in Asia had almost always been resident less than 18 years (91.1%).Similarly, speaking English at home was not included because most students resident in Australia more than 18 years (97.8%)spoke English at home.

Discussion
The results of this study demonstrated that students completing a TC and a PBL both have a significant improvement in diagnostic reasoning skill, as measured by the DTI, from early in clinical training to the end of the course.This is consistent with the findings of other investigators who have measured the diagnostic reasoning skills of students and doctors using the DTI. 14,15,18,20,21 Thesecond finding of our study was that students completing the TC had higher DTI scores than those completing the PBL.This was apparent both at the end of the first year of clinical training and at the end of the course.It was anticipated that the new PBL course would give students a stronger grounding in diagnostic reasoning which would better prepare them for the clinical setting.It was therefore surprising that in our study TC students displayed superior reasoning skills to students from the PBL course.
There are a number of differences in the curricula that may explain the pattern of findings in the current study.Both the TC and PBL were six years in duration for students entering the program from secondary school however the PBL included a year of research (Semesters 6 and 7) prior to the first fulltime clinical year (Semesters 8 and 9).This followed five semesters which included weekly discussion of clinical problems and patient-centred teaching in hospitals.Neither a research year nor early clinical learning were components of the TC course.It is possible that the yearlong break in the PBL significantly interfered with recall of previous clinical experience and learning from clinical cases thereby compromising the development of diagnostic reasoning skills.
Another difference between the TC and PBL was the structure of clinical experience in the fulltime clinical environment.In the TC, the students' terms were broadbased with exposure to a wide range of clinical problems while in the PBL most clinical training was in systems-based blocks with focus on a narrow range of clinical material in each block.It may be that this different experience of clinical learning affected the development of diagnostic reasoning skills and contributed to the differences found in DTI scores.In a review paper of diagnostic reasoning, Eva noted that mixed practice where students saw cases of multiple categories mixed together was pedagogically superior to 'blocked' practice. 4Sefton, commenting on students' acquisition of clinical reasoning skills, noted that students in PBL curricula may be inhibited from freeranging exploration of issues because of problems being collated into blocks of material.This criticism relates mostly to the early years with a less structured environment encouraging broader thinking. 7he time spent in clinical training was also different in the two curricula.In the TC clinical training was three years and in the PBL two and a half years.A number of researchers have concluded that clinical experience is an important aspect in the development of diagnostic reasoning skills to expert level with skills improving in relation to the number of clinical encounters. 1,2,4,22,23 Thu the change in the course to a reduction in fulltime clinical experience from three to two and a half years may be a significant factor in the difference found between senior clinical groups in this study, however, this does not explain the differences identified at the junior level.
The difference in diagnostic reasoning scores at the junior clinical level was most marked in the memory structure component of the DTI.In a study by Patel et al examining the link between basic science and diagnostic reasoning, the authors concluded that students' diagnostic reasoning may benefit from a strong biomedical knowledge foundation which is not necessarily embedded in PBL. 6Indeed, Patel et al postulated that confusion may arise when students are taught clinical and biomedical sciences together (as in PBL), which may result in the formation of two incomplete and ill-formed knowledge domains, whereas the separation of science and clinical instruction, as in TC, may help longterm in diagnostic reasoning and understanding. 8ew other investigators have assessed diagnostic reasoning skills of students in different curricula.Sobral, in a study from Brazil, used the DTI to compare students' ability from traditional and integrated PBL pre-clinical streams after 22 weeks of clinical training. 17There was no significant difference in total DTI scores, but in contrast to the current study, the PBL students had significantly higher scores than the TC students for the memory structure component.Groves, in a study in Queensland, found no difference in DTI scores between students from a traditional course compared with PBL but this study included only a small cohort of 24 final year TC students. 24Groves speculated that the failure to detect superior reasoning skills in students in a four-year PBL course compared with a six-year conventional course might be due to the reduction in course length.Using a different approach, Patel compared the effects of TC and PBL on reasoning strategies by assessing students' responses to clinical problems. 8This study found that a PBL did not confer better diagnostic reasoning skills in students and in keeping with our findings TC students were noted to use reasoning strategies more characteristic of experts than students from a PBL curriculum.
We also found that gender had a significant influence on some aspects of student reasoning performance.Average scores for the memory structure component of the DTI were significantly higher in female medical students compared with males.This finding was different from some investigators who found no gender difference in DTI scores. 17,20However, Groves reported significantly higher clinical reasoning scores in females when assessed on a different measure -Clinical Reasoning Problems (CRP). 15nother important finding of the study was that time spent in Australia appeared to affect diagnostic reasoning skills.Students who had spent less than six years in the country had significantly lower flexibility of thinking scores than other students at both a junior and senior clinical level.It may be that those students who do not have English as their first language have some difficulty with the language of the DTI or the skills of diagnostic reasoning in a foreign language.

Limitations of the study
There are several possible limitations to this study to consider.Only one of four clinical schools of the University of Melbourne was involved so the results are not necessarily generalisable as characteristics of both the teaching and students may differ.A single measure was used to assess clinical reasoning skills and this was a self-reported questionnaire.Whilst this tool has been widely used and our results were consistent with other studies showing increased DTI scores over time in clinical training -and consistent with Bordage's findings, in future consideration could be given to using a second measure of clinical reasoning and including an observed rating of skills.This is important because a broader assessment, which includes measures that do not only rely on self-report, is likely to provide a more accurate picture of diagnostic reasoning ability.There was only one opportunity for students to complete the questionnaire at an end of year evaluation session.Response rates varied but in most cohorts this was close to 70% or above.It is not possible to know whether the nonresponders differed in any way from the responders, however as reported, the DTI scores from the same level cohorts were very similar.

Conclusions
This study demonstrated a clear difference in diagnostic reasoning skills between junior and senior clinical students providing evidence that there is progression from novice to more expert skills during clinical training.Of interest was the finding that at both junior and senior levels students in a TC had superior diagnostic reasoning skills compared with students in a PBL.This is unexpected, given the emphasis in the PBL on enquiry-based learning and early clinical encounters.It is possible that this difference relates to both a reduction in overall time for clinical experience in the new course, as well as the break prior to commencement of fulltime clinical training resulting in loss of previously acquired skills.It is also possible that having a wellestablished scientific knowledge base and a programme of broad-based clinical instruction, as in the traditional course, enhances the development of clinical reasoning skills.
With medical schools throughout the world undertaking curricular change it is important to monitor the effects of these changes on medical graduates' skills.The University of Melbourne has recently introduced another curricu-lum change with a move to a post-graduate medicine model.All students entering the new course are required to have a strong foundation in biomedical science and the course includes a three-year clinical component following directly on from one pre-clinical year.It will be of considerable interest to investigate the clinical reasoning of students in the graduate course compared to that of students taught under the two previous curricula.

Table 1
The internal consistency of the scale overall (Cronbach's Alpha .86)and for the 21 items comprising the flexibility in thinking scale (Cronbach's Alpha .74)and the 20 items comprising the memory structure scale (Cronbach's Alpha .77)are comparable with those reported by Bordage et al. (overall .83;.72 and .74for the flexibility in thinking and memory structure scales respectively).

Table 3 .
Means, standard deviations and 95% confidence intervals (CI) for the diagnostic reasoning scales as a function of group (n = 431)*

Table 4 .
Means, standard deviations and 95% confidence intervals (CI) for the diagnostic reasoning scales as a function of gender and years of residence in Australia (n = 431)* * Comparisons of mean DTI scores were conducted using MANOVA.The numbers reported in Table4do not add to the total number of participants due to missing data.†F(2, 409) = 5.92, p = .003,ηp 2 = .03(7students did not report their years resident in Australia)‡ F (1, 409) = 5.46, p = .020,ηp 2 = .01(12 students did not report their gender)