Measuring approaches to learning in a problem based learning context

Objectives: Students in Problem-Based Learning (PBL) are assumed to adopt a deep learning approach and not a surface approach. This study investigated: 1) the reliability and validity of version of the Revised Study Process Questionnaire adapted to the PBL context (PBL-R-SPQ) and 2) the extent to which PBL students use deep or surface approaches, and whether this differs between first and second year students. Methods: The items of the R-SPQ were reformulated to better fit with a PBL environment, resulting in the PBL-RSPQ. In total 262 students from Maastricht Medical School responded to the PBL-R-SPQ. Results: A Confirmatory Factor Analysis (CFA) demonstrated that a 9-item Deep Approach scale and a 9-item Surface Approach scale fitted the observed set of data well. Cronbach alphas for the Deep and Surface scales were 0.76 and 0.74, respectively. First year students reported signifi cantly higher Deep Approach scores (M = 3.60, SD = .48) than second year students (M = 3.40, SD = .48) (p = .001, d = .42). Conversely, second year students reported significantly higher Surface Approach scores, (M = 2.45, SD = .48) than first year students (M = 2.26, SD = .52) (p = .003, d = .38). Conclusions: The 18-item PBL-R-SPQ provides a valid and reliable tool to measure students’ learning approach in PBL. In addition, PBL students tended to adopt a deep approach rather than a surface approach, which is in line with the assumptions behind PBL, although the second year students have a somewhat less deep approach than the first year students.


Introduction
In the mid-seventies, different approaches to student learning were identified: a deep approach and a surface approach.Based on intrinsic interest in the topic, students taking a deep approach try to understand ideas and seek meaning and understanding. 1 Often driven by a fear of failure, students adopting a surface approach are focused on meeting external requirements, such as assessments, particularly through rote learning. 21][12] Good constructive alignment between teaching and learning activities, assessments, and desired learning outcomes is therefore required if desirable approaches are to be promoted and undesired approaches minimised. 12ne form of curriculum often promoted by Student Learning theorists is Problem Based Learning (PBL).Although PBL looks different at various schools, three characteristics can be considered as essential: 1) problems as a stimulus for learning, 2) teachers as facilitators of the learning process and 3) group work as stimulus for interaction. 13Within PBL, students work in small groups to discuss problems under the guidance of a teacher Correspondence: Diana H.J.M. Dolmans, Department of Educational Development and Research, Maastricht University, The Netherlands, Email: D.Dolmans@educ.unimaas.nl5] The interaction in the small group around the problems is assumed to stimulate students to adopt a deep learning approach.One of the earliest studies in which students' approaches to learning in PBL were investigated was conducted by Newble and Clarke. 16n their study they demonstrated that students who have been taught PBL become increasingly deep and less surface in their orientations.Several studies appeared in the literature investigating students´ approaches to learning in PBL.][19][20] Others, like Nijhuis et al., reported that the students from the problem-based environment showed a high degree of surface learning and a low level of deep learning. 21Furthermore, some PBL researchers have noted a shift over the course of time in the quality of students' approaches, with students tending to adopt a more surface approach and a less deep approach over the course of the first year of their study. 17,19,20 I other words, the results so far are not unambiguous that PBL does indeed stimulate students to adopt a deep approach to learning.
Further research is needed to obtain better insight as to whether PBL curricula do indeed stimulate students towards deep learning instead of surface learning, and whether this changes over the years.In this study we investigated tendencies of PBL students to adopt a deep or surface approach to learning, and whether there are differences between year 1 and year 2 students.We used the Revised two-factor Study Process Questionnaire (R-SPQ) of Biggs et al. 2 , a 20-item questionnaire measuring a deep and a surface approach to learning.The 20-item R-SPQ was adapted to the PBL-environment, since some items within the R-SPQ focus on lecturing and classes, and therefore might not be fully appropriate to measure students' approaches to learning in a constructivist learning environment, as was also argued by Gijbels, et.al. 22Although the R-SPQ has been tested for its validity and reliability we tested the validity and reliability of this to a PBL environment adapted version (PBL-R-SPQ).In addition, we measured whether students' deep and surface approaches to learning differed between year 1 and 2 students within a PBL-curriculum.Thus, the aim of this study was to investigate whether: 1) an adapted version of the R-SPQ, entitled PBL-R-SPQ yields valid and reliable data about a student's deep and surface approach to learning in a PBL curriculum, and 2) PBL students' relative use of deep and surface approaches to learning, and whether this differs between first and second year students.

Context
The study was conducted in the PBL curriculum of the Maastricht Medical School in the Netherlands.Within this problem-based curriculum, years 1 and 2 are divided into several units in which a multidisciplinary theme is covered.About 340 students enter this medical school per year.All units use PBL as the instructional format and are organized around PBL cases.Students meet twice a week to discuss these PBL cases during a two-hour session.
A teacher guides the tutorial group session.Each unit takes six weeks.At the start of each unit, students within the tutorial groups are rearranged and a new teacher is assigned to each group.All teachers involved in this curriculum had a two-day PBL training and a two-day teacher or tutor training before they started as PBL tutors.During each unit, the teacher works together with one group of about 9 or 10 students.
First and second year students were asked to fill out the questionnaire at the end of their first unit.Participants 160 first year students and 133 second year students were invited to complete the PBL-R-SPQ.In total 154 first year students and 108 second year students (in total 262 students) filled out the questionnaire, giving a response rate of 96% in year 1 and 81% in year 2, and a total response rate of 89%.The average age of students when entering the programme is about 18 years, and the majority (about 60%) of students were female.

Instruments
The revised two-factor Study Process Questionnaire 2 is a 20-item self-report instrument.This instrument measures two latent variables: a deep approach to learning (10 items) and a surface approach to learning (10 items).The deep approach latent variable was indicated by two observed variables (Deep Motive and Deep Strategy sub-scales), while the surface approach latent variable was indicated by another two observed variables (Surface Motive and Surface Strategy sub-scales).Nine items were reformulated and adapted to a PBL environment, since these items of the R-SPQ focus on lecturing and classes instead of blocks, tutors and groups.The items were rated on a 1-5 scale (1 = never or rarely, 2 = sometimes, 3 = half of the time, 4 = frequently and 5= always or almost always).The PBL-R-SPQ refers to the students' preferred approach to learning and the actual or ongoing approach to learning in a PBL curriculum.The PBL-R-SPQ measures whether a student's motives and strategies are predominantly deep or surface.

Data Analysis
Prior to analysis, the dataset was inspected to determine the extent of missing data.Less than 1% of cells had missing data, and these data points were imputed using the Expectation Maximization Algorithm.This algorithm in LISREL (a statistical program for structural equation modelling) uses an iterative procedure to create a dataset with no missing values whose covariance matrix is as close as possible to the original dataset with missing values.In order to test the construct validity of the PBL-R-SPQ a confirmatory factor analysis was conducted in which it is tested whether the hypothesized two factor model fits with the observed data, because the original SPQ has a clearly hypothesized two-factor structure.4] Mean scores per scale were calculated to examine the nature of students' approaches in this cohort.T-tests were conducted to investigate whether first and second year students' scale scores differed significantly (p < .01).

Ethics
Although ethical approval was officially not required at the institute in which the study was conducted, the students were given information about the study before they filled out the questionnaire and the anonymity and confidentiality of the students' responses on the questionnaire were safeguarded.

Confirmatory factor analysis and reliability analysis of the PBL-R-SPQ
Because the SPQ has a clearly hypothesised factor structure, analyses reported below follow the sequence of analyses used by Biggs et al. 2,25 This sequence began with tests of congeneric measurement models of the 4 sub-scales (Deep Motive, Deep Strategy, Surface Motive, and Surface Strategy).These analyses were followed by tests of fit of the overall model, in which sub-scales loaded on two higher order factors (Deep and Surface Approach) which were allowed to correlate.LISREL 8.53 was used to conduct these confirmatory factor analyses.Prior to these analyses, we calculated univariate and multivariate normality statistics using PRELIS 2.54.The data exhibited significant multivariate non-normality (multivariate skewness z = 11.21,p<0.001, multivariate kurtosis z = 7.44, p<0.001).To take account of this potential violation of a key assumption of structural equation modelling that the data are normally distributed, all analyses used robust maximum likelihood estimation, a method less sensitive to violations of the normality assumption with large models than other estimation methods. 26,27 t indices reported for the various models include the Comparative Fit Index (CFI) and the standardized root mean square residual (SRMR).The CFI compares the proposed model to the independence model, in which all variables are uncorrelated; values greater than 0.90 and 0.95 indicate acceptable and excellent fit respectively.The SRMR is the mean difference, based on standardized residuals, between the predicted and observed variances and covariances in the model; values below 0.05 indicate good fit.28 Finally, having tested the dimensionality of each of the sub-scales and scales, internal consistency estimates of reliability (Cronbach's alpha) were calculated using SPSS.
A congeneric measurement model of the five Deep Motive items had a good fit to the data, CFI = 0.97, SRMR = 0.038, but inspection of the completely standardised factor loadings revealed item 17 ("I come to most tutorial group sessions with questions in mind that I want answering") had a very low factor loading (0.11) on the Deep Motive latent construct, suggesting this item was not a reliable indicator of the construct.A congeneric measurement model without this item had a better fit to the data, CFI = 0.98, SRMR = 0.029, and was accepted; Cronbach's alpha = 0.59.A congeneric measurement model of the five Deep Strategy items had a good fit to the data, CFI = 1.00,SRMR = 0.013; Cronbach's alpha = 0.67.A congeneric measurement model of the five Surface Motive items had a moderately acceptable fit to the data, CFI = 0.91, SRMR = 0.049; Cronbach's alpha = 0.59.A congeneric measurement model of the five Surface Strategy items had a marginally acceptable fit to the data, CFI = 0.91, SRMR = 0.053, with inspection of the completely standardised factor loadings revealing item 8 ("I learn some things by rote until I know them by heart even if I do not understand them") had a very low factor loading (0.11).A congeneric measurement model without this item had a better fit to the data, CFI = 0.99, SRMR = 0.028, and was accepted; Cronbach's alpha = 0.54.
Based on the results of the above measurement models, sub-scale scores were calculated based on the average response across the 18 items, since item 8 and 17 were removed.Confirmatory factor analysis was used to test the full model, with Deep Motive and Deep Strategy scores loading on a Deep Approach latent variable, and Surface Motive and Surface Strategy scores loading on a Surface Approach latent variable.The hypothesized full model, given in Figure 1, explained the covariances among the four sub-scales well, CFI = 0.97, SRMR = 0.032.The correlation between the Deep and Surface Approach latent constructs was estimated as -0.60.For the total 9-item Deep Approach scale, Cronbach's alpha = 0.76, and for the total 9-item Surface Approach scale, Cronbach's alpha = 0.74.The factor loading of the items of the PBL-R-SPQ are given in Table 1.The average scores per item are given in Table 2.

Approaches to learning in PBL and differences between cohorts
As noted above, some PBL researchers have noted a shift over the course of time in the quality of students' approaches, with students tending to adopt a more surface approach and a less deep approach over the course of the first year of their study. 17,20 e explored whether this pattern of results was present in the Maastricht Medical School PBL programme by comparing the Deep and Surface Approach scores of first and second year students.First year students reported higher Deep Approach scores, M = 3.60, SD = .48,than second year students, M = 3.40, SD = .48,and this difference was statistically significant, t(260) = 3.41, p = .001,standardised mean difference (Cohen's d) = .42.Conversely, second year students reported higher Surface Approach scores, M = 2.45, SD = .48,than first year students, M = 2.26, SD = and this difference was also statistically significant, t(251) = 2.98, p = .003,Cohen's d = .38.

Discussion
The first aim of this study was to test the validity and reliability of the PBL-R-SPQ.The results of the confirmatory factor analysis demonstrated that a 9-item deep approach scale and a 9-item surface approach scale fitted the observed data-set well.The Cronbach alphas of the PBL-R-SPQ scales were 0.76 for the deep approach and 0.74 for the surface approach, representing acceptable levels of reliability.In general it can be concluded that the 18-item PBL-R-SPQ provides a valid and reliable tool to measure students' learning approach in a PBL curriculum.Since it only contains 18 items, the instrument can be easily applied to measure PBL students' approaches to learning.
The second aim of this study was to investigate students' relative use of deep and surface approaches to learning in PBL and whether this differs between year 1 and 2 students.The results demonstrated that both the PBL first-year and second-year students tend to adopt a deep approach (M = 3.60 and M = 3.40, for year 1 and 2 respectively) rather than a surface approach (M = 2.26 and M = 2.45, for year 1 and 2 respectively).8][19][20] The results furthermore demonstrated significant differences in scores between first and second year students on the deep and surface approach.Although the scores differed significantly from a statistical point of view, the differences were small from a practical point of view (0.2 points on a scale 1-5).] In conclusion, the PBL students seem to use a deep approach rather than a surface approach although the second year students tend to have a somewhat less deep approach and more surface approach than first year students.Perceptions of inappropriate assessment may move students towards a surface approach. 5,29 tudents might have the perception that they need to study less in depth to obtain a sufficient mark on the test in year 2.
This study has some limitations.First of all, the data reported are self-reports.Since the students were aware of their involvement in a PBL curriculum, it might be possible that they have particular conceptions about learning which might have influenced the results of this study. 30Second, the differences between first and second year students should be viewed with some caution given that they are cross-sectional rather than longitudinal in nature: thus, the pattern of results may represent idiosyncrasies related to the particular cohorts.
Further research is needed.First, the PBL-R-SPQ needs to be applied in more PBL settings, to ensure its validity and reliability in more than one setting.Second, longitudinal studies are needed to investigate the development of learning approaches over academic years.Third, it would be interesting to measure not only students' approaches to learning in PBL, but also students' perceptions about the assessment in PBL contexts and whether this influences their learning approach. 31

Table 1 .
Confirmatory factor analysis structure for PBL-R-SPQ -completely standardised factor loadings from congeneric measurement models for the total sample 12. I generally restrict my self-study to what is specifically set as I think it is unnecessary to do anything extra 0.64 15.I find it not very meaningful to study topics in depth; it confuses and wastes times, when all you need is superficial knowledge of the topics 0.49 16.I believe that tutors shouldn't expect students to spend much time on topics that will not be examined 0.46 19.I see no importance in studying topics which are not likely to be

Table 2 .
Mean and standard deviation per item for all students (N = 262)