Validation of a Self-report Tool to Measure Self-study in Medical School – Applying the Triads Method

Many students have difficulty mastering the demands of medical courses. Applying the Triads method, the present research compares the validity of three self-reported methods that measure self-study hours in a medical course. Thirty randomly selected students participated in this study. Three surveys were developed: continuous weekly records (WR) and two end-course retrospective self-reported surveys, AR (asking average hours/week during academic year) and FR (asking study frequency throughout academic year and mean hours of studying on a given day). Validity coefficients (VCs), which measure the correlation between observed and “true” self-study, were estimated using the Triads method. The estimated VC was 0.86, 0.94 and 0.87 for WR, FR and AR, respectively. Therefore, given the demands of WR and, since FR showed more accuracy than AR, we propose a novel method, FR, to retrospectively estimate self-study in a medical course.


INTRODUCTION
The relationship between time and learning is one of the most challenging issues related to the teaching-learning process (Karjalainen, Alha, & Jutila, 2006).It is the responsibility of teachers to ensure that tasks required in a course are appropriate for the times allocated to students to perform those tasks.In fact, the allocation of working time to study activities is itself an act of teaching.However, the curriculum scheduled is, in most cases, not the curriculum experienced by students (Wilkinson, Wells, & Bushnell, 2005).
Research consistently demonstrates the overload of medical education.An undergraduate course in medicine is considered difficult and demanding, with a high intensity of study throughout the course duration.This excessive work may adversely impact students' academic success but also their wellness and mental health (Cope & Staehr, 2005;Jacobs & Dodd, 2003;Kember, 2004;Woodley & Parlett, 1983).Indeed, in the last years, there has been a concern that medical students may be at an increased risk of psychological problems due to the stress caused by demands of medical course and concern for failing (Aherne et al., 2016).Therefore, it is essential to monitor and seek to maximise the amount of time that medical students invest in their studies.
In the context of European higher education, students' working time is measured as the number of contact hours spent in classes (i.e., attending lectures, seminars) plus the time spent on self-study (i.e., time outside the classroom dedicated to learning purposes, e.g.preparing projects or revising for examinations) (Pogacnik et al., 2004).Contact hours can be calculated from the planned activities of the course.However, self-study learning differs from student to student, and there is no effort to estimate the number of study hours.Instead, this is simply assumed in the program.An analysis of the students' self-study may lead to the reorganization of curricula or course units and to Barbosa et al. / Measuring Self-study in Medical School 2 / 10 a reduction of their scope.The greatest challenge in determining student workload is to obtain reliable data on the number of hours that students dedicate to self-study (Bowyer, 2012).

Self-report Methods of Self-study
Previous published studies have reported possible methods for estimating study time.The most common method is student reporting, at the end of the course or semester or on two/three times occasions during the course, indicating the average hours per week that they devoted to study (Dolmans, Wolfhagen, Essed, Scherpbier, & Van Der Vleuten, 2001;Guillaume & Khachikian, 2011;Pogacnik et al., 2004;Van den Hurk, Wolfhagen, Dolmans, & Van der Vleuten, 1998).A possible limitation of using this method is the risk of students' over-or underestimating study hours because it's difficult for students to retrospectively remember the time that they spent studying.Another readily available method is for students to keep records for short periods of time, usually up to 1 week (Kember, 2004;Kember, Ng, Tse, Wong, & Pomfret, 1996).Since this method tracks study time daily, it eliminates the previous risk of recall error in over-or under estimation.However, because study behaviour may change over the course, especially during the assessment period (Kerdijk, Cohen-Schotanus, Mulder, Muntinghe, & Tio, 2015;Mueller et al., 2011), the representativeness of this method for the entire term or academic year is doubtful.In this scenario, continuous weekly records (Mueller et al., 2011), the less frequently used method, may improve the estimates of self-study because it minimises the limitations of the previously described methods, including recall error and changes in students' study habits.This method consists of asking students, every week, to report the time that they devoted to self-study during the previous week.The major disadvantages of this method are that it requires significant involvement from students, and is difficult to apply to large samples.

Purpose
Many students have difficulty mastering the demands of medical courses.Planning student workload, having feedback from students about the actual time involved in their studies, and adjusting the allocated time to study or other educational activities, is a continuous process that puts students at the centre of the educational process (González & Wagenaar, 2003;Mueller et al., 2011).The development of more accurate and precise methods of assessing the actual time that students dedicate to their studies is an important education research priority.There are practical advantages to encouraging the use of retrospective self-reported measures because they are simple and can be administered quickly.Nevertheless, the method should be carefully selected to obtain the most accurate measure.Therefore, the present research compares, for the first time, the validity of three different methods of selfreporting that measure self-study in a medical course, applying the Triads method.This method is a triangular approach that uses the correlations between each of three methods to estimate a validity coefficient that expresses the correlation between, in this case, reported self-study and the subject's unknown 'true' self-study (Ocke & Kaaks, 1997).

Program Setting
The data used in the analysis were obtained from a public medical school in the North of Portugal.The medical course takes six years and is divided into two cycles, pre-clinical years (first three years) and clinical years (last three years).The student workload is required to be 1,620 hours per academic year over 40 weeks, corresponding to a workload of approximately 40.5 hours per week.As proposed by European Credit Transfer and Accumulation System (Bologna Declaration, 1999), this time is divided between 2/3 of self-study hours and 1/3 of contact hours.
The pre-clinical years comprise 32 semester courses of basic and preclinical sciences.The period of classes lasts 14 weeks per semester, and the assessment period is 6 weeks per semester.The clinical years cover 25 courses of

Contribution of this paper to the literature
• Feedback from students about the actual time involved in their studies might lead to an adjustment of the time that is allocated to study.
• There are practical advantages to encouraging the use of retrospective self-reported measures because they are simple and can be administered quickly.However, a major limitation is the validity and reliability of these measures.
• Applying a triangular approach this paper demonstrates that self-reported measures give a valid estimate for self-study and we propose a novel method, based on frequency and study hours, to retrospectively estimate self-study in a medical course.

EURASIA J Math Sci and Tech Ed
3 / 10 clinical practice (clerkships), one optional course, and one course to develop a research project.Clerkships are completed one at a time over the course of 40 weeks.

Participants
At the beginning of the 2014/15 academic year, this study was promoted at a meeting with the student representatives of each curricular year of the medical course, so that five students, representing each year were proposed as volunteers to participate in the project.In order to avoid including only the students who were well disposed to answer our questions, the student representatives of each curricular year randomly selected 5 students.As reward for study participation, additional support was provided to the students in the development of the endof-course thesis.From September 2014 to July 2015, a total of 30 students (8 boys and 22 girls) from all curricular years took part in the validation study.
This study was approved by the Faculty of Medicine of the University of Porto/São João Hospital Ethics Committee.Students' anonymity was ensured and participation in the study was voluntary.

Data Collection
This study involved three methods to measure self-study (Table 1).We used the most common method, endcourse survey, to ask the students to report their average study hours per week, designated Average Report (AR).However, when asked for their average study hours per week throughout the year, the students first had to think of the number of days that they had studied, and second, the number hours that they studied on those days.On this basis, we also created an end-course survey that asked students to report their study frequency over the academic year and the mean hours of studying on a given day, designated Frequency Report (FR).This survey was adapted from food frequency questionnaires (McNaughton, Marks, Gaffney, Williams, & Green, 2005).As we believed that it was highly likely that students' study behaviour would change mainly during the assessment period, on the end-course surveys (AR and FR) administered to pre-clinical years students, the self-study questions were asked twice: during class and assessment periods.Since there is no end-of-course assessment in the clinical years, the self-study questions were placed on the survey only once.Additionally, the students completed the AR and FR surveys twice, at the end of first and second semesters.Finally, we created a continuous weekly survey, which was administered throughout the academic year, asking students to report their study frequency (from never to 7 times a week) and study hours on the days they studied during the previous week, designated Weekly Report (WR).All the students were informed that they would report their self-study time every Monday during the academic year.To ensure participation, a SMS message and a link to the survey were sent weekly.Twelve hundred records were submitted.

Self-study Hours
Methods were compared to calculate the total self-study (Table 1).For AR and FR, the total self-study time was the sum of self-study during the two semesters.In pre-clinical years, the total self-study time for each semester was the sum of self-study during the class period (14 weeks) plus self-study time during the assessment period (6 weeks).In clinical years, where there is no assessment period, the total self-study time for each semester was the self-study time throughout the semester (20 weeks).Additionally, for FR, self-study was the reported frequency category converted to mean times/week (ex: 2 to 3 times per month = 2.5/4 = 0.625) and multiplied by the number of self-study hours.This method has already been used in another study (Barbosa, Silva, Ferreira, & Severo, 2016).
For WR, the reported frequency category for each week (0 to 7) was multiplied by the number of self-study hours for that week.The total self-study time was the sum of all the records.Data was gathered in the form of an online survey.The self-study records were monitored by a member of the project team.The students never had access to their own answers on any previously completed forms.To maintain data independence, students only knew that they would answer the AR and FR questionnaires at the end of each semester.

Data Analysis
Considering the weekly self-study records (WR), we used a linear mixed-effects model to compare self-study during the class and assessment periods stratified by cycle, using student level as a random effect.The data from the variance components (student and residual) were used to estimate the optimal number of weeks needed to estimate the students' self-study time (D-study).The Intraclass Correlation Coefficient (ICC) was calculated to measure the reliability of the methods.
Repeated measures analysis of variance (ANOVA) was used to determine significant differences among the total self-study hours in WR, AR, and FR.Pairwise Pearson's correlation coefficients were calculated to measure the linear relationship among the selfstudy hours that were assessed by the three methods.
The relationship among the methods was validated using the Triads method (Ocke & Kaaks, 1997).This method is used to estimate the association among the three measurements and, in this case, the latent true self-study value using a structural equation approach (Supplemental Data, Appendix 1).For each of the methods, the coefficient of correlation with the estimated true value is referred to as the validity coefficient (VC), with a higher VC indicating a better approximation of true exposure (Kaaks, 1997).This approach assumes that the three methods are linearly related to the common true value and that the errors associated with each of them are independent.The 95% confidence intervals (95% CI) of the correlation and validity coefficients were calculated using the "bootstrap" method, in which 1000 samples of equal size (n = 30) were randomly obtained with replacement from the study subjects.
Statistical analyses were performed using R software version 2.14.1 and a P-value of less than 0.05 was considered significant.

Self-study during Academic Year
Using records from WR, Figure 1 presents how students distribute time devoted to self-study during academic year.Results from the linear mixed-effects model showed that differences between the cohorts in pre-clinical and clinical years were marked for study during the assessment period.During pre-clinical years, students devoted an average of 18.8 hours/week (CI 95%: 13.8-23.4)throughout the class period.The load increased during the assessment period (16 to 22 and 38 to 45 weeks) to approximately 27.8 hours/week (CI 95%: 24.0-31.6).In clinical years, students devoted an average of 13.4 hours/week (CI 95%: 11.0-15.8)throughout the academic year.The variation in study hours among students in different cohorts was greater during the assessment period than the class period.

/ 10
Figure 2 summarises the total self-study hours during the academic year, based on weekly records and retrospective self-reported measures.The estimated mean totals of self-study hours were not significantly different among WR, FR and AR.
The reliability of the methods, as assessed by a single ICC, was also obtained from the same population.The ICC of WR was 0.89 for all cohort students.Based on the available estimated variance components, at least 13 weeks would be needed to achieve a reliability of 0.70 for the students' self-study reporting.The ICC was 0.73 for FR and 0.53 for AR.

DISCUSSION
This study assessed the feasibility of using self-reported measures to determine required self-study time in medical school.Most studies assume no measurement error in the gold standard method (Spiegelman, Schneeweiss, & McDermott, 1997).For example, in our study, we observed a measurement error of 11.0% in the reference method (reliability of 0.89 for WR).To consider a method as the gold standard, it should have no measurement error, and, the Triads method allowed us to estimate the association between each measurement and the unknown "true" self-study without measurement error.
The results indicated that self-reported measures provide a valid estimate of self-study time.Correlations between methods were good.Additionally, the WR method appeared to be better correlated with the FR than the AR method.Based on the estimated VCs, the association among FR or AR and true self-study increased.Surprisingly, the VC for FR was higher than for WR.Greater accuracy was expected for WR.This result may have occurred because FR and AR were recorded at the same time, and we cannot disregard the potential for correlated errors.Like other studies, we considered the upper limits of the true validity coefficient, while, the sample correlations between WR and FR and between WR and AR are the lower limits (Ocke & Kaaks, 1997;Yokota, Miyazaki, & Ito, 2010).Interpreted this way, a single self-reported measure appears to reflect self-study hours with reasonable accuracy, with the lowest validity coefficient ranging from 0.75 to 0.87 (ρART).
When compared with WR, the total average of self-study estimates was not significantly different for FR or AR, supporting the notion that self-study is consistent across the methods.
Additionally, it is important to take into account how students develop their self-study time throughout the academic year.Our study revealed significant differences between cycles marked for study during the assessment period.During pre-clinical years, student performance is mainly assessed by written examinations.Students engage in more study closer to the examination deadlines (Kerdijk, Tio, Mulder, & Cohen-Schotanus, 2013;Lemanski, 2011).This fact may explain the variability of student effort that was observed in our study among students during the assessment period.As discussed in earlier studies, to promote an even distribution of study time over the academic year and create good learning habits in students, more frequent testing is needed (Cohen-Schotanus, 1999;Kerdijk et al., 2015;Mueller et al., 2011).However, strategies to achieve more accurate estimations of self-study averages using retrospective self-reported measures in curricula with end-of-course assessments require evaluating self-study by class and by assessment period.In addition, it seems that students spend more time studying in preclinical years.Taken together these results suggest that calculation survey at course level should only be used if the course has a similar structure along cycles or curricular years.Furthermore, from the variance estimates and given the demands of WR, we determined that we could use this method for 13 weeks to obtain a reliability of 0.70.By way of illustration, in reviewing the literature we found a study that developed a diary to quantify learning activities for medical students during their clinical years (Wilkinson et al., 2005).That was the only study in the medical context to establish the reliability and validity of diaries.Each student completed a diary on 3 randomly selected days.It was shown that to obtain generalisable information about a student's self-study, information from 9 days of reporting would be needed.Similarly to WR, the diary method minimised recall error, required more of the students' time and could lead to student drop-out.One of the strengths of our study was obtaining a large set of student records in this field, twelve hundred records collected during 40 weeks without student drop-out.On this point, the self-reporting methods in our study had acceptable reliabilities and validities that could quantify self-study time.In a sensitive analysis, we checked the validity coefficients stratifying by cycle and with moderate values for clinical years, we also had acceptable validity (Suplemental Data, Appendix 2).Although this validation study was limited by its small sample size, the combination of the three methods by the Triads method allowed us to estimate validity coefficients, and, to detect a moderate correlation of 0.5 we have a power of 82%.

CONCLUSION
Our results showed that self-study hours can be measured retrospectively.Therefore, we propose a novel method, based on frequency and study hours, FR, to retrospectively estimate self-study in a medical course.The results of this study will allow other studies that use the same methods to correct their estimates between reported self-study and other variables that affect self-study, such as curriculum and individual characteristics.The next step of our research should be to create an information system that integrates self-study measured with FR.As data accumulate, medical school will gather a full picture of students' self-study.Feedback from students about the actual time involved in their studies is the key to promote and reward students' efforts, and to allow a fair distribution between and within course units.

Figure 1 .Figure 2 .
Figure 1.Distribution of self-study hours per week by cycle.Vertical lines delimitate the assessment period.Boxplots represent self-study distribution in classes' period and assessment period by cycle (pre-clinical years grey boxplots, clinical years white boxplots)

Figure 3 .
Figure 3. Triads' method -Triangular comparison among self-study methods.The relationship between each two methods is denoted by correlation coefficient (rWRFR, rWRAR, rRFRAR) outside the triangle.Validity coefficients between each self-study method and the true self-study are shown inside the triangle (ρWRT, ρFRT, ρART)

Table 1 .
Summary of self-report methods to measure self-study