Abstract
Objective To assess use of the remediation instrument that has been implemented in training sites at the University of Montreal in Quebec to support faculty in diagnosing and remediating resident academic difficulties, to examine whether and how this particular remediation instrument improves the remediation process, and to determine its effects on the residents’ subsequent rotation assessments.
Design A multimethods approach in which data were collected from different sources: remediation plans developed by faculty, program statistics for the corresponding academic years, and students’ academic records and rotation assessment results.
Setting Family medicine residency program at the University of Montreal.
Participants Family medicine residents in academic difficulty.
Main outcome measures Assessment of the content, process, and quality of remediation plans, and students’ academic and rotation assessment results (successful, below expectations, or failure) both before and after the remediation period.
Results The framework that was developed for assessing remediation plans was used to analyze 23 plans produced by 10 teaching sites for 21 residents. All plans documented cognitive problems and implemented numerous remediation measures. Although only 48% of the plans were of good quality, implementation of a remediation plan was positively associated with the resident’s success in rotations following the remediation period.
Conclusion The use of remediation plans is well embedded in training sites at the University of Montreal. The residents’ difficulties were mainly cognitive in nature, but this generally related to deficits in clinical reasoning rather than knowledge gaps. The reflection and analysis required to produce a remediation plan helps to correct many academic difficulties and normalize the academic career of most residents in difficulty. Further effort is still needed to improve the quality of plans and to support teachers.
Each year, 3% of residents in the family medicine program at the University of Montreal (U of M) in Quebec do not achieve a passing grade in rotations, and approximately 10% experience academic difficulties severe enough to prolong their residency. This is consistent with current literature on students experiencing academic difficulties.1–4
This situation has received considerable attention, both from the perspective of the learner and from the perspective of the clinical educator. Hauer and colleagues identified obstacles on the learners’ part, such as anxiety and lack of motivation, which prevent them from listening to teachers’ remarks and suggestions. They also pointed to learner unwillingness to engage in reflective processes or remediation activities.5 Clinical educators, on the other hand, bemoan the added time and energy required to assist such students.6,7 Two contextual factors that further compound the problem have been described: clinical workload that restrains supervision of essential aspects of resident training (such as time management, clinical flow, variety, and complexity of patient issues)8 and the dual role9,10 of the clinical educator as he or she is called upon to simultaneously manage the clinical case load, centred on patients, and observe and assess student performance.11 The clinical educator must constantly engage in clinical reasoning and pedagogic reasoning.12 In spite of being aware of the double role, many clinical educators tend to place greater emphasis on solving clinical problems, staying within their comfort zone of medical expertise rather than devoting time to helping struggling students.13 Add to this the limited number of available tools and insufficient knowledge of learning processes, it is not surprising that many clinical educators are uncomfortable and unsure of their teaching skills.14,15
Cleland and colleagues recently provided useful up-to-date knowledge on the conceptual and methodologic issues related to remedial interventions in medical education.16 One of their key findings is that many remedial interventions lack a theoretical foundation: more often than not, interventions amount to providing “more of the same” to learners in difficulty.16 Instead of diagnosing the nature of the difficulties and customizing remedial teaching appropriately, additional or intensive teaching is blindly provided. Residents thus continue to encounter similar difficulties that they are not able to overcome without targeted help.
The few published studies on this issue, as in higher-education literature overall, clearly indicate a need for an array of instruments that can help students overcome learning difficulties. These instruments should allow for a correct description of the learning difficulty at the outset, the subsequent development and implementation of a targeted remediation plan, and, finally, assessment of the effects of the resulting changes.17–22 Recent medical education literature describes a number of implementation guidelines for faculty, such as those published by the Council of Emergency Medicine Residency Directors Remediation Task Force. The task force recently put forth a number of recommendations for residency programs (Box 1).23 These recommendations also highlight the need to define structured, documented, well planned remediation processes.23
Council of Emergency Medicine Residency Directors guidelines for resident remediation
Recommendations for residency programs include the following:
|
Data from Katz et al.23
Development of the remediation process
Since 2006, the Family Medicine Residency Program Assessment Committee at the U of M has had the responsibility of deciding whether a student passes or fails any rotation within the residency program. This decision is made after careful consideration of the assessments completed by clinical educators in the different rotation sites. From the start, the committee was acutely aware of the variable value of the assessments completed in different teaching sites using different assessment practices. Further, the information submitted to the committee about residents undergoing academic difficulty did not always include details about the nature of the difficulties or the delayed advancement of the residents in their training. Finally, closer scrutiny of rotation assessment sheets revealed that clinical educators often described the difficulties encountered but rarely ventured to provide any diagnostic information. Even when the clinical educator correctly identified difficulties, a remediation process was not systematically implemented.
In order to improve the assessment process in the program, the Family Medicine Residency Program Assessment Committee designed a remediation instrument and procedure24 inspired by existing literature.18,21,23,25–27 The aim was to enhance faculty support for diagnosing academic difficulties and subsequently developing, planning, and implementing remediation plans. The instrument is based on an iterative mechanism of consultation and joint construction between clinical educators and residents, who act as partners in the plan’s development and implementation. This approach yielded 2 important outcomes: a more comprehensive instrument better suited to the needs of users and the possibility for the teachers to gradually take ownership of the whole project.28,29
The remediation instrument is based on Vaughn and colleagues’ taxonomy that distinguishes between 4 types of problem learners, which provides useful cues for the action-planning phase of remediation (Box 2).27,30 The instrument is also constructed with reference to problem-solving principles described by Vaughn et al,27 namely the S-T-P model (specify the problem, target state, procedure plan), and to the 4-part remediation process suggested by Hauer et al (ie, competence assessment, diagnosis of deficiency and development of an individualized learning plan, instruction and remediation, and focused reassessment).18
Categories of learner difficulties
Learner difficulties can be divided into 4 categories:
|
Adapted from Vaughn et al.27
Our instrument consists of 2 components: a remediation plan form containing 5 sections (pedagogic diagnosis, remediation intentions, remediation methods, remediation assessment, and a contract) and remediation plan writing guidelines. Space is provided on the remediation plan form to enter a midrotation assessment and a final assessment.
The instrument was introduced at the U of M teaching sites in July of the 2009 to 2010 academic year.24 A focused, purpose-specific teacher-training strategy was implemented. This strategy consisted not only of facilitating workshops on themes such as assessment, pedagogic diagnosis, and implementing the remediation instrument, but also of providing clinical educators with access to 3 experts to coach them in diagnosing pedagogic difficulties and in the implementation of targeted remediation plans. Over the months following implementation, the quality of information transmitted to the assessment committee revealed that working with the remediation instrument made the assessment process more rigorous. It became easier for the committee to decide whether a student’s remedial rotation had been successful and made it easier to monitor the student’s overall training.24
On the heels of these positive implementation results, we wished to gain a deeper understanding of the effects of this important change on the faculty’s ability to manage residents experiencing difficulties. We sought to understand whether and how the remediation plans jointly developed by clinical educators and residents in family medicine units led to more precise pedagogic diagnoses and implementation of targeted remediation strategies; whether implementation of a remediation plan was positively associated with the resident’s success in rotations following the remediation period; and whether the student success rate in rotations following the remediation period increased in accordance with the number of quality criteria a remediation plan met.
We sought to better apprehend issues involved in the use of remediation plans by clinical educators, as well as to generate reliable, valid, and useful information to adapt remediation plan development and implementation in our department.
METHODS
The purpose of our study was 2-fold: to clarify how and why our intervention in medical education has provided certain results and to assess whether there was a difference before and after remediation plan implementation in residents’ rotation assessments that could be attributed to the presence of quality criteria in the plan.31 As such, our study is both a system-level assessment of effect32 and a clarification study.33
Clarification studies are premised on predictions made possible by previous research and aim to test these predictions. Confirmation of the prediction would support the proposed model and strengthen its assumptions. Nonconfirmation of the prediction would allow for identification of possible weaknesses and point to areas for improvement.33
Our predictions are embedded in the analytic framework that we have developed.24 Table 1 presents this framework and its 5 dimensions, 3 of which relate to the content of remediation plans (types of difficulties,27 remedial methods used, and severity of the difficulties) and 2 of which relate to the implementation process (latency and duration of the remediation period, quality criteria for the remedial plans). This framework is based predominantly on quality criteria for remediation plans described by Vaughn and colleagues27 and Katz and colleagues.23
Framework for analyzing remediation plans
Similarly, studies of system-level effects aim to determine whether educational interventions have directly or indirectly contributed to changed outputs.31 This approach is widely used in educational program assessment and can provide educational designers with evidence of desired results. Without pretence of a direct cause-and-effect relationship, evidence of the desired outcomes suggests, minimally, that the educational intervention did not hinder the results. For residents experiencing academic difficulties, the desired outcome is for them to pass a greater number of rotations and ultimately successfully complete their residency program.
Data analysis
Remediation plans were examined with deductive analysis using our analytic framework (Table 1).27 The goal of the analysis was to verify the predictive value of the all items in the analytic framework.
To assess the effects of implementing the remediation plans, we proceeded with quantitative analysis. The dependent variables comprised the ratios, expressed as percentages, of the number of successful rotation periods (1 period was 4 weeks) relative to the total number of periods preceding the first sign of academic difficulty (ie, the first unsuccessful rotation) compared with the number of successful rotation periods relative to the total number of periods following the remediation to the end of the residency program. The independent variables considered were subjects’ personal characteristics (age, sex, year of residency, teaching site, undergraduate and postgraduate academic course of study) and the quality criteria of the remediation plans (Table 1).27
To specifically determine whether a remediation plan was positively associated with the resident’s success in future rotations, correlations were calculated between the subjects’ personal characteristics, the qualitative criteria of the remediation plans, and the proportion of periods considered to have been successful relative to the total number of periods in the program. In addition, a 1-way repeated ANOVA (analysis of variance) test was conducted. As described in the literature,34 this analysis is used to evaluate the difference between 2 time measures (before and after the intervention) using continuous dependent variables (in our case, the proportion of periods considered to have been successful, relative to the total number of periods, before and after the remediation).
To determine whether resident success was related to the quality of the remediation plan, a linear regression was used to evaluate if some quality criteria were significantly associated with the proportion of periods considered to have been successful relative to the total number of periods after the remediation.
RESULTS
We analyzed all remediation plans (N = 23) produced by the teaching sites in the program during the period under study. The characteristics of the study participants are shown in Table 2.
Residents who completed remediation plans compared with their peer group
Characteristics of the remediation plans
A total of 23 remediation plans were developed for 21 residents in the residency program (2 residents underwent 2 consecutive remediation plans), representing 10% of the residents in the program during the study period (N = 205 residents). Of the 16 teaching sites in the program, 10 produced the 23 plans in our sample; 16 plans (70%) were produced in urban teaching sites and 7 plans (30%) were produced in rural teaching sites.
Types of difficulties
As illustrated in Table 3,27 cognitive problems were identified in 100% of the remediation plans. Closer inspection of the plans showed that the problems mainly involved clinical reasoning, not just lack of knowledge. Further, in 67% of cases, cognitive problems coexisted with other issues.
Distribution of academic difficulties
Severity of the difficulties
The severity of the difficulties was slight for 6 of the plans (26%), moderate for 8 of the plans (35%), and severe for 9 of the plans (39%). We did not identify any differences related to the participants’ age or sex. However, we did observe that, in this group, the difficulties identified were often more severe among international medical graduates (IMGs) compared with Canadian medical graduates (Table 4).
Severity of academic difficulties according to location of undergraduate medical study
Remediation methods and strategies
Numerous measures were implemented to achieve the various remediation objectives. Table 5 lists the remedial methods used according to the type of difficulty.
Frequency of measures used for different types of difficulties
Strengths and weaknesses of the remediation processes
Remediation latency and duration
The latency period (the time between the first signs of academic difficulty and the beginning of the remediation process) ranged from 1 to 29, 4-week periods with a median of 7 periods. The remediation processes varied in duration from 1 to 7 periods with a median of 3 periods.
Quality criteria
Based on the quality criteria in Table 1,27 11 plans (48%) were deemed good, 1 plan (4%) was average, and 11 plans (48%) were poor. The main shortcomings of the poor-quality plans were failure to follow through with the remediation measures, the absence of documentation for the various steps in remediation, and a lack of remediation assessment.
Effects of remediation plans
Effects of the plans for residents in difficulty
The study of residents’ postremediation rotation evaluations revealed 3 career profiles.
Substantial improvement: Twelve residents (57%) showed substantial improvement. Eight of them had a faultless course of study, 3 were successful in more than 80% of their rotations, and 1 showed improvement of more than 30%.
Persistence of difficulties: Two residents (10%) needed a second plan shortly after the end of the remediation period to correct residual difficulties. In both cases, where a second remediation plan was necessary, the second postremediation period showed that the difficulties persisted; success rate was 58% for 1 resident at the end of residency and 50% for the other resident. Further, 4 other residents (19%) had a comparable success rate before and after remediation; that is, the remediation plan does not seem to have made a difference.
Persistence of considerable difficulties: Three residents (14%) were dismissed from the residency training program for academic reasons after the remediation ended.
System-level effects
Correlations and linear regressions for the subjects’ personal characteristics, the quality criteria of the remediation plans, and the percentage of successful rotation periods were calculated. Surprisingly, they did not show any significant statistical relationships. We used a 1-way repeated ANOVA to compare the percentages of successful rotation periods before remediation (N = 23, mean [SD] of 53% [29%]) and after remediation (N = 23, mean [SD] of 73% [29%]). The goal of this ANOVA was to verify if this mean difference was statistically significant. In order to compute this ANOVA, the data set had to be transformed to meet normality assumption requirements. A slight skew in the subjects’ independence had to be tolerated because 2 residents were involved in 2 remediation plans during the data-collection period. The final ANOVA results (F1,22 = 8.19, P = .0091) confirm a statistically significant difference between the 2 measurement times. These results confirm that implementation of a remediation plan is positively associated with the resident’s success in rotations following the remediation period.
Owing to the absence of significant statistical relationships between the quality criteria of the remediation plans and the proportion of successful postremediation rotation periods, we were unable to definitively determine whether the number of quality criteria plans met was associated with resident success. However, our failure to find evidence of this relationship might be related to a lack of statistical power, as the number of participants is relatively small.
DISCUSSION
Remediation processes were contextualized within the training sites. We consider the remediation plan sample to be representative of both the training sites and the student population in our residency program. The plans come from 10 of our training sites: their distribution adequately reflects the geographic distribution (rural vs urban) and the number of years of existence (more than 5 years vs less than 5 years of existence) of the training sites as a whole.
Representative study group
The percentage of residents for whom a remediation plan was developed (10%) is proportional to the overall percentage of residents experiencing difficulty in our program during the study period (13%). This proportion is consistent with what is reported in the literature, depending on how a resident in difficulty is defined.2,3 The ratios of male to female and first- to second-year residents in our sample reflect the ratios in the peer group. Statistics from our department show that 10% of Canadian medical graduates and 50% of IMGs experience substantial academic difficulties during their residency training. Hence, it is not surprising that IMGs are overrepresented in the study group, as they tend to encounter greater difficulties in our program.
Primarily cognitive difficulties
The difficulties for which a remediation plan had to be developed were mainly cognitive in nature. In 67% of the cases, residents presented cognitive difficulties, as well as other types of difficulties. This echoes reported results that residents rarely present just 1 type of academic difficulty.2,3,7,22 However, unlike in these past studies, our qualitative study of the plans showed that cognitive problems were mostly difficulties with clinical reasoning or organization of knowledge rather than knowledge gaps.
Need to quickly initiate a concrete, targeted, and organized remediation process
Great variations were observed in the time between the first signs of academic difficulty and the beginning of the remediation process. We were able to identify some factors that can explain this, such as the difficulty of making an early diagnosis and a lack of expertise and skills in preparing a remediation plan. More research would be necessary to identify other factors that could be involved. Note, however, that in some cases long latencies were due to the fact that no remediation plan existed at the time problems were identified.
Plans deemed to be poor lacked practical information (eg, who does what, when, and how), had no record of the actual implementation and follow-up, and had no remediation assessment reports. The fact that there was no individual nominally in charge of the remediation process, in a limited number of cases, was associated with the absence of practical information and remediation assessment reports. It is important to note that plans were judged on the basis of the criteria in Table 123 and not supervisors’ teaching. These results lead us to think that, although supervisor training is necessary, it is equally important to offer organizational support for the remediation process given the unpredictability of clinical contexts and locations in which supervisors juggle multiple duties. Having one person nominally in charge of the remediation process would appear to be a good solution.
Preference for remediation methods focusing on knowledge development
Our results show that multiple remedial methods were used. Indeed, the number of different methods for any given plan ranged from 1 to 6. Consistent with difficulty prevalence, the remedial methods for cognitive problems were the most frequent. However, although the cognitive problems identified mostly involved clinical reasoning as opposed to lack of knowledge, the most popular remediation method (reading program used in 74% of the plans) targeted knowledge improvement. Organizational methods (intensified direct supervision, lighter clinical duties, change in clinical duties) were the second most prevalent. In only 48% of the plans did supervision specifically focus on clinical reasoning. In some plans (13% to 30%), specific teaching means (eg, clinical reasoning exercises, simulated medical interviews, in-depth reasoning exercises involving development of algorithms or concept mapping)—likely to correct clinical reasoning problems— were introduced. Some methods (eg, meetings with a tutor, teacher review of recorded interviews, objective structured clinical examination) could be specifically designed to rectify clinical reasoning problems, but we were unable to identify this reliably with our data.
These results confirm our findings from a previous, more specific, qualitative study on identification and handling of clinical reasoning problems, in which we observed that the perception of clinical reasoning difficulties is not supported by an in-depth, explicit understanding of how clinical reasoning develops. This contributed to costly time wasting before obtaining a diagnosis, mistaking clinical reasoning problems for knowledge problems, and implementing remediation plans that were insufficiently supported by specific clinical reasoning tools.13
Effects of the plans for residents in difficulty
Most residents (57%) showed substantial improvement after completing their remediation plans. For 29% of residents, difficulties persisted. A further 3 residents continued to experience substantial difficulty and were ultimately dismissed from the program. These results are consistent with those from studies by Reamy and Harman2 and Zbieranowski and colleagues,22 which also showed that most remediation processes led to academic success.
Although our quantitative data did not support the hypothesis that the various criteria contributing to plan quality would have an influence on remediation, we were nevertheless able to observe that the simple fact of developing and implementing a remediation plan helped to improve the rotation success rate. While our results did not yield a statistically significant covariance with plan quality, it is logical to believe that the observation, analysis, teamwork, and reflection required to develop a remediation plan, whatever its quality, could improve resident success.
As noted by Katz et al,23 some problems are not remediable. But, in cases where residents are dismissed, this process of creating a remediation plan can provide reassurance to teachers that everything possible to support their students was done.
Study limitations and strengths
Educational program evaluation is a complex undertaking. The literature on the subject makes it abundantly clear that it is nearly impossible to find evidence of a direct link between observed system-level effects and an educational intervention.35,36 The accepted approach to mitigate this complexity has generally been to multiply input from stakeholders, use multiple data sources, and prepare for impact assessment concurrently with educational program design.31
From the start, we have insisted that faculty be directly involved in developing our remediation instrument, which has led to positive results in our study of effect. This is certainly our strength. A further strength is our use of multiple data sources to observe any possible effects. Finally, because our approach to remedial plans aims to improve diagnosis of academic difficulties, we believe it has a greater effect on resident success in the long run.
One of the limitations of our study arises from the absence of data collected directly from our stakeholders. For example, it would be interesting to record faculty experiences with using the remediation plan and their opinions about its strengths and weaknesses.
Also, our research was conducted in the very specific context of clinical teaching in Quebec. It would be interesting to study similar experiences of development and implementation of remediation plans in different settings and in other postgraduate residency programs to verify the transferability of our results.
Finally, plans for only 21 students were included in our study. Further research with a large sample of students would afford a more nuanced portrayal of the effect these plans can have.
Conclusion
This research allowed us to draw several conclusions that could interest institutions and teachers concerned with improving their remediation processes.
When implemented strategically and supported by the department and the institution’s graduate education assessment committee, remediation processes are accepted and used by teachers.
Academic problems are often mixed and mainly cognitive: contrary to what might be believed, it is clinical reasoning and not a knowledge gap that is involved in most cases.
The remediation methods used by teachers are often not perfectly congruent with the nature of the difficulties. Teachers tend to give priority to reading programs and organizational measures instead of correcting specific problems.
Faculty training is needed to help teachers better diagnose residents’ difficulties and target the best ways to fix problems faster. Specific faculty development initiatives involving a coaching support program for on-site supervisors should be implemented.
The presence of a remediation coordinator who organizes and supports the implementation of the teaching activity planning chart would maximize the probability that the teaching activities take place.
The reflection and analysis required to produce a remediation plan seems to help correct most academic difficulties and normalize the academic career of most residents in difficulty.
Notes
EDITOR’S KEY POINTS
Approximately 10% of family medicine residents at the University of Montreal in Quebec experience academic difficulties. Academic problems are often mixed and are usually cognitive in nature. Early intervention is key.
A concrete, targeted, and organized remediation process is necessary to address academic difficulties. Developing and implementing such a remediation plan helped to improve rotation success rates (P = .0091). Even though less than half of the remediation plans examined in this study were deemed to be of good quality, 57% of residents showed substantial improvement, and success was not necessarily related to the assessed quality of the plan (although the power of the study might have been too low to detect such an association).
Teachers tended to focus on improving knowledge even if other difficulties were identified. Teachers need to be supported in the diagnostic and remediation process by faculty development. Coaching for on-site supervisors might be a good way to achieve this.
POINTS DE REPÈRE DU RÉDACTEUR
À l’Université de Montréal, environ 10 % des résidents en médecine familiale éprouvent des difficultés d’ordre académique. Ces problèmes sont souvent mixtes et sont généralement de nature cognitive. Une intervention précoce est la solution.
Les difficultés académiques exigent des mesures correctives concrètes, ciblées et bien organisées. Le fait de développer et de mettre en place un tel projet a entraîné un meilleur taux de succès dans les stages (P = .0091). Même si moins de la moitié des mesures correctives mentionnées dans cette étude étaient jugées de bonne qualité, 57 % des résidents ont connu une amélioration appréciable, et leur succès n’était pas nécessairement en rapport avec la qualité du plan (la puissance de l’étude pourrait toutefois avoir été trop faible pour détecter une telle association).
Les professeurs avaient tendance à concentrer leurs interventions sur l’amélioration des connaissances, même si d’autres difficultés étaient présentes. Ces intervenants avaient donc besoin d’être aidés dans le processus de diagnostic et de remédiation par une formation facultaire. Un encadrement des superviseurs pourrait être une bonne façon d’y arriver.
Footnotes
This article has been peer reviewed.
Cet article fait l’objet d’une revision par des pairs.
Contributors
Drs Audétat and Sanche designed the study. Drs Audétat, Sanche, and Béland conducted qualitative data collection. Dr Voirol conducted quantitative data processing, analysis, and interpretation. Drs Audétat, Sanche, Béland, Voirol, and Fernandez participated in qualitative data analysis and interpretation. Drs Audétat and Sanche produced a first draft of the manuscript, and Drs Beland, Voirol, and Fernandez provided suggestions and modifications to successive versions of the manuscript until the final version.
Competing interests
None declare
- Copyright© the College of Family Physicians of Canada
References
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.