Abstract
Objective To assess the communication and interviewing skills of incoming residents and provide formative feedback to residents early in their training.
Design New residents completed a 15-minute objective structured clinical examination (OSCE) assessing communication skills and a 12-question, self-administered content quiz at the start of their residency. Each resident was directly observed by a family physician in the OSCE and provided with 15 minutes of structured feedback, with an opportunity for questions and discussion. The entire process remained private and did not affect summative evaluations.
Setting Family medicine residency training program at the University of Alberta in Edmonton.
Participants First-year family medicine residents.
Main outcome measures Residents’ scores on the OSCE and the content quiz; residents’ rating of the usefulness of the assessment and the likelihood it would lead to practice change.
Results A total of 61 residents (93.8%) completed the skills assessment (50 Canadian graduates, 11 international graduates). The mean score for the content quiz was 20.6 out of a total possible score of 24. Resident scores ranged from 8 to 24. The mean score on the OSCE practice interview was 21.1 out of 30, with a range of 13 to 29. Learner feedback indicated that the skills assessment was useful (4.68 out of 6) and would lead to a change in practice (4.43 out of 6).
Conclusion The introductory communication OSCE and quiz offer new residents an opportunity to gauge their baseline skill level, become aware of program expectations early in their training, and garner specific suggestions in a nonthreatening environment. This tailored approach helps orient residents while taking into account their previous experiences.
The doctor-patient relationship is the cornerstone of medical care1 and has been shown to affect treatment outcomes and patient satisfaction.2–6 The need for strong communication skills has been well explored.7,8 As such skills are an essential component of medical school and postgraduate curricula, further advances are necessary to ensure the development of competent clinicians and communicators.
A variety of assessment tools have been developed to evaluate the doctor-patient relationship,9 with a particular emphasis on communication.9,10 With the shift to competency-based curricula, there will be an increased need for evaluation tools that can offer formative and constructive feedback via observed interactions. Formalized communication will therefore be necessary11,12 and will require trained observers and checklists, with the possible addition of multiple observers or videorecording.13–15 Objective structured clinical examinations (OSCEs) remain the cornerstone of clinical examination, despite offering poor reliability in evaluating communication skills alongside clinical acumen.16–18
Many individuals change cities or countries when transitioning from medical school to residency. The resulting diverse backgrounds and previous experience create the potential for unequal footing during residency. In the family medicine program at the University of Alberta in Edmonton, we piloted an initial evaluative OSCE to ascertain the baseline characteristics of residents while offering a structured and formalized avenue for feedback specifically on communication skills. Given the paucity of observed interactions noted in many residency programs, it is essential that residents are offered the opportunity to have direct, specific feedback targeting their style, technique, and execution in patient interviews. As the patient history is key to medical diagnosis and the patient encounter is a critical component of the doctor-patient relationship, investing time and resources to ensure development of baseline expectations might enhance the communication skills of residents.
METHODS
In 2012, the Department of Family Medicine at the University of Alberta introduced a skills assessment at the beginning of residency. The purpose was to assess residents’ baseline skills in order to better support their training.
The skills assessment involved 2 components: an online written quiz and an observed OSCE practice interview. The online quiz evaluated the resident’s knowledge of communication microskills, such as paraphrasing and use of open-ended questions. Behaviour relating to personal well-being and resiliency was also assessed. The practice interview was observed live on camera by experienced faculty. Oral and written feedback were provided to the resident on listening skills, verbal and nonverbal communication, and the overall effectiveness of the interview. All the residents completed a formal evaluation of the process following the assessment, rating elements of the assessment on a 6-point Likert scale.
The content of the online written quiz and the evaluation criteria for the observed practice interview were determined by a community of family medicine faculty and reflect some basic competencies in interviewing and communication necessary for doctor-patient interaction. This 2-part skills assessment was based on an evaluation framework used by the College of Family Physicians of Canada.19,20 The skills assessment was then piloted with both faculty and residents in family medicine to ensure acceptability and feasibility. Ethics approval was obtained through the University of Alberta Health Research Ethics Board.
RESULTS
A total of 61 residents completed the skills assessment process, representing 93.8% of the first-year residents. Of the residents who completed the assessment, 26 were female (42.6%); 50 (82.0%) were Canadian medical graduates, while the remaining 11 were international medical graduates (IMGs). Some residents could not participate owing to scheduling conflicts with mandatory service-based rotations.
Residents scored well on the online quiz (Table 1). The mean (SD) score was 20.6 (3.05) out of a total possible score of 24 (range 8 to 24). The mean (SD) score on the observed practice interview was 21.1 (4.32) out of 30 (range 13 to 29). Table 2 shows the results of the practice interviews broken down by types of skills observed. In addition to scoring the resident, the faculty observer looked for specific behaviour during the practice interview. Elements of FIFE (feelings, ideas, function, expectations) were demonstrated by only half of the residents. Table 3 lists the behaviour observed and the frequency with which each was observed among the residents.
Results of the online assessment of communication skills
Results of the communication OSCE assessment: Each skill is scored out of 5 and the total is scored out of 30.
Behaviour demonstrated by residents during the communication OSCE: N = 61 residents.
Overall, the residents found the 2-part skills assessment quite valuable (Table 4), citing introspective learning and potential future integration of feedback, and believed it was an appropriate measure of their skill.
Results from the resident evaluation of the combined skills assessment
DISCUSSION
As in previous studies, residents’ skills assessment scores were highly varied, likely owing to their different experiences before commencing training.21,22 Therefore, the skills assessment can offer educators insight into the trends and baseline characteristics of the incoming cohort. Further, it creates an organized avenue for residents to receive targeted feedback in an area that will likely go unsupervised for most of their training.
Although the online assessment was unsupervised, it is an emerging avenue for assessing knowledge of foundational concepts, including definitions and frameworks. Determining the value of such assessment will require further research to elucidate possible relationships with the skills assessment and with resident performance.
Residents who might not have been aware of these key concepts beforehand were able to take the opportunity to supplement their knowledge before commencing their training. It is interesting to note that some residents still performed poorly in the online quiz. We hypothesize 2 possible reasons: either residents thought that they were not able to use resources while completing the quiz, or perhaps they trained at institutions that used alternate methods to impart these skills, as these individuals seemed to perform as well as their peers in the observed OSCE practice interviews.
Fortunately, most residents demonstrated solid foundational knowledge in their interviews, possibly owing to the increasing emphasis placed on communication skills in medical schools. Of note, individuals seemed to perform poorest in the FIFE domain, an interesting development given the relative ease of scoring points in this category. Feedback given after the interviews was well received by residents, with by far most indicating that it would lead to practice change and also noting that it offered introspective discovery.
Limitations
Comparing IMGs to graduates of the Canadian system poses an interesting challenge, given the relative differences in curriculum designs and the heterogeneity of the former group. Previous literature has documented the need for communication training in IMG populations.23,24 In our study the groups scored similarly on both assessments (t test, P = .52 for the quiz and P = .81 for the OSCE); however, the small sample size and single-site design are notable limitations. In Alberta, IMGs must first complete a longitudinal observership and are prescreened with communication OSCEs before receiving acceptance into our residency program.25 It is plausible that this leads to a selection bias when considering how this cohort performs on a downstream assessment, and our results might not be generalizable to other programs.
Conclusion
An introductory assessment of communication skills through observed interviews and an online quiz might be a valuable tool to stratify the baseline characteristics of new trainees. Country of training might not play a pivotal role or might be minimized by the varying entry requirements for IMGs and Canadian graduates. Residents found this to be a useful exercise with the potential to change practices as a result of the specific feedback offered. Further research can examine if these assessments can be used to identify residents who might struggle in the future.
Notes
EDITOR’S KEY POINTS
The diverse backgrounds and previous experience of residents creates the potential for unequal footing during training. This study aimed to assess the communication and interviewing skills of incoming residents and provide early formative feedback.
New residents completed an online quiz and an objective structured clinical examination assessing communication skills and received structured feedback from experienced family medicine faculty. Generally, residents performed well on both assessments, with similar performance among Canadian and international medical graduates.
The residents found the assessment useful and indicated that it offered the opportunity for introspective learning and had the potential to change their practice.
POINTS DE REPÈRE DU RÉDACTEUR
Parce qu’ils ont eu des conditionnements et des expériences différents, les résidents risquent de commencer leur formation sur des bases inégales. Cette étude voulait évaluer la capacité des nouveaux résidents de communiquer et d’effectuer des entrevues, tout en leur offrant un feedback précoce.
Les nouveaux résidents ont répondu à un quiz en ligne et à un examen clinique objectif structuré qui évaluaient leur habileté à communiquer pour ensuite profiter d’un feedback structuré de la part de professeurs expérimentés en médecine familiale. Les résidents ont généralement bien performé à ces deux évaluations, les diplômés canadiens et étrangers obtenant des résultats semblables.
Les résidents ont trouvé cette évaluation utile et souligné qu’elle leur offrait une occasion d’apprentissage introspectif susceptible de modifier leur pratique.
Footnotes
This article has been peer reviewed.
Cet article fait l’objet d’une révision par des pairs.
Contributors
Both authors contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
References
In this issue
