EducationAssessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability
Introduction
Emergency medicine (EM) residents must develop effective communication skills to take accurate and efficient medical histories, enhance patient compliance and satisfaction with treatment, optimize clinical outcomes, provide emotional comfort, assure informed consent, and reduce the risk of litigation. Educators, therefore, need a reliable method for evaluating how well residents communicate with patients.
We describe our experience in developing a process for assessing residents’ communication skills. We hypothesized that the reliability of this process would be enhanced through: 1) a faculty development initiative that increased knowledge about physician-patient communication in the emergency department (ED); 2) videotaping of actual resident-patient encounters in the ED; and 3) development of a list of key communication behaviors relevant to the emergency department encounter. Through this process, we created an observation instrument for assessing communication skills that we tested for inter-rater reliability.
Section snippets
Population and setting
The study was conducted within an EM residency at an urban teaching hospital with an annual ED census of 52,000 visits. The protocol was approved by the Institutional Review Board for Human Experimentation.
A video camera was installed in one of 13 ED examination rooms used for patients whose illness fell within a middle range of acuity. These are patients who require urgent attention but do not require resuscitation. The camera was mounted above and behind the patient, facing the resident. It
Results
A convenience sample of 50 patients gave consent for videotaping between December 1992 and May 1993. Thirty-nine tapes were excluded because the camera had not been activated in time to capture the beginning of the encounter. Ten residents were recorded on the remaining 11 videotapes. Age, sex, diagnosis, and disposition of the entered patients are listed in Table 1.
Data are presented in Table 2 on agreement among raters regarding whether or not an individual behavior was performed. Agreement
Rationale of assessment process
Most EM residencies rely upon random and informal observations of trainees to evaluate communication skills (4). However, the adequacy and reliability of bedside assessment of clinical skills has been brought into question 5, 6, 7, 8. To be reliable, evaluations must be based upon multiple observations and should involve multiple observers 9, 10.
The assessment process developed in this study preserved features typical of the bedside method used in residency programs: independent raters observed
Conclusions
We describe a process to assess EM resident communication skills that incorporates videotaping of actual patient encounters, faculty development through literature review and group discussion of videotapes, and the creation of an observation instrument. We then prospectively applied this instrument and tested its reliability. Inter-rater reliability was high for behaviors related to personal introductions, conflict management, non-verbal behavior, and overall performance. There was no agreement
Acknowledgements
This study was supported in part through the “Innovation in Medical Education Award,” Emergency Medicine Foundation/Society for Academic Emergency Medicine, 1992–1993.
References (16)
Teaching the art of emergency medicine
Ann Emerg Med
(1991)- et al.
Teaching medical interviewing: a basic course on interviewing and the physician-patient relationship
Arch Intern Med
(1992) - et al.
Assessment of medical interview skills
Statistical methods for rates and proportions, 2nd edn
(1981)- et al.
How accurate are faculty evaluations of clinical competence?
J Gen Intern Med
(1989) - et al.
How well do internal medicine faculty members evaluate the clinical skills of residents?
Ann Intern Med
(1992) - Consensus statement of the researchers in clinical skills assessment (RCSA) on the use of standardized patients to...
- et al.
Observation of emergency medicine residents at the bedside: how often does it happen?
Acad Emerg Med
(1995)
Cited by (14)
Self-reported use of communication techniques in the emergency department
2012, Journal of Emergency MedicineCitation Excerpt :Consequently, many patients leave the ED without an understanding of their diagnosis, follow-up plan, or how to care for themselves at home (14,15). Several studies have begun to describe specific aspects of communication in the ED, including global patient assessments of the quality of communication, content analyses of audiorecorded conversations, and codifying the verbal utterances in the encounter for elements, such as rapport building and patient centeredness (16–21). These studies highlight important aspects of the verbal communication that occurs in the ED; however, to our knowledge, no studies have quantified the use of different communication techniques in the ED environment.
Conflict Resolution in Emergency Medicine
2012, Emergency Medicine: Clinical Essentials, SECOND EDITION"Sign right here and you're good to go": A content analysis of audiotaped emergency department discharge instructions
2011, Annals of Emergency MedicineCitation Excerpt :Instead, individual residency programs are left to develop their own methods. A survey of ED program directors found that nearly all (98%) of the program directors agreed that communication skills should be taught.38 When asked about bedside instruction/evaluation of residents' interpersonal skills, program directors reported mean supervision rates of 14.7% for giving discharge instructions.
Assessing provider-patient-parent communication in the pediatric emergency department
2002, Ambulatory PediatricsCombining in-situ simulation and live HEMS mission facilitator observation: a flexible learning concept
2021, BMC Medical Education