Elsevier

The Journal of Emergency Medicine

Volume 17, Issue 2, March–April 1999, Pages 355-361
The Journal of Emergency Medicine

Education
Assessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability

https://doi.org/10.1016/S0736-4679(98)00181-4Get rights and content

Abstract

We report on a process for assessing the communication skills of emergency medicine residents that includes 1) a faculty development initiative; 2) videotaping of actual resident–patient encounters in the emergency department; and 3) creation of an observation instrument for evaluating communication behaviors. We tested this observation instrument for inter-rater reliability, finding moderate-to-high agreement for only 11 of 32 items. These related to personal introductions, conflict management, nonverbal communication, and overall performance. There was poor or no agreement for behaviors related to establishing rapport, gathering information, and contracting or informing. Challenges of assessing interpersonal skills of emergency medicine residents are discussed.

Introduction

Emergency medicine (EM) residents must develop effective communication skills to take accurate and efficient medical histories, enhance patient compliance and satisfaction with treatment, optimize clinical outcomes, provide emotional comfort, assure informed consent, and reduce the risk of litigation. Educators, therefore, need a reliable method for evaluating how well residents communicate with patients.

We describe our experience in developing a process for assessing residents’ communication skills. We hypothesized that the reliability of this process would be enhanced through: 1) a faculty development initiative that increased knowledge about physician-patient communication in the emergency department (ED); 2) videotaping of actual resident-patient encounters in the ED; and 3) development of a list of key communication behaviors relevant to the emergency department encounter. Through this process, we created an observation instrument for assessing communication skills that we tested for inter-rater reliability.

Section snippets

Population and setting

The study was conducted within an EM residency at an urban teaching hospital with an annual ED census of 52,000 visits. The protocol was approved by the Institutional Review Board for Human Experimentation.

A video camera was installed in one of 13 ED examination rooms used for patients whose illness fell within a middle range of acuity. These are patients who require urgent attention but do not require resuscitation. The camera was mounted above and behind the patient, facing the resident. It

Results

A convenience sample of 50 patients gave consent for videotaping between December 1992 and May 1993. Thirty-nine tapes were excluded because the camera had not been activated in time to capture the beginning of the encounter. Ten residents were recorded on the remaining 11 videotapes. Age, sex, diagnosis, and disposition of the entered patients are listed in Table 1.

Data are presented in Table 2 on agreement among raters regarding whether or not an individual behavior was performed. Agreement

Rationale of assessment process

Most EM residencies rely upon random and informal observations of trainees to evaluate communication skills (4). However, the adequacy and reliability of bedside assessment of clinical skills has been brought into question 5, 6, 7, 8. To be reliable, evaluations must be based upon multiple observations and should involve multiple observers 9, 10.

The assessment process developed in this study preserved features typical of the bedside method used in residency programs: independent raters observed

Conclusions

We describe a process to assess EM resident communication skills that incorporates videotaping of actual patient encounters, faculty development through literature review and group discussion of videotapes, and the creation of an observation instrument. We then prospectively applied this instrument and tested its reliability. Inter-rater reliability was high for behaviors related to personal introductions, conflict management, non-verbal behavior, and overall performance. There was no agreement

Acknowledgements

This study was supported in part through the “Innovation in Medical Education Award,” Emergency Medicine Foundation/Society for Academic Emergency Medicine, 1992–1993.

References (16)

  • S Rosenzweig

    Teaching the art of emergency medicine

    Ann Emerg Med

    (1991)
  • D.H Novack et al.

    Teaching medical interviewing: a basic course on interviewing and the physician-patient relationship

    Arch Intern Med

    (1992)
  • A.B Ovadia et al.

    Assessment of medical interview skills

  • J.L Fleiss

    Statistical methods for rates and proportions, 2nd edn

    (1981)
  • J.E Herbers et al.

    How accurate are faculty evaluations of clinical competence?

    J Gen Intern Med

    (1989)
  • G.L Noel et al.

    How well do internal medicine faculty members evaluate the clinical skills of residents?

    Ann Intern Med

    (1992)
  • Consensus statement of the researchers in clinical skills assessment (RCSA) on the use of standardized patients to...
  • W.P Burdick et al.

    Observation of emergency medicine residents at the bedside: how often does it happen?

    Acad Emerg Med

    (1995)
There are more references available in the full text version of this article.

Cited by (14)

  • Self-reported use of communication techniques in the emergency department

    2012, Journal of Emergency Medicine
    Citation Excerpt :

    Consequently, many patients leave the ED without an understanding of their diagnosis, follow-up plan, or how to care for themselves at home (14,15). Several studies have begun to describe specific aspects of communication in the ED, including global patient assessments of the quality of communication, content analyses of audiorecorded conversations, and codifying the verbal utterances in the encounter for elements, such as rapport building and patient centeredness (16–21). These studies highlight important aspects of the verbal communication that occurs in the ED; however, to our knowledge, no studies have quantified the use of different communication techniques in the ED environment.

  • Conflict Resolution in Emergency Medicine

    2012, Emergency Medicine: Clinical Essentials, SECOND EDITION
  • "Sign right here and you're good to go": A content analysis of audiotaped emergency department discharge instructions

    2011, Annals of Emergency Medicine
    Citation Excerpt :

    Instead, individual residency programs are left to develop their own methods. A survey of ED program directors found that nearly all (98%) of the program directors agreed that communication skills should be taught.38 When asked about bedside instruction/evaluation of residents' interpersonal skills, program directors reported mean supervision rates of 14.7% for giving discharge instructions.

View all citing articles on Scopus
View full text