We report on a process for assessing the communication skills of emergency medicine residents that includes 1) a faculty development initiative; 2) videotaping of actual resident-patient encounters in the emergency department; and 3) creation of an observation instrument for evaluating communication behaviors. We tested this observation instrument for inter-rater reliability, finding moderate-to-high agreement for only 11 of 32 items. These related to personal introductions, conflict management, nonverbal communication, and overall performance. There was poor or no agreement for behaviors related to establishing rapport, gathering information, and contracting or informing. Challenges of assessing interpersonal skills of emergency medicine residents are discussed.