Perennial post-examination surprises

History of SOO examinations

Before the introduction of the SOO in 1969, assessment of clinical competency was based on a combination of written tests and formal oral examinations.⁴ Formal oral examinations, the gold standard of the day, required learners to conduct an observed or unobserved patient assessment, followed by an examiner’s review of their findings and an assessment of their clinical reasoning skills.⁵

Oral examinations prior to the SOO had problems, including the unintended, inconsistent way in which the “patient” could present to the student compared with to the examiner.⁵ However, the SOO’s use of a “programmed” patient—an actor able to realistically and reproducibly portray a patient in both healthy and diseased states—proved useful in addressing these concerns.⁵ This approach also made the CFPC the first national certification body to use standardized patients in its certification exam.^5-7

Continuous quality improvement and the patient-centred clinical method

Between 1969 and 1986 the SOO underwent many changes.⁸ While the original SOO involved 3 participants (an examiner, a programmed patient, and the candidate), by 1986 the roles of patient and examiner had been combined, based on the belief it would improve the authenticity and accuracy of assessment decisions.⁸ The number of SOO cases per examination had increased to 5, and the time allowed to complete each SOO had increased to 15 minutes with a 3-minute warning.⁸ The SOO scoring rubric was based on 8 objectives, chosen to be consistent with the CFPC’s newly developed assessment guide.⁸ Esoteric in nature, the rubric lacked the clarity that would later be seen.

Before 1986, assessment of family medicine residents followed a generic approach, applying many of the same techniques as other medical disciplines. However, an article published in 1984 by Levenstein, describing patient-centred general practice consultations and emphasizing the unique interaction between general practitioners and patients, provided the discipline of family medicine with a much-needed framework.⁹ Key to the discipline and to the framework was the importance of addressing patients’ illness experiences. The model would come to be known as the patient-centred clinical method.^10-12

Subsequently, researchers from the University of Western Ontario (now Western University) in London, Ont—Drs Judith Belle Brown, Moira Stewart, and Wayne Weston—would work with the CFPC to develop a new rubric for SOOs.¹³ Based on the patient-centred clinical method, the new rubric allowed the CFPC to assess a candidate’s ability to elicit a patient’s illness experience, elicit the impact of the illness on the patient’s social environment, and find common ground with the patient while attending to the patient’s health concern.¹³

Dual process theory

Our understanding of clinical reasoning is multifaceted.¹⁴ While no single theory of learning completely explains all observations seen in medical education, information processing theories are best suited to the SOO.¹⁵ The most common of these models is the dual process theory,¹⁶ which holds that 2 cognitive processes are used to reason: system 1 and system 2.

Referred to as the “slow” process, system 2 is based on the hypothetico-deductive model of clinical reasoning (ie, the methodologic approach by which a physician generates and tests a hypothesis against data deliberately obtained during a clinical encounter).¹⁶ On its own, the hypothetico-deductive model was found to be too generalized, with novices and experts equally likely to use the approach.¹⁷

Conversely, the system 1 process is theorized to be “fast,” intuitive, and occurring almost unconsciously.^16,17 Several theories account for this process, including pattern recognition and encapsulation of clinical knowledge into illness scripts.¹⁸ Repeated exposure of a physician to similar clinical problems, along with successful outcomes, builds that physician’s facility in solving such problems. In essence, that physician has experienced a form of repeated, deliberate practice.^19,20 When the system 2 process is used in conjunction with coaching, physician learners incrementally sort their clinical understanding of diseases into illness scripts, producing their own “database” of exemplars that promotes an increased level of knowledge organization and theoretically allows for quicker retrieval of this encapsulated knowledge.^18,21,22 In the end, this quicker retrieval of information is postulated to lie at the heart of the post-examination surprise.

Building an argument

The ability to reason clinically is central to the practice of all health care professions and is an important component of any final assessment of competence.²² In medical education, the dual process theory conceptualizes the process of clinical reasoning and can be visualized using the unified model proposed by Croskerry.¹⁴

Croskerry’s model distinguishes between expert and novice clinicians by allowing age and experience to act as determinants for dominant use of the system 1 process.¹⁴ Stated differently, if a clinician were to recognize a pattern as a result of age or experience, that clinician would default to using the system 1 process and subsequently require less time to arrive at a diagnosis.^23,24 This facility might manifest itself as a need to ask fewer questions or to conduct fewer elements of a physical examination before arriving at the correct diagnosis.^25,26

In 1 US study, standardized patients visited family medicine residents in their clinics during their second and third years of training.²⁵ Using a checklist similar to that of an objective structured clinical examination (OSCE), standardized patients evaluated residents’ performances at visits in both years using the same cases. While there was no difference in diagnostic accuracy from year to year, total examination scores generated by residents were higher in their second postgraduate year than in their third postgraduate year. The authors proposed that with increased experience, residents needed fewer questions and fewer physical examination maneuvers to arrive at correct final diagnoses, resulting in lower OSCE scores and possibly demonstrating the phenomenon known as the intermediate effect.

Intermediate effect revisited

Considered a paradox, the intermediate effect is the finding that clinicians at an intermediate level (ie, less experienced than experts but more experienced than novice medical students) will do better than more experienced clinicians on an examination. While the phenomenon was first described with written tests, similar findings have been reported for clinical performance examinations, including OSCEs.²⁵ Given the similarities between SOOs and OSCEs in their use of checklists, I contend that the SOO rubric might exhibit the same inability to reward clinical proficiency seen in OSCE-type examinations. If so, how might we mitigate this outcome?

Intermediate effect–resistant testing

In 2000 Charlin et al developed a new assessment tool called the Script Concordance test.²⁷ While its details are beyond the scope of this article, the test is an example of a written examination that does not demonstrate the intermediate effect. Its developers cite several reasons for this, including the theoretical foundation on which Script Concordance testing is built—namely, the developmental theory of clinical competence.

However, a careful review of this work provides another cause. A Script Concordance test is standardized based on answers generated by a panel of expert (experienced) clinicians. The scoring rubrics thus take into account the varied and nuanced ways in which medicine is practised. Parallels can be drawn between this process and the concept of piloting examination questions. To this end, what effect would piloting SOOs with a number of experienced clinicians have, if their responses could guide the development of checklists and rubrics? Alternatively, is there value in conducting a post hoc analysis of examination data? While item analysis is commonplace for written examinations, its use for clinical performance examinations is not; the discouraging influence of work by Swanson and van der Vleuten likely plays an important role here.²⁸ However, some researchers are beginning to explore the use of item analysis with both OSCE stations and individual OSCE checklist items as units of analysis.^29,30 In each case, the reliability coefficient of the examination improved when the statistical analysis was used to modify the examination rubric post hoc.