Assessing students and residents

How to render the invisible visible?

The concern with the assessment process is that learning is, alas, invisible. We could never accurately visualize student and resident learning. Therefore, the challenge is to implement a system that renders learning visible, allowing us to measure or observe it to ensure the validity and fidelity of the inferences. Laurier et al⁵ place assessment within a dynamic of change; learning represents the result located between a final situation and an initial situation. Within a competency-based framework, it is possible to implement a number of strategies that allow us to infer the competency being assessed based on a set of performances. The key is to strategically choose performances, such as answering questions on an examination, discussing case studies, or conducting physical examinations, that would allow evaluators to make informed decisions on the quality of the progression of learning.

Keep the assessment contract in mind

The concept of the assessment contract is borrowed from the didactic contract developed by Brousseau.⁶ Generally speaking, learners and teachers negotiate, more or less explicitly, the elements of the assessment. This can be illustrated by the propensity of some learners, knowing they are undergoing a summative assessment, to adopt behaviour that emphasizes the best aspect of their competencies while masking the less favourable aspects. Another classic example is the fact that learners will study and primarily focus on learning that is assessed, to the detriment of learning that is not. All learning is important, but that which is assessed and could impact their progression is considered more important!

Assessing means respecting a rigorous process

For many people, assessments are reduced to a mark (eg, 90%) or a grade (eg, pass). These aspects are important because they are effectively a part of the assessment approach and represent the most visible component of it. Yet, to arrive at a decision, assessing physicians must implement an approach that is formal or informal, instrumented or not, and planned or not. In all cases, the assessment will consist of the following dimensions: planning, information gathering, the judgment and the decision, and finally, communication of the result. If the approach is formal, instrumented, and planned, as is often the case in preclerkships, the dimensions will be easily observable. Learners will be informed of the examination sessions (planning). Following the administration of the examination, teachers will grade the examination, or it will be graded automatically (information gathering). Once information is revealed through the grading process, a judgment will be made followed by a decision (eg, learners who have not passed the threshold score must retake the examination). Finally, there is the communication of the decision (eg, communicating the grade to the learner or informing them of the need to retake the examination).

The situation I have just described is characteristic of a summative approach. Yet, it is also possible to identify these dimensions when an assessment is informal and not instrumented or planned, as is often the case with formative assessments. A teacher responding to a question from a learner may use these same dimensions but in a quasi-simultaneous manner. Planning would therefore be minimal, even if competent and experienced teachers would tell us that the questions are often predictable. Teachers would collect data from the answers while judging the value of the question. They would exercise their expert judgment and make a decision (respond with another question, connect the answer with a previously taught concept, etc), and, finally, communicate it to the learner. We could compare these 2 methods to type 1 and type 2 clinical reasoning.⁷ Type 1 reasoning (rapid and intuitive) relates to the formative context, while type 2 reasoning (logic and analysis) relates to the summative context.

Accept that assessments will never be completely objective

An assessment, whether formative or summative, is first and foremost what we saw in the preceding point: a judgment. By definition, a judgment inherently possesses a subjective component. One might incorrectly think that multiple-choice questions are objective. However, making most of the questions about a small fraction of the material taught is enough to render the examination biased and not reflective of learners’ true progress. The grading process is the only aspect of multiple-choice questions that is objective; the questions are not. In other contexts where observation is required, as in objective structured clinical examinations, several researchers have become interested in conditions that legitimately reduce subjectivity.⁸ We must accept the fact that assessments are an imperfect human activity. That said, we must also work tirelessly to eliminate obstacles potentially affecting the validity of assessments.

To measure or to observe, that is the question

While assessing learning, we can rely on 2 types of information: measurement and observation. Measurement most frequently refers to marks obtained, for example, on multiple-choice questions. The number of “correct responses” serves as a “measurement”: the higher the number of correct responses, the better the learning. This correlation is not always true, but it is the essence of measuring learning. In preclerkships, learning progress is generally measured. During clerkships and residencies, observations are more heavily relied upon, which requires recording tools such as field notes and criteria grids based on the CanMEDS framework.⁹ The nature of the information collected, therefore, depends on its context. It is advisable to identify attributes of this context that allow one to develop or rely upon tools adapted to a specific situation.