Abstract
Objective To assess the extent to which demographic characteristics are related to international medical graduate (IMG) candidate performance on the Centre for the Evaluation of Health Professionals Educated Abroad General Comprehensive Clinical Examination 1 (CE1).
Design Retrospective study.
Setting Toronto, Ont.
Participants All IMG candidates who registered for and took the CE1 in 2007 (n = 430), 2008 (n = 480), and 2009 (n = 472) were included in this analysis. All candidates completed the Centre for the Evaluation of Health Professionals Educated Abroad CE1, a 12-station objective structured clinical examination.
Main outcome measures Mean (SD) examination scores for groups based on demographic variables (age, region of medical training, and Medical Council of Canada Qualifying Examination Part 1 [MCCQE1] score) were calculated. Analysis of variance was done using CE1 examination total scores as the dependent variables.
Results Candidates from countries where both medical education and patient care are conducted in English and those from South America and Western Europe achieved the highest scores, while candidates from the Western Pacific region and Africa achieved the lowest scores. Younger candidates achieved higher scores than older candidates. These results were consistent across the 3 years of CE1 examination administration. There was a significant relationship between MCCQE1 and CE1 scores in 2 of the 3 years: 2007 (r = 0.218, P < .001) and 2008 (r = 0.23, P < .01).
Conclusion The CE1 includes an assessment of communication skills; hence it is reasonable that candidates with stronger English skills have the highest scores on the CE1. Age, as a proxy for time since graduation, also has a substantial effect on examination scores, possibly owing to those further from their training lacking some currency of knowledge or being in focused rather than general practices. It is reasonable that those who had higher scores on the written test (the MCCQE1) would also have higher scores on the clinical test (the CE1). Demographic characteristics appear to be related to performance on the CE1.
International medical graduates (IMGs) make up approximately one-quarter to one-third of the Canadian1 and American2 physician work force. Ontario has a long history of assessing and integrating IMGs through various programs funded by the Ministry of Health and Long-Term Care, including the Ontario Pre-Internship Program in 1986, which lasted for approximately 15 years; the Ontario International Medical Graduate Program, which ran from 1999 to 2003; the Assessment Program for International Medical Graduates, which ran from 2003 to 2004; and International Medical Graduates—Ontario, which ran from 2003 to 2007.
The number of postgraduate positions funded by the Ministry of Health and Long-Term Care for IMGs residing in Ontario has increased steadily from 271 in 2003 to 630 in 2008.3,4 The number of first-year residency positions available for IMGs through the Canadian Resident Matching Service has also increased.5 Despite these increases, it is estimated that between 2000 and 4000 unlicensed IMGs reside in Ontario.6,7 In response to the increasing demand for assessment of IMGs, the Centre for the Evaluation of Health Professionals Educated Abroad (CEHPEA) was launched in April 2007 with a mandate to assess IMGs who are applying to enter training or practice in Ontario. Most IMGs apply to enter a family medicine residency. An important assessment tool used by the CEHPEA is the General Comprehensive Clinical Examination 1 (CE1), an objective structured clinical examination. The approach of assessment through an examination is reasonable, as other programs have demonstrated that scores on examinations are related to performance in residency.8
The CE1 consists of 12 stations: medicine (3 stations), surgery (2 stations), pediatrics (2 stations), obstetrics and gynecology (2 stations), psychiatry (2 stations), and ethics and communication (1 station). Stations include a 7-minute patient encounter and 3 minutes of oral questions. This examination is administered to approximately 500 candidates each year. The purpose of the examination is to assign candidates to one of several levels of competence: primary care–ready, second-year residency level and above, first-year residency level, clerkship, and unsatisfactory. Previous work demonstrated the reliability and validity of the CE1 to accurately assign IMGs to clerkship or first-year residency levels of competence.9 The purpose of this study was to assess the extent to which demographic factors (eg, region of undergraduate medical education, region of postgraduate medical education, age, and Medical Council of Canada Qualifying Examination Part 1 [MCCQE1] score) were related to IMG candidate performance on the CE1.
METHODS
Subjects
Results from 3 annual assessment cycles were analyzed. All candidates in 2007 (n = 430), 2008 (n = 480), and 2009 (n = 472) were included in this analysis. As candidates who are not successful in obtaining residency positions have the option of retaking the examination in subsequent years, the candidate pools were not independent samples. The registration process does not provide unique identifiers to applicants, so it is not possible to track candidates from year to year. For this reason, each year was analyzed separately. On registration, candidates provided date of birth and the countries where they received medical training (both undergraduate and postgraduate medical education). For the purpose of this study, countries of training were grouped into geographic regions similar to the World Health Organization regions. For example, “the Americas” were divided into North America, South America, and the Caribbean. Europe was split into Eastern Europe and Western Europe. An overseas English-language region was defined that excluded North America; it included Australia (AU), New Zealand (NZ), Ireland (IRL), the United Kingdom (UK), and South Africa (SA). Scores on the MCCQE1 for each candidate were also collected.
Scoring system of the CE1
Candidate performance was rated using a 5-level rating scale with anchors related to level of training: unsatisfactory (1), clerkship (2), first- year residency (3), second -year residency or higher (4), and primary care–ready (5). Ratings were provided for 7 domains during the patient encounter and 4 domains in the postencounter examiner questions. To arrive at station scores, the average rating of the patient encounters and the average rating of the answers to the oral questions were assigned equal weight and averaged. A candidate's total test score was the mean of his or her respective 12 station scores.
Analysis
Analysis of variance (ANOVA) was done using CE1 total scores as the dependent variable, and region of undergraduate medical education, region of postgraduate medical education, and age (grouped into 5-year intervals) as the independent variables. In addition, a Duncan post hoc analysis was carried out, with region of undergraduate medical education and age interval as the dependent variables. Correlation of the MCCQE1 and CE1 scores was also examined. The same analysis was replicated for each of the 3 years— 2007, 2008, and 2009.
RESULTS
Geographic region
The geographic regions of undergraduate medical education of candidates are presented in Table 1. More than 80% of candidates obtained their undergraduate medical education from Eastern Europe, the Eastern Mediterranean region, and Southeast Asia. Candidates whose undergraduate medical education was in Africa, the Eastern Mediterranean region, the Western Pacific region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, Southeast Asia, Western Europe, and the overseas English-speaking regions (AU, NZ, SA, the UK, and IRL) (Table 2). Post hoc tests indicated that candidates educated in AU, NZ, SA, the UK, and IRL, South America, and Western Europe achieved the highest scores, while those from the Western Pacific region and Africa achieved the lowest scores. Locations of postgraduate medical education are presented in Table 3. Almost 70% of candidates obtained their postgraduate medical education from Eastern Europe, the Eastern Mediterranean region, and Southeast Asia. Not all applicants provided complete records on where they undertook their postgraduate medical education.
At the postgraduate level, candidates whose training was in AU, NZ, SA, the UK, and IRL, and South America achieved the highest scores (Table 4). There was little consistency among the lowest scoring regions.
Age
The distribution of candidates' dates of birth is presented in Table 5. Candidate age was related to performance on the CE1. Post hoc tests indicated that younger candidates (ie, the more recent graduates) achieved higher scores than older candidates did, and a significant result was achieved on ANOVA (Table 6).
Correlation between MCCQE1 and CE1 scores
There was a significant but relatively weak correlation between MCCQE1 and CE1 scores in 2007 (r = 0.218, P < .001) and 2008 (r = 0.23, P < .01); however, in 2009, the correlation was not significant (r = 0.076, P > .05).
DISCUSSION
This study examined the relationship between demographic factors (eg, region of undergraduate medical education, region of postgraduate medical education, and age) and scores of IMG candidates on an examination used to assess candidates applying to enter residency programs or practice in Ontario (the CE1). It also examined the relationship between MCCQE1 scores and CE1 scores. The same analysis was replicated over 3 administration years: 2007, 2008, and 2009.
The results generally indicated that candidates whose medical education was in English-speaking Commonwealth and former Commonwealth countries (AU, NZ, the UK, and SA) and IRL, the Caribbean, and South America achieved higher test scores. Results demonstrated that across the 3 years studied, significant differences existed in the test scores of candidates in different age intervals, with younger candidates achieving higher test scores (P < .05). Finally, there was a positive correlation between the MCCQE1 and the CE1 test scores in 2 of the 3 years.
Consistency was observed in the mean CE1 test scores, the test reliability, the relationship of age with the test scores, and the relationship of location of undergraduate medical education with the test scores. In 2 of the 3 years the relationship between the CE1 scores and the MCCQE1 scores was similar. The consistency of the CE1 results during the 3 years strengthens the confidence in the results. Communication skills play a part in any patient interaction. The CE1 includes an assessment of communication skills, including ratings for verbal communication; nonverbal communication; response to patient's feelings, needs, and values; and management of a standardized patient and case. The latter includes explaining the rationale for the candidate's recommendations to the patient, obtaining consent, and educating or counseling the patient. All of these ratings would be affected by a candidate's language skills. It is not surprising, then, that candidates from regions where both medical education and patient care are conducted in English, and the primary language spoken is English, have the highest scores on the CE1.
Longer time since graduation can also have a considerable effect on examination scores. There are 3 likely reasons for this: 1) candidate knowledge is not up to date; 2) candidates might have focused their practices narrowly and can no longer respond to general-knowledge scenarios; and 3) candidates might have been out of clinical practice.
Medical knowledge evolves over time and treatment approaches change with advances in knowledge. New knowledge is disseminated to students through their education programs and to practitioners through various continuing competence activities. As individuals approach continuing competence in a variety of ways, their integration of newer approaches and knowledge will vary. On the other hand, medical education programs will be teaching the up-to-date approaches and integrating new knowledge on an ongoing basis. Students will be exposed to the newer information.
While year of graduation was not included in this analysis, age is a proxy for year of graduation. Older candidates are generally further from their undergraduate programs than younger candidates are. This study shows that those who were younger than 38 years of age, and therefore closer to their academic experiences, achieved significantly higher scores than older candidates (P < .05). This is similar to the results found by the Education Commission for Foreign Medical Graduates in the United States, which reported that those who had graduated less than 5 years previously passed their clinical skills assessment at a higher rate than those who had graduated 5 or more years previously.10 It is also supported by the observation that younger IMG residents in an internal medicine residency program had significantly higher competency-based evaluation scores than the median score for the program (P < .05).11
The second factor related to time since graduation, or age, is the clinical focus of the examination. The CE1 is a generalist-based examination, set at the level of exiting medical school and entering residency. The blueprint includes internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and general surgery. Older candidates, even those registered as generalists, might have focused their practices in specific clinical content areas. They might not have any recent experience in one or more of these clinical domains, and as a result might not perform well in those domains, resulting in lower examination scores.
Finally, it is likely that those who have been out of clinical practice for longer periods will have lost some of their knowledge and competencies. The longer they have been out of practice, the more likely this is to be a factor in lower examination scores. As residency status in Canada is not a factor in taking the examination, some candidates will already be living in Canada and unable to work as physicians, while others will be coming directly from their home countries to take the examination. In the latter case, they are more likely to either be just graduating from their undergraduate program or be still working clinically in their home countries. Those currently residing in Canada will necessarily be working in some other field. Length of time out of medical practice could negatively affect the examination score. This is consistent with a study that showed that recent clinical experience (within 2 years of an internal medicine residency) was significantly related to subsequent performance in the residency program (P = .005).12
In examining the relationship between the CE1 and the MCCQE1 scores, it is important to consider that the CE1 is a clinical test involving interactions with standardized patients, while the MCCQE1 is a written test. The different test formats present different challenges and likely assess different knowledge and competencies. This study found a weak but significant correlation between these 2 examination scores, which is similar to results reported by other studies.13 Limitations include that this study was conducted in the one province that holds this examination, so the results might not be generalizable to other provinces and their particular assessment processes. Additionally, the examination is not mandatory for entry to residency training, so the applicants might not be completely reflective of the whole pool of IMGs applying for residency positions.
A follow-up study might investigate the relationships between test scores and which candidates are accepted into residency, test scores and the results of in-training evaluation reports, and finally, test scores and success in licensing examinations.
Conclusion
Candidates who trained in Africa, the Eastern Mediterranean region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, the Western Pacific region, and AU, NZ, SA, the UK, and IRL. Younger candidates (date of birth 1970 and later) achieved higher scores than candidates born before 1970 did. There was a weak but significant association between candidates' MCCQE1 and CE1 scores in 2 of the 3 years, indicating that the performance-based CE1 is likely measuring a different construct than the written examination is. Finally, these results were consistent across 3 years of examination administration. Therefore, it would appear that demographic characteristics have an effect on performance on the CE1.
Notes
EDITOR'S KEY POINTS
-
Candidates whose undergraduate medical education took place in Africa, the Eastern Mediterranean region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, the Western Pacific region, and English-speaking overseas countries. Younger candidates (date of birth 1970 and later) achieved higher scores than candidates born before 1970.
-
There was a weak but significant association between candidates' Medical Council of Canada Qualifying Examination Part 1 score and General Comprehensive Clinical Examination 1 scores in 2 of the 3 years, indicating that the performance-based General Comprehensive Clinical Examination 1 is likely measuring a different construct than the written examination is.
-
These results were consistent across 3 years of examination administration.
POINTS DE REPÈRE DU RÉDACTEUR
-
Les candidats qui ont eu leur formation médicale de premier cycle en Afrique, dans la région orientale de la Méditerranée et en Europe de l'Est obtiennent des scores plus bas que ceux qui ont fait leur formation en Amérique du Sud, dans les Caraïbes, dans la région du Pacifique occidental et dans les pays étrangers de langue anglaise. Les candidats plus jeunes (nés en 1970 ou plus tard) ont obtenu des meilleurs résultats que ceux nés avant 1970.
-
Il y avait une association faible mais significative entre le score des candidats à l'examen d'aptitude du Conseil Médical du Canada, partie 1, et celui obtenu au General Comprehensive Clinical Examination 1 (CE1), et ce, pour 2 des 3 années à l'étude, ce qui indique que les éléments évalués par le CE1 pourraient être différents de ceux de l'examen écrit.
-
Ces résultats étaient semblables pour les 3 années où les résultats de l'examen ont été étudiés.
Footnotes
-
Competing interests: None declared
-
This article has been peer reviewed. Can Fam Physician 2013;59:170-6
- Copyright© the College of Family Physicians of Canada
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.