Skip to main content

Main menu

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums

User menu

  • My alerts

Search

  • Advanced search
The College of Family Physicians of Canada
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums
  • My alerts
The College of Family Physicians of Canada

Advanced Search

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • RSS feeds
  • Follow cfp Template on Twitter
Research ArticleResearch

IMG candidates' demographic characteristics as predictors of CEHPEA CE1 results

Marla Nayer and Arthur Rothman
Canadian Family Physician February 2013, 59 (2) 170-176;
Marla Nayer
Dr Nayer worked at the Centre for the Evaluation of Health Professionals Educated Abroad at the time this work was completed, and is Assistant Professor in the Department of Physical Therapy at the University of Toronto in Ontario. Dr Rothman is Professor Emeritus in the Faculty of Medicine at the University of Toronto
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: marlanayer@rogers.com
Arthur Rothman
Dr Nayer worked at the Centre for the Evaluation of Health Professionals Educated Abroad at the time this work was completed, and is Assistant Professor in the Department of Physical Therapy at the University of Toronto in Ontario. Dr Rothman is Professor Emeritus in the Faculty of Medicine at the University of Toronto
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF
Loading

Abstract

Objective To assess the extent to which demographic characteristics are related to international medical graduate (IMG) candidate performance on the Centre for the Evaluation of Health Professionals Educated Abroad General Comprehensive Clinical Examination 1 (CE1).

Design Retrospective study.

Setting Toronto, Ont.

Participants All IMG candidates who registered for and took the CE1 in 2007 (n = 430), 2008 (n = 480), and 2009 (n = 472) were included in this analysis. All candidates completed the Centre for the Evaluation of Health Professionals Educated Abroad CE1, a 12-station objective structured clinical examination.

Main outcome measures Mean (SD) examination scores for groups based on demographic variables (age, region of medical training, and Medical Council of Canada Qualifying Examination Part 1 [MCCQE1] score) were calculated. Analysis of variance was done using CE1 examination total scores as the dependent variables.

Results Candidates from countries where both medical education and patient care are conducted in English and those from South America and Western Europe achieved the highest scores, while candidates from the Western Pacific region and Africa achieved the lowest scores. Younger candidates achieved higher scores than older candidates. These results were consistent across the 3 years of CE1 examination administration. There was a significant relationship between MCCQE1 and CE1 scores in 2 of the 3 years: 2007 (r = 0.218, P < .001) and 2008 (r = 0.23, P < .01).

Conclusion The CE1 includes an assessment of communication skills; hence it is reasonable that candidates with stronger English skills have the highest scores on the CE1. Age, as a proxy for time since graduation, also has a substantial effect on examination scores, possibly owing to those further from their training lacking some currency of knowledge or being in focused rather than general practices. It is reasonable that those who had higher scores on the written test (the MCCQE1) would also have higher scores on the clinical test (the CE1). Demographic characteristics appear to be related to performance on the CE1.

International medical graduates (IMGs) make up approximately one-quarter to one-third of the Canadian1 and American2 physician work force. Ontario has a long history of assessing and integrating IMGs through various programs funded by the Ministry of Health and Long-Term Care, including the Ontario Pre-Internship Program in 1986, which lasted for approximately 15 years; the Ontario International Medical Graduate Program, which ran from 1999 to 2003; the Assessment Program for International Medical Graduates, which ran from 2003 to 2004; and International Medical Graduates—Ontario, which ran from 2003 to 2007.

The number of postgraduate positions funded by the Ministry of Health and Long-Term Care for IMGs residing in Ontario has increased steadily from 271 in 2003 to 630 in 2008.3,4 The number of first-year residency positions available for IMGs through the Canadian Resident Matching Service has also increased.5 Despite these increases, it is estimated that between 2000 and 4000 unlicensed IMGs reside in Ontario.6,7 In response to the increasing demand for assessment of IMGs, the Centre for the Evaluation of Health Professionals Educated Abroad (CEHPEA) was launched in April 2007 with a mandate to assess IMGs who are applying to enter training or practice in Ontario. Most IMGs apply to enter a family medicine residency. An important assessment tool used by the CEHPEA is the General Comprehensive Clinical Examination 1 (CE1), an objective structured clinical examination. The approach of assessment through an examination is reasonable, as other programs have demonstrated that scores on examinations are related to performance in residency.8

The CE1 consists of 12 stations: medicine (3 stations), surgery (2 stations), pediatrics (2 stations), obstetrics and gynecology (2 stations), psychiatry (2 stations), and ethics and communication (1 station). Stations include a 7-minute patient encounter and 3 minutes of oral questions. This examination is administered to approximately 500 candidates each year. The purpose of the examination is to assign candidates to one of several levels of competence: primary care–ready, second-year residency level and above, first-year residency level, clerkship, and unsatisfactory. Previous work demonstrated the reliability and validity of the CE1 to accurately assign IMGs to clerkship or first-year residency levels of competence.9 The purpose of this study was to assess the extent to which demographic factors (eg, region of undergraduate medical education, region of postgraduate medical education, age, and Medical Council of Canada Qualifying Examination Part 1 [MCCQE1] score) were related to IMG candidate performance on the CE1.

METHODS

Subjects

Results from 3 annual assessment cycles were analyzed. All candidates in 2007 (n = 430), 2008 (n = 480), and 2009 (n = 472) were included in this analysis. As candidates who are not successful in obtaining residency positions have the option of retaking the examination in subsequent years, the candidate pools were not independent samples. The registration process does not provide unique identifiers to applicants, so it is not possible to track candidates from year to year. For this reason, each year was analyzed separately. On registration, candidates provided date of birth and the countries where they received medical training (both undergraduate and postgraduate medical education). For the purpose of this study, countries of training were grouped into geographic regions similar to the World Health Organization regions. For example, “the Americas” were divided into North America, South America, and the Caribbean. Europe was split into Eastern Europe and Western Europe. An overseas English-language region was defined that excluded North America; it included Australia (AU), New Zealand (NZ), Ireland (IRL), the United Kingdom (UK), and South Africa (SA). Scores on the MCCQE1 for each candidate were also collected.

Scoring system of the CE1

Candidate performance was rated using a 5-level rating scale with anchors related to level of training: unsatisfactory (1), clerkship (2), first- year residency (3), second -year residency or higher (4), and primary care–ready (5). Ratings were provided for 7 domains during the patient encounter and 4 domains in the postencounter examiner questions. To arrive at station scores, the average rating of the patient encounters and the average rating of the answers to the oral questions were assigned equal weight and averaged. A candidate's total test score was the mean of his or her respective 12 station scores.

Analysis

Analysis of variance (ANOVA) was done using CE1 total scores as the dependent variable, and region of undergraduate medical education, region of postgraduate medical education, and age (grouped into 5-year intervals) as the independent variables. In addition, a Duncan post hoc analysis was carried out, with region of undergraduate medical education and age interval as the dependent variables. Correlation of the MCCQE1 and CE1 scores was also examined. The same analysis was replicated for each of the 3 years— 2007, 2008, and 2009.

RESULTS

Geographic region

The geographic regions of undergraduate medical education of candidates are presented in Table 1. More than 80% of candidates obtained their undergraduate medical education from Eastern Europe, the Eastern Mediterranean region, and Southeast Asia. Candidates whose undergraduate medical education was in Africa, the Eastern Mediterranean region, the Western Pacific region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, Southeast Asia, Western Europe, and the overseas English-speaking regions (AU, NZ, SA, the UK, and IRL) (Table 2). Post hoc tests indicated that candidates educated in AU, NZ, SA, the UK, and IRL, South America, and Western Europe achieved the highest scores, while those from the Western Pacific region and Africa achieved the lowest scores. Locations of postgraduate medical education are presented in Table 3. Almost 70% of candidates obtained their postgraduate medical education from Eastern Europe, the Eastern Mediterranean region, and Southeast Asia. Not all applicants provided complete records on where they undertook their postgraduate medical education.

View this table:
  • View inline
  • View popup
Table 1 Geographic region of candidates' undergraduate medical education by year of CE1
View this table:
  • View inline
  • View popup
Table 2 Mean CE1 scores by year of testing and region of undergraduate medical education
View this table:
  • View inline
  • View popup
Table 3 Geographic region of candidates' postgraduate medical education by year of CE1 test

At the postgraduate level, candidates whose training was in AU, NZ, SA, the UK, and IRL, and South America achieved the highest scores (Table 4). There was little consistency among the lowest scoring regions.

View this table:
  • View inline
  • View popup
Table 4 Mean CE1 scores according to year of testing and region of postgraduate medical education

Age

The distribution of candidates' dates of birth is presented in Table 5. Candidate age was related to performance on the CE1. Post hoc tests indicated that younger candidates (ie, the more recent graduates) achieved higher scores than older candidates did, and a significant result was achieved on ANOVA (Table 6).

View this table:
  • View inline
  • View popup
Table 5 Candidates' dates of birth in 5-year intervals
View this table:
  • View inline
  • View popup
Table 6 Mean CE1 score by date of birth

Correlation between MCCQE1 and CE1 scores

There was a significant but relatively weak correlation between MCCQE1 and CE1 scores in 2007 (r = 0.218, P < .001) and 2008 (r = 0.23, P < .01); however, in 2009, the correlation was not significant (r = 0.076, P > .05).

DISCUSSION

This study examined the relationship between demographic factors (eg, region of undergraduate medical education, region of postgraduate medical education, and age) and scores of IMG candidates on an examination used to assess candidates applying to enter residency programs or practice in Ontario (the CE1). It also examined the relationship between MCCQE1 scores and CE1 scores. The same analysis was replicated over 3 administration years: 2007, 2008, and 2009.

The results generally indicated that candidates whose medical education was in English-speaking Commonwealth and former Commonwealth countries (AU, NZ, the UK, and SA) and IRL, the Caribbean, and South America achieved higher test scores. Results demonstrated that across the 3 years studied, significant differences existed in the test scores of candidates in different age intervals, with younger candidates achieving higher test scores (P < .05). Finally, there was a positive correlation between the MCCQE1 and the CE1 test scores in 2 of the 3 years.

Consistency was observed in the mean CE1 test scores, the test reliability, the relationship of age with the test scores, and the relationship of location of undergraduate medical education with the test scores. In 2 of the 3 years the relationship between the CE1 scores and the MCCQE1 scores was similar. The consistency of the CE1 results during the 3 years strengthens the confidence in the results. Communication skills play a part in any patient interaction. The CE1 includes an assessment of communication skills, including ratings for verbal communication; nonverbal communication; response to patient's feelings, needs, and values; and management of a standardized patient and case. The latter includes explaining the rationale for the candidate's recommendations to the patient, obtaining consent, and educating or counseling the patient. All of these ratings would be affected by a candidate's language skills. It is not surprising, then, that candidates from regions where both medical education and patient care are conducted in English, and the primary language spoken is English, have the highest scores on the CE1.

Longer time since graduation can also have a considerable effect on examination scores. There are 3 likely reasons for this: 1) candidate knowledge is not up to date; 2) candidates might have focused their practices narrowly and can no longer respond to general-knowledge scenarios; and 3) candidates might have been out of clinical practice.

Medical knowledge evolves over time and treatment approaches change with advances in knowledge. New knowledge is disseminated to students through their education programs and to practitioners through various continuing competence activities. As individuals approach continuing competence in a variety of ways, their integration of newer approaches and knowledge will vary. On the other hand, medical education programs will be teaching the up-to-date approaches and integrating new knowledge on an ongoing basis. Students will be exposed to the newer information.

While year of graduation was not included in this analysis, age is a proxy for year of graduation. Older candidates are generally further from their undergraduate programs than younger candidates are. This study shows that those who were younger than 38 years of age, and therefore closer to their academic experiences, achieved significantly higher scores than older candidates (P < .05). This is similar to the results found by the Education Commission for Foreign Medical Graduates in the United States, which reported that those who had graduated less than 5 years previously passed their clinical skills assessment at a higher rate than those who had graduated 5 or more years previously.10 It is also supported by the observation that younger IMG residents in an internal medicine residency program had significantly higher competency-based evaluation scores than the median score for the program (P < .05).11

The second factor related to time since graduation, or age, is the clinical focus of the examination. The CE1 is a generalist-based examination, set at the level of exiting medical school and entering residency. The blueprint includes internal medicine, obstetrics and gynecology, pediatrics, psychiatry, and general surgery. Older candidates, even those registered as generalists, might have focused their practices in specific clinical content areas. They might not have any recent experience in one or more of these clinical domains, and as a result might not perform well in those domains, resulting in lower examination scores.

Finally, it is likely that those who have been out of clinical practice for longer periods will have lost some of their knowledge and competencies. The longer they have been out of practice, the more likely this is to be a factor in lower examination scores. As residency status in Canada is not a factor in taking the examination, some candidates will already be living in Canada and unable to work as physicians, while others will be coming directly from their home countries to take the examination. In the latter case, they are more likely to either be just graduating from their undergraduate program or be still working clinically in their home countries. Those currently residing in Canada will necessarily be working in some other field. Length of time out of medical practice could negatively affect the examination score. This is consistent with a study that showed that recent clinical experience (within 2 years of an internal medicine residency) was significantly related to subsequent performance in the residency program (P = .005).12

In examining the relationship between the CE1 and the MCCQE1 scores, it is important to consider that the CE1 is a clinical test involving interactions with standardized patients, while the MCCQE1 is a written test. The different test formats present different challenges and likely assess different knowledge and competencies. This study found a weak but significant correlation between these 2 examination scores, which is similar to results reported by other studies.13 Limitations include that this study was conducted in the one province that holds this examination, so the results might not be generalizable to other provinces and their particular assessment processes. Additionally, the examination is not mandatory for entry to residency training, so the applicants might not be completely reflective of the whole pool of IMGs applying for residency positions.

A follow-up study might investigate the relationships between test scores and which candidates are accepted into residency, test scores and the results of in-training evaluation reports, and finally, test scores and success in licensing examinations.

Conclusion

Candidates who trained in Africa, the Eastern Mediterranean region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, the Western Pacific region, and AU, NZ, SA, the UK, and IRL. Younger candidates (date of birth 1970 and later) achieved higher scores than candidates born before 1970 did. There was a weak but significant association between candidates' MCCQE1 and CE1 scores in 2 of the 3 years, indicating that the performance-based CE1 is likely measuring a different construct than the written examination is. Finally, these results were consistent across 3 years of examination administration. Therefore, it would appear that demographic characteristics have an effect on performance on the CE1.

Notes

EDITOR'S KEY POINTS

  • Candidates whose undergraduate medical education took place in Africa, the Eastern Mediterranean region, and Eastern Europe achieved lower scores than candidates who trained in South America, the Caribbean, the Western Pacific region, and English-speaking overseas countries. Younger candidates (date of birth 1970 and later) achieved higher scores than candidates born before 1970.

  • There was a weak but significant association between candidates' Medical Council of Canada Qualifying Examination Part 1 score and General Comprehensive Clinical Examination 1 scores in 2 of the 3 years, indicating that the performance-based General Comprehensive Clinical Examination 1 is likely measuring a different construct than the written examination is.

  • These results were consistent across 3 years of examination administration.

POINTS DE REPÈRE DU RÉDACTEUR

  • Les candidats qui ont eu leur formation médicale de premier cycle en Afrique, dans la région orientale de la Méditerranée et en Europe de l'Est obtiennent des scores plus bas que ceux qui ont fait leur formation en Amérique du Sud, dans les Caraïbes, dans la région du Pacifique occidental et dans les pays étrangers de langue anglaise. Les candidats plus jeunes (nés en 1970 ou plus tard) ont obtenu des meilleurs résultats que ceux nés avant 1970.

  • Il y avait une association faible mais significative entre le score des candidats à l'examen d'aptitude du Conseil Médical du Canada, partie 1, et celui obtenu au General Comprehensive Clinical Examination 1 (CE1), et ce, pour 2 des 3 années à l'étude, ce qui indique que les éléments évalués par le CE1 pourraient être différents de ceux de l'examen écrit.

  • Ces résultats étaient semblables pour les 3 années où les résultats de l'examen ont été étudiés.

Footnotes

  • Competing interests: None declared

  • This article has been peer reviewed. Can Fam Physician 2013;59:170-6

  • Copyright© the College of Family Physicians of Canada

References

  1. ↵
    1. Dauphinee WD
    . The circle game: understanding physician migration patterns within Canada. Acad Med 2006;81(12 Suppl):S49-54.
    OpenUrlPubMed
  2. ↵
    1. Hallock JA,
    2. Kostis JB
    . Celebrating 50 years of experience: an ECFMG perspective. Acad Med 2006;81(12 Suppl):S7-16.
    OpenUrlCrossRefPubMed
  3. ↵
    HealthForceOntario. HealthForceOntario year end report 2007. Toronto, ON: Ministry of Health and Long-Term Care; 2007. Available from: www.healthforceontario.ca/upload/en/newsletter/hfo_2007%20yearend%20report.pdf. Accessed 2013 Jan 15.
  4. ↵
    HealthForceOntario. HealthForceOntario year-end report 2008/09. Toronto, ON: Ministry of Health and Long-Term Care; 2009. Available from: www.healthforceontario.ca/upload/en/whatishfo/hfo_annual%20rep_20091029.pdf. Accessed 2013 Jan 15.
  5. ↵
    Canadian Resident Matching Service [website]. Positions available for IMGs. 2009/2008 R-1 match comparison. Ottawa, ON: Canadian Resident Matching Service; 2009. Available from: www.carms.ca/pdfs/2009R1_MatchResults/7MatchReport_E.pdf. Accessed 2013 Jan 15.
  6. ↵
    1. Crutcher RA,
    2. Banner SR,
    3. Szafran O,
    4. Watanabe M
    . Characteristics of international medical graduates who applied to the CaRMS 2002 match. CMAJ 2003;168(9):1119-23.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    Canadian Task Force on Licensure of International Medical Graduates. Report of the Canadian Task Force on Licensure of International Medical Graduates. Ottawa, ON: Canadian Task Force on Licensure of International Medical Graduates; 2004.
  8. ↵
    1. Shiroma PR,
    2. Alarcon RD
    . Selection factors among international medical graduates and psychiatric residency performance. Acad Psychiatry 2010;34(2):128-31.
    OpenUrlCrossRefPubMed
  9. ↵
    1. Takahashi SG,
    2. Rothman A,
    3. Nayer M,
    4. Urowitz MB,
    5. Crescenzi AM
    . Validation of a large-scale clinical examination for international medical graduates. Can Fam Physician 2012;58:e408-17. Available from: www.cfp.ca/content/58/7/e408.full.pdf+html. Accessed 2013 Jan 15.
    OpenUrlAbstract/FREE Full Text
  10. ↵
    1. Van Zanten M,
    2. Boulet JR,
    3. McKinley D
    . Correlates of performance of the ECFMG Clinical Skills Assessment: influences of candidate characteristics on performance. Acad Med2003;78(10 Suppl):S72-4.
    OpenUrlPubMed
  11. ↵
    1. Kanna B,
    2. Gu Y,
    3. Akhuetie J,
    4. Dimitrov V
    . Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated internal medicine residency training program. BMC Med Educ 2009;9:42.
    OpenUrlPubMed
  12. ↵
    1. Part HM,
    2. Markert RJ
    . Predicting the first-year performances of international medical graduates in an internal medicine residency. Acad Med 1993;68(11):856-8.
    OpenUrlPubMed
  13. ↵
    1. Muller ES,
    2. Harik P,
    3. Margolis M,
    4. Clauser B,
    5. McKinley D,
    6. Boulet JR
    . An examination of the relationship between clinical skills examination performance and performance on USMLE Step 2. Acad Med 2003;78(10 Suppl):S27-9.
    OpenUrlPubMed
PreviousNext
Back to top

In this issue

Canadian Family Physician: 59 (2)
Canadian Family Physician
Vol. 59, Issue 2
1 Feb 2013
  • Table of Contents
  • About the Cover
  • Index by author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on The College of Family Physicians of Canada.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
IMG candidates' demographic characteristics as predictors of CEHPEA CE1 results
(Your Name) has sent you a message from The College of Family Physicians of Canada
(Your Name) thought you would like to see the The College of Family Physicians of Canada web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
IMG candidates' demographic characteristics as predictors of CEHPEA CE1 results
Marla Nayer, Arthur Rothman
Canadian Family Physician Feb 2013, 59 (2) 170-176;

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Respond to this article
Share
IMG candidates' demographic characteristics as predictors of CEHPEA CE1 results
Marla Nayer, Arthur Rothman
Canadian Family Physician Feb 2013, 59 (2) 170-176;
Reddit logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • METHODS
    • RESULTS
    • DISCUSSION
    • Notes
    • Footnotes
    • References
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Natural history of abdominal pain in family practice
  • Factors influencing family physician engagement in practice-based quality improvement
  • Impact of early waves of the COVID-19 pandemic on family medicine residency training
Show more Research

Similar Articles

Subjects

  • Collection française
    • Résumés de recherche

Navigate

  • Home
  • Current Issue
  • Archive
  • Collections - English
  • Collections - Française

For Authors

  • Authors and Reviewers
  • Submit a Manuscript
  • Permissions
  • Terms of Use

General Information

  • About CFP
  • About the CFPC
  • Advertisers
  • Careers & Locums
  • Editorial Advisory Board
  • Subscribers

Journal Services

  • Email Alerts
  • Twitter
  • RSS Feeds

Copyright © 2023 by The College of Family Physicians of Canada

Powered by HighWire