Research ArticleResearch

Examination outcomes for international medical graduates pursuing or completing family medicine residency training in Quebec

Anne-Marie MacLellan, Carlos Brailovsky, Paul Rainsberry, Ian Bowmer and Micheline Desrochers
Canadian Family Physician September 2010; 56 (9) 912-918;

ABSTRACT

OBJECTIVE To review the success of international medical graduates (IMGs) who are pursuing or have completed a Quebec residency training program and examinations.

DESIGN We retrospectively reviewed IMGs’ success rates on the pre-residency Collège des médecins du Québec medical clinical sciences written examination and objective structured clinical examination, as well as on the post-residency Certification Examination in Family Medicine.

SETTING Quebec.

PARTICIPANTS All IMGs taking their examinations between 2001 and 2008, inclusive, and Canadian and American graduates taking their examinations during this same period.

MAIN OUTCOME MEASURES Success rates for IMGs on the pre-residency and post-residency examinations, compared with success rates for Canadian and American graduates.

RESULTS Success rates on the pre-residency clinical examinations remained below 50% from 2001 to 2008 for IMGs. Similarly, during the same period, the average success rate on the Certification examination was 56.0% for IMGs, compared with 93.5% for Canadian and American medical graduates.

CONCLUSION Despite pre-residency competency screening and in-program orientation and supports, a substantial number of IMGs in Quebec are not passing their Certification examinations. Another study is under way to analyze reasons for some IMGs’ lack of success and to find ways to help IMGs complete residency training successfully and pass the Certification examination.

Like other Canadian provinces, Quebec faces an increasing shortage of family physicians. In addition to finding ways to encourage more Canadian medical graduates (CMGs) to enter family medicine training and practice, preparing international medical graduates (IMGs) for successful clinical practice is a possible solution to this shortage. In the past several years, substantial financial and human resources have been mobilized to support IMGs in Quebec so that once they are admitted to family medicine or specialty residency programs they can complete their training successfully. Ko et al, for example, found that IMGs and CMGs provided similar quality of care in the management of acute myocardial infarction: “the evidence supports the notion that carefully selected IMGs can provide quality health care to [acute myocardial infarction] patients, after passing national certification examinations and having time to assimilate into practice in Canada.”1

International medical graduates’ ability to provide high-quality care, not only in terms of skills but in terms of their relationships with patients, depends largely upon their training and examination performance. Part and Markert concluded that recent clinical experience and performance on standardized examinations were the 2 criteria most predictive of foreign-born IMGs’ first-year performance as internal medicine residents.2 Similarly, Tamblyn et al have highlighted the association between family medicine examination scores and the quality of medical practice.3

Degree equivalency and residency process

The Collège des médecins du Québec (CMQ) is the regulatory authority in Quebec that protects the public, promotes quality medicine, and grants permits to practise medicine in Quebec. The CMQ has evaluated IMGs for more than 25 years; from the early 1980s on, the CMQ has searched for reliable and valid assessment tools that can assist in comparing IMGs’ competencies with those of medical students graduating in Quebec.4 Medical graduates from schools that are not accredited by the Liaison Committee on Medical Education or the Committee on the Accreditation of Canadian Medical Schools are considered to be IMGs. Admittedly, this definition is broad, and IMGs are a heterogeneous group with varied academic backgrounds. Nevertheless, regardless of background or nation of training, all who fall into this category must complete a multi-step process to achieve recognition of the equivalency of their degrees.4 (Note that graduates of American medical schools are not considered to be IMGs, and so they do not need to go through this process.) The steps that IMGs must take to pursue residency in Quebec are as follows:

  1. Apply to the CMQ for recognition of the equivalency of the medical degree.

  2. Pass the Medical Council of Canada Evaluating Examination. (Before December 2007, IMGs had to show proof of success on the Medical Council of Canada Evaluating Examination or the United States Medical Licensing Examination Step 2 Clinical Knowledge, or hold a certificate from the Educational Commission for Foreign Medical Graduates.)

  3. Pass the Medical Council of Canada Qualifying Examination (MCCQE) Part I. (Before December 2007, IMGs had to be successful on the CMQ medical clinical sciences written examination or the MCCQE Part I.)

  4. Pass the CMQ medical clinical sciences objective structured clinical examination (OSCE) or the MCCQE Part II. The OSCE consists of 13 stations with simulated patients and has been developed each year at the Laval University Faculty of Medicine in Quebec. It is almost identical to the exit OSCE for final-year medical students at Laval, prepared by the Centre d’évaluation des sciences de la santé de l’Université Laval. The OSCE assesses history-taking, physical examination, investigation, differential diagnosis, treatment, organization of the encounter, and communication skills (language skills are not evaluated in this OSCE). This clinical measurement tool therefore assesses IMGs at a level comparable to that of CMGs.

  5. Pass the Office québécois de la langue française knowledge of the French language examination, or meet the exemption criteria.

  6. Receive recognition of the equivalency of the medical degree.

Those IMGs whose degrees are recognized through this process can apply to 1 of the 4 Quebec faculties of medicine for family medicine or specialty residency positions if they meet Quebec government requirements for funding eligibility. All residency programs in Quebec are accredited through the same process and with the same standards used in the rest of Canada.

International medical graduates who complete a family medicine residency program are eligible to take the family medicine Certification examination in Quebec in order to obtain a permit to practise. The examination comprises the College of Family Physicians of Canada (CFPC) Certification examination and the MCCQE Part II. Candidates who have already passed the MCCQE Part II do not have to take it again. (Until October 2006, the Quebec Certification examination also included the CMQ family medicine OSCE.)

Purpose

An important first step in evaluating and improving the performance of IMGs who are pursuing or have completed a Quebec residency program is reviewing their success on examinations. In this study, we evaluated and compared IMGs’ and CMGs’ results on pre-residency and family medicine Certification examinations. The next step, which is under way, is to analyze variables from the files of IMGs to see if predictive success factors can be identified.

METHODS

After approval by the CMQ Ethics Committee on Research, a retrospective evaluation was conducted of IMGs’ examination performances. This evaluation involved reviewing IMGs’ results on the CMQ medical clinical sciences examination (written examination and OSCE) from 2001 to 2008, as well as comparing IMGs’ results with those of Canadian and American physicians on CMQ family medicine Certification examinations from 2001 to 2008. All IMGs and CMGs taking the examinations in those years were included. For the purpose of the study, the term CMGs includes graduates of American medical schools accredited by the Liaison Committee on Medical Education. Independent-samples t tests were used to compare the mean scores of CMGs with those of IMGs.

The OSCE passing score is 60%, which is the same as the one used at the Laval University Faculty of Medicine for this type of examination.

The reliability of scores for the different OSCEs was assessed using Cronbach α internal consistency coefficients. We used standardized scores to compare the results from the different examinations, an approach that Tamblyn and colleagues have described.3 For the analyses, we used averaged standardized scores so that we were able to compare the different groups’ performances on examinations by putting the results on the same scale.3

Generalizability studies were performed to analyze whether the circuits in which candidates sat their CMQ OSCE examinations had an effect on their scores. Candidates were nested in circuits by cases (Ca:Ci × Cases) as described by Brailovsky et al.5

RESULTS

Pre-residency CMQ written examination and OSCE

Table 1 outlines IMG candidates’ success rates for the CMQ medical clinical sciences written examination and IMG and Laval University CMG candidates’ success rates for the CMQ OSCE. Candidates are allowed 3 attempts at each of these examinations. The success rates described do not differentiate between first-time examination takers and repeaters, although determining whether repeaters pass on subsequent attempts or consistently do poorly on examinations would be valuable.

View this table:
Table 1

Success rates on the CMQ medical clinical sciences examination: Written examination pass rates for IMGs and OSCE pass rates for IMGs vs CMGs; an independent-samples t test for the comparison of the groups’ mean scores revealed a significant difference between each pair of groups (for each year); P < .0001.

The IMGs’ success rate for the written examination is higher than that for the clinical OSCE. Scores for written examinations are known to correlate poorly with scores obtained in examinations that measure clinical competence.6,7 Clinical competence examinations require other skill sets, such as history-taking and patient management skills. The success rate for the CMQ OSCE examination remains below 50% for IMGs, even though preexamination OSCE workshops have been offered for the past 5 years. Laval University students completing their final year of undergraduate medical studies have an average success rate of about 98% when they take this exit OSCE before receiving their medical degrees.8

Cronbach α coefficients varied from 0.784 to 0.878 for the different examination sessions. The case-to-case homogeneity to measure the construct of clinical competence in each examination was supported through the item-item total correlations (> 0.300 in 93% of cases). The calculation of the coefficient “α minus the item” for each case demonstrated that more than 94% of cases contributed positively to the reliability of the examinations.

Our generalizability studies indicated that the percentage of variance explained by circuits was almost nil, and the percentage of variance explained by cases by circuits was very low (< 3% in all examinations). This result was considered sufficient indication that candidates were not penalized or advantaged by the examination circuit where they sat the examination.5

When we analyzed candidates’ performance on the different components of the OSCE, we found that IMGs perform poorly in history-taking, investigation, differential diagnosis, and treatment. When we used standardized scores to compare IMGs’ and CMGs’ performances, the IMGs’ z scores were approximately 1.5 to 2 standard deviations lower than CMGs’ scores on the same skills. The IMGs’ mean aggregate global scores were 15% to 20% lower than CMGs’ scores.

Results on CMQ family medicine Certification examinations

Table 2 shows that, from 2001 to 2008, the success rates of IMGs in Quebec were significantly lower than those of CMGs graduating from Quebec faculties of medicine. Again, the success rates described in Table 2 do not differentiate between first-time examination takers and repeaters. The table also shows that the removal of the additional Quebec family medicine examination in 2007 and 2008 did not substantially improve the success rates for IMGs. Between 2001 and 2008, the average success rate for IMGs was 56.0%. Quebec medical school graduates who completed the same family medicine residency programs had an average success rate of 93.5% on the same Certification examinations during this period (Table 2).

View this table:
Table 2

Quebec-trained physicians’ pass rates on CMQ family medicine Certification examinations: IMGs vs candidates from Canada and the United States; an independent-samples t test for comparison revealed a significant difference between each pair of groups (for each year); P < .0001 for results from all years presented.

While the Quebec IMG numbers are small, the findings are confirmed by results on the CFPC examination for all Canadian family medicine residency programs (Tables 3 and 4). In 2007, CMGs’ overall success rate on the CFPC examination was 90.4%, which is much higher than the 66.0% overall success rate for IMGs (Table 3). The discrepancy in success rates was confirmed in 2008.

View this table:
Table 3

Overall pass rates on the CFPC Certification Examination in Family Medicine for CMGs and IMGs: An independent-samples t test for the comparison of the groups’ mean scores revealed a significant difference between each pair of groups (for each year); P < .0001.

View this table:
Table 4

Results for CMGs and IMGs on SAMPs and SOOs in the CFPC Certification Examination in Family Medicine: An independent-samples t test for the comparison of the groups’ mean scores reveals a significant difference between each pair of groups (for each year); P < .0001.

DISCUSSION

In spite of the pre-residency assessment tools used in Quebec, including the assessment of medical knowledge and clinical skills, a substantial number of IMGs accepted into residency training have difficulty completing their programs and passing the Certification Examination in Family Medicine. Data from the Conférence des recteurs et des principaux des universités du Québec, available on the CMQ website, indicate that at least 10% of IMG residents leave their family medicine residency programs without completing them, and that 55% require additional training.9

Integration issues

One hypothesis, which Bates and Andrew proposed in 2001, is that unsuccessful IMGs have been unable to integrate into the Canadian health system.10 The authors discuss the challenges that IMGs face when they enter the North American postgraduate educational system: difficulties with English; differences in medical education; length of time since medical school graduation and clinical experience; financial and family obligations; traumatic experiences; and different cultural beliefs about gender roles, communication, and similar issues.10 In that article and another on the 7-year experience of the program to assess IMGs in British Columbia,11 they therefore recommend strategies such as an evaluation process that assesses attitudes and behaviour, clear articulation of the program’s expectations and North American expectations for physicians, and experienced faculty who can explore the root causes of academic problems.10,11

The IMGs in our study, however, had had some time to adapt to the Canadian system, although transitioning from one professional and personal culture to another is, of course, an ongoing, long-term process. Our IMGs passed the screening competency examinations leading to recognition of the equivalency of their diplomas, and they spent at least 24 months in a Canadian family medicine residency program. In addition, since 2004, the 4 Quebec family medicine programs have organized a pre–family medicine residency orientation program for IMGs to help them integrate into the Canadian health care system and prepare for their residencies. During residency, family medicine tutors and clinical preceptors devote additional time to helping these physicians integrate and acquire the competencies needed for independent practice.

Clinical decision making

We offer some other tentative explanations. Possibly the poorer performance is not attributable to a lack of medical knowledge and competency in clinical skills, but rather to how and when an IMG learns to translate the knowledge and integrate it with clinical decision making—an ability that CMGs acquire during their clerkships, if not earlier in their training. Recently, Palés et al surveyed medical graduates in Spain about their acquisition of learning outcomes.12 Eleven domains were examined. Students reported that they felt competent in 8 domains—clinical skills, practical procedures, patient investigation, health promotion, communication skills, attitudes and ethical and legal responsibilities, role of the doctor, and personal development—but only partially competent in 3 domains: patient management, medical information search skills, and decision-making skills and clinical reasoning and judgment.12 On the other hand, data from an ongoing study by Brailovsky and MacLellan suggest that IMGs who return to medical school, usually at a clerkship level, complete their family medicine residency and Certification examinations in a fashion comparable to that of CMGs.

Varied educational backgrounds

Another hypothesis is that, unlike CMGs’ accredited, homogeneous medical education, IMGs’ undergraduate medical education is variable, and they are a diverse group. For some IMGs, therefore, the educational process might not provide the foundation in knowledge, skills, and attitudes they need to succeed in Canadian residency training and Certification examinations. For example, van Zanten and Boulet’s recent study revealed considerable variability in undergraduate medical education and performance on the United States Medical Licensing Examination among students from medical schools in different Caribbean nations.13 This variability was evident on the 3 examination components (basic science, clinical knowledge, and clinical skills and clinical skills assessment), although, interestingly, the pass rates were generally higher for the clinical skills component.

Limitations

The main limitation of our study is a lack of specific information on which IMGs succeed in passing examinations. The study also did not reveal predictors of success. However, the preliminary finding that fewer IMGs than CMGs pass examinations of clinical skills will guide our future research on improving IMGs’ examination performance and ensuring their successful entry into practice.

Conclusion

In Quebec, IMGs have a much lower success rate on the Certification Examination in Family Medicine than CMGs do. This is the case even though the IMGs have passed several screening competency examinations and have successfully completed a 2-year accredited family medicine residency program.

Many questions remain about the reasons for family medicine IMGs’ lack of success on examinations. A clearer understanding of which IMGs are more likely to fail or pass might be achieved by exploring differences between candidates who grew up in Canada but attended international medical schools and candidates raised in different nations. In addition, a closer look could be taken at whether candidates’ performance differs depending on whether or not they attended medical schools in nations where accreditation criteria are similar to those used by American and Canadian medical schools.14

Currently, we are conducting a retrospective analysis of several variables in the 2000 to 2007 files of IMGs who requested that the CMQ recognize the equivalency of their medical diplomas. Data concerning IMGs who complete Quebec clerkships are also being analyzed. One goal of this new study is to explore why many IMGs who pass the pre-residency screening examinations fail the post-residency Certification examination. Another goal is to identify factors that can help IMGs complete their residency training successfully, pass their Certification examinations, and enter family practice in Quebec.

Acknowledgments

Funding for this study was provided by the Collège des médecins du Québec and by the Ministère de la santé et des services sociaux du Québec. We thank the Collège des médecins du Québec Medical Education Division staff, particularly Ginette Plante-Lauzier. Alison Harkin provided editorial assistance with the manuscript.

Notes

EDITOR’S KEY POINTS

  • Preparing international medical graduates (IMGs) for successful family practice could be part of the solution to Canada’s shortage of family physicians. The ability of IMGs to provide high-quality care depends largely on their training and examination performance. A first step in evaluating and improving the performance of IMGs is reviewing their success on examinations.

  • This study compared the results of IMGs completing residency in Quebec with those of Canadian and American graduates in Quebec and found that only about half of IMGs passed their Certification examinations, compared with almost all Canadian and American graduates. The IMGs’ success rates were higher for written examinations than for clinical examinations.

  • The authors suggest IMGs’ poorer performance could be related to how and when IMGs learn to translate their knowledge and integrate it with clinical decision making or to the diversity of IMGs as a group and the variability of their undergraduate medical educations.

POINTS DE REPÈRE DU RÉDACTEUR

  • Une façon de corriger la pénurie de médecins de famille au Canada serait de préparer les diplômés internationaux en médecine (DIM) à une pratique familiale adéquate. La capacité des DIM à dispenser des soins de haute qualité dépend en bonne partie de leur formation et de leurs résultats aux examens. Une vérification des résultats des DIM aux examens constitue une première étape dans l’évaluation et l’amélioration de leur rendement.

  • Cette étude a comparé les résultats des DIM ayant complété leur résidence au Québec aux résultats des diplômés canadiens et américains et trouvé qu’environ la moitié seulement des DIM réussissaient leurs examens de certification comparativement à la presque totalité des diplômés canadiens et américains. Le taux de réussite des DIM était meilleur aux examens écrits qu’aux examens cliniques.

  • Les auteurs suggèrent que les moins bons résultats des DIM pourraient être en relation avec la façon et le moment où ils apprennent à traduire leurs connaissances et à les intégrer à la prise de décision clinique, ou encore avec la diversité des DIM comme groupe et la variabilité de leur formation médicale de premier cycle.

Footnotes

  • This article has been peer reviewed.

  • Cet article a fait l’objet d’une révision par des pairs.

  • Contributors

    All the authors contributed to concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.

  • Competing interests

    None declared

References

  1. 1.↵
    1. Ko DT,
    2. Austin PC,
    3. Chan BT,
    4. Tu JV
    . Quality of care of international and Canadian medical graduates in acute myocardial infarction. Arch Intern Med 2005;165(4):458-63.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Part HM,
    2. Markert RJ
    . Predicting the first-year performances of international medical graduates in an internal medicine residency. Acad Med 1993;68(11):856-8.
    OpenUrlPubMed
  3. 3.↵
    1. Tamblyn R,
    2. Abrahamowicz M,
    3. Brailovsky C,
    4. Grand’Maison P,
    5. Lescop J,
    6. Norcini J,
    7. et al
    . Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA 1998;280(11):989-96.
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Collège des médecins du Québec
    . Practicing medicine in Québec. Montreal, QC: Collège des médecins du Québec; 2009.
  5. 5.↵
    1. Brailovsky CA,
    2. Grand’Maison P,
    3. Lescop J
    . A large-scale multicenter objective structured clinical examination for licensure. Acad Med 1992;67(10 Suppl):S37-9.
    OpenUrlPubMed
  6. 6.↵
    1. Auewarakul C,
    2. Downing SM,
    3. Jaturatamrong U,
    4. Praditsuwan R
    . Sources of validity evidence for an internal medicine student evaluation system: an evaluative study of assessment methods. Med Educ 2005;39(3):276-83.
    OpenUrlPubMed
  7. 7.↵
    1. Page G
    . The summative assessment of clinical performance in the workplace. Presented at: Conference at the Centre for Medical and Health Sciences Education, Monash University; February 22, 2007; Monash, Australia. Available from: www.med.monash.edu.au/cmhse/docs/workplace-assessment-07.ppt. Accessed 2009 Apr 27.
  8. 8.↵
    1. Le Centre d’évaluation des sciences de la santé de l’Université Laval
    . Laval University Medical School internal reports from 2000 to 2008. Quebec, QC: Laval University; 2008.
  9. 9.↵
    1. Collège des médecins du Québec
    . Statistics on the issuance of permits to practice in recent years. Montreal, QC: Collège des médecins du Québec; 2009. Available from: www.cmq.org/en/ObtenirPermis/DiplomesInternationaux/Statistiques.aspx. Accessed 2009 Apr 27.
  10. 10.↵
    1. Bates J,
    2. Andrew R
    . Untangling the roots of some IMGs’ poor academic performance. Acad Med 2001;76(1):43-6.
    OpenUrlCrossRefPubMed
  11. 11.↵
    1. Andrew R,
    2. Bates J
    . Program for licensure for international medical graduates in British Columbia: 7 years’ experience. CMAJ 2000;162(6):801-3.
    OpenUrlAbstract/FREE Full Text
  12. 12.↵
    1. Palés J,
    2. Gual A,
    3. Gomar C,
    4. Estrach MT
    . Acquisition of learning outcomes by students from the Medical School of the University of Barcelona (Catalonia, Spain): a student survey. Med Teach 2008;30(7):693-8.
    OpenUrlPubMed
  13. 13.↵
    1. Van Zanten M,
    2. Boulet JR
    . Medical education in the Caribbean: variability in medical school programs and performance of students. Acad Med 2008;83(10 Suppl):S33-6.
    OpenUrlPubMed
  14. 14.↵
    1. US Department of Education [website]
    . National Committee on Foreign Medical Education and Accreditation (NCFMEA). US Department of Education; 2009. Available from: www.ed.gov/about/bdscomm/list/ncfmea.html#reports. Accessed 2009 Dec 20.
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Respond to this article
Examination outcomes for international medical graduates pursuing or completing family medicine residency training in Quebec
Anne-Marie MacLellan, Carlos Brailovsky, Paul Rainsberry, Ian Bowmer, Micheline Desrochers
Canadian Family Physician Sep 2010, 56 (9) 912-918;
Twitter logo Facebook logo Mendeley logo

Jump to section

Related Articles

  • No related articles found.

Cited By...

More in this TOC Section

Similar Articles

Subjects