Study design, participant recruitment and data collection

All learners are tracked through administrative databases (eg, medical school admissions and educational programmes databases, medical licencing agencies registration databases), which provide basic demographic data (eg, table 1), details of the learner’s educational experience at NOSM and selected information on outcomes (eg, provincial health insurance (billing) databases). Additional demographic data as well as the learner's perspective on factors that influence key outcomes plus detailed information on the outcomes are obtained by surveys or interviews.

This prospective comparative longitudinal study follows multiple cohorts from entry into NOSM's UG or PG programmes, and at least 5 years into independent (fully qualified) practice. A purposive sampling strategy invites all NOSM UG and PG medical learners to voluntarily participate in surveys or interviews. Every year, 56 UG students (64 since 2010) are tracked throughout their UG education and into PG residency, when they are joined by additional 40–60 PG residents who are new to NOSM. NOSM UG students are asked to participate in surveys and interviews at the first-year orientation, end of second year, and end of fourth year (figure 1). NOSM PG residents are invited to participate in surveys during orientation and just prior to completion of residency. NOSM UG students who go elsewhere for their PG residency training (30–40 residents per year go to other medical schools for PG training, included in the count of NOSM UGs) are invited to continue their participation in surveys at entry and completion of their residency. To summarise, the study tracks learners who finish (1) their UG education and PG training at NOSM, (2) only their UG at NOSM and go to other schools to complete their PG training or (3) only their PG at NOSM having completed their UG medical education at other schools. These three groups are followed for residents who become family physicians or other specialists (combined) to yield six groups for comparison.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Flow of medical learners through the NOSM as of January 2014 (CFPC, College of Family Physicians of Canada; NOSM, Northern Ontario School of Medicine; PG, postgraduate; PGE, postgraduate entry survey; PGX, postgraduate exit survey; PGY, postgraduate year; RCPSC, Royal College of Physicians and Surgeons of Canada; UG, undergraduate; UGE, undergraduate entry survey; UGM, undergraduate midway survey; UGX, undergraduate exit survey; UGY, undergraduate year).

CRaNHR researchers invite NOSM medical students, but not NOSM personnel, to a CRaNHR-sponsored meal, to explain the study and distribute the survey in paper or electronic format, or by a web page link. Wherever possible, a similar event is organised for PG residents at NOSM. NOSM UGs who go elsewhere for their residency training are invited by email or mail to start or continue their participation in the study. All subsequent contact with participants is by email or mail. A modified tailored design method31 is used for all surveys with at least two reminders, excepting those who have opted out. Each UG survey round lasts until all learners have responded or for 3 months, whichever comes first. PG survey rounds last up to 4 months and are initiated throughout the year because of staggered starts and exits due to, for example, parental leave or extra training requirements for internationally trained medical graduates. Since, contact information for residents at other medical schools can be difficult to obtain, we send an invitation whenever we have updated contact information. Participants can complete an on-line questionnaire, electronic MS Word document or paper form. For each survey round, a draw is held for a $C50 gift card from a national retail store as an incentive to participate.

During the UG entry survey, students in the first 5 years were also invited to participate in short-duration semistructured interviews in their first year and again in their fourth year. Interviews were conducted face-to-face, by telephone, or by Skype, depending primarily on learner preference. All interviews were digitally recorded with the interviewee's permission and conducted by Dr Hoi Cheu (CRaNHR Faculty Investigator) using a six-question interview guide, with prompting questions as needed. Questions were shared at least 1 day prior to the interview. Interviewees were given an honorarium of a $C25 gift card from a national retail store.

Exposure

NOSM's UG and PG admissions criteria and medical education programmes comprise the exposure. NOSM serves as the Faculty of Medicine of Laurentian University in Sudbury (2011 census metropolitan area population: 161 000) and of Lakehead University in Thunder Bay (2011 census metropolitan area population: 122 000)—located 1000 km apart by road. NOSM selects medical school (UG) applicants with a grade point average (GPA) of ≥3.0 of 4.0 in science and non-science university degrees, and does not require the Medical College Admission Test.32 Mean GPA was 3.8 for NOSM students starting in 2015, and this falls within the 3.7–3.9 range for all other Canadian medical schools.33 Preference is given to students from northern, rural, remote, Aboriginal or Francophone backgrounds so as to reflect Northern Ontario demographics. Learners must also have a strong interest in the understanding of, and aptitude for, practising medicine in Northern Ontario. NOSM provides medical learners with educational and clinical experiences in different health service settings in over 90 rural, remote and northern communities.34 ,35 For instance, all first and second year medical students undertake a 1-month Integrated Community Experience in Northern Ontario Aboriginal and rural or remote communities. In the third year, all medical students complete an 8-month Longitudinal Integrated Clerkship, grounded in family practice, and located in 1 of 15 large rural or small urban communities in Northern Ontario, away from Sudbury or Thunder Bay. Similarly, NOSM's PG residency programmes combine learning at the regional hospitals in Sudbury and Thunder Bay with clinical rotations throughout rural and Northern Ontario. NOSM offers PG residency training in family medicine and in eight additional specialist programmes.36 All of this is designed to select learners from rural areas or who are aware of the healthcare needs of the rural underserved, and enable learners to be trained and mentored by physicians who have chosen to live and practice in Northern Ontario so as to prepare learners for practices with fewer resources than in major population centres.

Research questions, study outcomes and explanatory variables

The main research questions and key variables were derived from the literature with selected input from the funder (MOHLTC). Questions and variables were outlined in a research framework adopted by the advisory committee, updated annually and critically reviewed in the 5th and 9th study year. All tools and methods are being reviewed in 2015—the 11th study year. The main research outcomes (table 2) include:

• Clinical and organisational practice characteristics: medical discipline, medical services and procedures, patient population, practice organisation (eg, solo, interprofessional care team);

• Practice location: categorised by geographic region, population size/density and rural–urban continuums.

Explanatory variables include:

• Learner traits: selected socioeconomic and education demographic characteristics including rural or northern background and language/culture/ethnicity.

• Medical education: level (ie, UG or PG) and medical school (ie, NOSM or other school).

• Influential factors: opportunity, personal, familial and societal imperatives that affect decision-making around the main outcomes.

Many study outcomes are collected first as intention and then as actual outcome (eg, intended and actual medical discipline). Intended influential factors are those considered by the respondent as important prior to decision-making, and actual influential factors are those that respondents report in hindsight as having affected their decision.

Development and assessment of study tools

Tools to extract administrative data, questionnaires and interview guides were developed as the charter class progressed through their medical education: UG entry questionnaire and interview guide were developed in the academic year 2005/2006; UG midway questionnaire in 2006/2007; and the UG exit questionnaire and interview guide in 2008/2009. PG residents were tracked since 2009 by using administrative data. However, funding delays meant that the PG entry and exit surveys were not developed until 2011/2012. Measures to fill this data gap are described in the section on limits and strengths.

Operationalisation of outcomes, linkages among independent and dependent variables as well as question wording were based on the literature available when the study began,11 informed by a workshop to evaluate the impact of medical education initiatives in Canada,46 and updated with literature being published as the study progressed.12–14 To provide additional content validation and to facilitate comparisons with other medical educational programmes, most questions were based on similar CRaNHR studies.47–49 Other questions were based on the literature, including a block of questions to measure student attitudes on working and living in rural areas,50 modified to the Canadian context and used with permission (Adams ME, Dollard J, Hollins J and Petkov J, personal communications, 2005). Questions from earlier studies were revised to reflect choices available to NOSM learners. NOSM UG medical students, PG residents and practising MDs (two of each and all located in Sudbury, Ontario) reviewed surveys for content validity and readability. Interview questions inquire about selected key outcomes and related decision-making in greater detail.

Multiple data sources (eg, surveys, interviews or administrative data) for several outcomes improved content validity and allowed checking consistency of response. Test–retest reliability of the questionnaires was not assessed because the research team judged that the likelihood of respondents remembering their answers would be too high over the short term, and that answers to many questions would be expected to change in as little as a few months as respondents became immersed in NOSM's distributed medical education programmes.

Dealing with potential bias in surveys and interviews

To reduce social desirability bias, CRaNHR researchers ensure that NOSM faculty or staff are absent during surveys or interviews. Learners are told that their responses will not affect their academic standing and that only aggregate data would be published or shared with NOSM and other stakeholders. Researchers seek to reduce non-response bias by providing multiple mediums (ie, paper, electronic, or online surveys) for up to 3–6 months, to facilitate participation at the learners’ convenience. Recall bias may be an issue only for selected questions about the geographic location of where respondents or their spouses have lived previously. Researchers use administrative data to assess non-response and recall bias for selected information on demographics and outcomes.

Analytical approach

Data comprise multiple measurements on individual learners generated from an (in)complete census of each cohort, and therefore, descriptive statistics or randomisation tests will be used to determine associations or group differences.51 ,52 The χ² tests, logistic regression and hierarchical log-linear models will be the most frequently used statistical methods. Cohorts are stratified by medical school (ie, NOSM vs other), education level (ie, UG vs PG) and medical specialty (ie, family medicine vs other specialties). The use of multiple imputation techniques to handle missing data will be considered in the context of the specific analytical method or research question. Every effort will be made to contact non-respondents, provided they have not explicitly declined to participate. Administrative data are used to track learners, obtain basic demographic information, details of the learners’ NOSM education experience as well as selected outcomes, while surveys and interviews allow for collection of more detailed data.

Interview transcripts and responses to open-ended questions are analysed using an iterative analytical and inductive approach to group findings within each question.53 Transcripts and electronic recordings are re-examined to ensure that context is preserved, and that confirmatory and contradictory findings are noted. Researchers’ interpretations are distinguished from key informants’ statements,54 while anonymised quotes illustrate the scope and depth of groupings plus exceptions, if any.

Dissemination

All data are stored on a secure server hosted by Laurentian University with access to individual-level data restricted to CRaNHR researchers directly involved in the study. CRaNHR shares only aggregated data (cell size >5) with NOSM personnel or other stakeholders and researchers, and follows other Statistics Canada guidelines to reduce identity, attribute or residual disclosure.55

Results will be published in peer-reviewed scientific journals and presented at one or more scientific conferences. Research highlights will also be shared with policymakers and decision-makers and the public through 4-page reader-friendly summaries of research results (^{Research In} FOCUS _{On Research}), and by social media such as Facebook (http://www.facebook.com/cranhr) and Twitter (@CRaNHR, @NOSM and researchers’ accounts).

Limits and strengths of the approach

One limitation arises in assessing the exposure because NOSM selects UG medical students (but not necessarily PG learners) with rural or northern Canada backgrounds. Given that rural background is strongly associated with practice in rural areas,7 ,8 ,11–14 there may not be much variation remaining among NOSM medical students to predict outcomes such as location of rural practice. However, the evidence for the influence of other factors, such as northern Canada background, language/culture, gender or marital/partnership status, on outcomes such as medical discipline and practice location varies among contemporary studies11–14 and may be evolving over time and so the study will assess these influences. In addition, the tracking study is able to isolate the influence of different medical schools (ie, NOSM vs other) at different levels (ie, UG vs PG), and for different medical disciplines (ie, family medicine vs other specialties).

Small population size may limit some analyses given that there are 56 new UG students each year (64 since 2010) and a lower number in some PG programmes, especially specialties other than family medicine. Groups may be combined to achieve adequate numbers for analysis, albeit at the loss of some detail.

Choice of outcome measures derived from medical care needs of Northern Ontario and situated in the political context may be interpreted as a limitation as well as a strength. Perhaps a more important limitation is that study outcomes (ie, practice location and scope of practice) are proxies of the ultimate outcome—the health of Northern Ontarians. However, choice of proxy outcomes is reasonable given that NOSM is an important step in ensuring that there are sufficient numbers of skilled and locally trained MDs in Northern Ontario.1 ,2 The expectation is that improved access to MDs will help improve the health of Northern Ontarians.

Other limitations include delays and gaps in execution of surveys. UG surveys and interviews have been on schedule since early 2006 (a 6-month delay), while PG surveys have been on schedule since 2012 (2 prior cohorts had incomplete coverage). Fortunately, administrative data is available from NOSM, and missed PG entry surveys had near-temporal equivalents in the UG exit survey, and so gaps in PG entry survey coverage exist only for PGs new to NOSM in 2009 and 2010. Changes in the wording of questions, or response options, create challenges for temporal continuity that are addressed by a detailed codebook that facilitates appropriate comparisons and provisos.

Study tools and methods are reviewed in-house, which increases internal utility, but may reduce external validity. Although there is no third-party review, many of the indicators and outcomes are copied or derived from the international literature. In addition, several advisory committee members are experts in rural or distributed medical education in Canada, the USA and Australia, and the study benefits accordingly.

Future study

The tracking study will be integrated within a broader research programme assessing the medical, social and economic impact of NOSM on Northern Ontario communities building on previous research.56 Detailed individual-level data allows for investigations into the relationship between specific aspects of NOSM's programmes and medical education outcomes or socioeconomic impacts. For instance, practice characteristics (eg, medical discipline, geographic location) of medical students with science backgrounds could be compared with students with non-science backgrounds.57 Other examples would be to compare performance and practice characteristics of students who had their third year clerkship in larger versus smaller communities (Ellaway RH and Graves L, personal communication, 2011), or to assess the effect of cultural safety training (Jacklin K and Maar M, personal communication, 2012). The integrated study will include investigations into NOSM's admission criteria and processes.