ABSTRACT
OBJECTIVE To increase the overall 2007 response rate of the National Physician Survey (NPS) from the survey’s 2004 rate of response with the implementation of various methodologic strategies.
DESIGN Physicians were stratified to receive either a long version (12 pages) or a short version (6 pages) of the survey (38% and 62%, respectively). Mixed modes of contact were used—58% were contacted by e-mail and 42% by regular mail—with multiple modes of contact attempted for nonrespondents. The self-administered, confidential surveys were distributed in either English or French. Medical residents and students received e-mail surveys only and were offered a substantial monetary lottery incentive for completing their surveys. A professional communications firm assisted in marketing the survey and delivered advance notification of its impending distribution.
SETTING Canada.
PARTICIPANTS A total of 62441 practising physicians, 2627 second-year medical residents, and 9162 medical students in Canada.
RESULTS Of the practising physicians group, 60811 participants were eligible and 19239 replied, for an overall 2007 study response rate of 31.64% (compared with 35.85% in2004). No difference in rate of response was found between the longer and shorter versions of the survey. If contacted by regular mail, the response rate was 34.1%; the e-mail group had a response rate of 29.9%. Medical student and resident response rates were 30.8% and 27.9%, respectively (compared with 31.2% and 35.6% in 2004).
CONCLUSION Despite shortening the questionnaires, contacting more physicians by e-mail, and enhancing marketing and follow-up, the 2007 NPS response rate for practising physicians did not surpass the 2004 NPS response rate. Offering a monetary lottery incentive to medicalresidents and students was also unsuccessful in increasing their response rates. The role of surveys in gathering information from physicians and physicians-in-training remains problematic. Researchers need to investigate alternative strategies for achieving higher rates of response.
If the Canadian health care system is to meet the needs of Canadians, health work force planning must be based on accurate information from both existing and future health care providers. The College of Family Physicians of Canada (CFPC), the Canadian Medical Association (CMA), and the Royal College of Physicians and Surgeons of Canada (RCPSC) collaboratively collect such information every 3 years through the National Physician Survey (NPS). In 2004, 21 296 physicians across Canada completed the NPS questionnaire (35.9% response rate), along with 598 second-year medical residents (31.4% response rate) and 2721 medical students (35.6% response rate). The 2004 physician questionnaire was 16 pages long and used a modified Dillman methodology1 and relatively little marketing communication to encourage participation. In response to feedback on the 2004 NPS results, one of the methodologic objectives of the 2007 NPS was to achieve an overall response rate of at least 50%.
The NPS is unique in that it surveys all physicians, second-year medical residents (PGY2s), and medical students (MSs) in Canada. Although the NPS is able to achieve small error rates because of the large sample size (± 0.7% 19 times out 20 for the 2004 physician results), the most important aspect of any survey design is its ability to maximize the response rate.1
As early as 1978, Cartwright reported that response rates of health professionals, especially GPs, to postal surveys had consistently decreased from 1961 to 1977.2 Several studies2–8 have shown that this trend has persisted. A 1994 review of published GP studies found a mean response rate of 61% but indicated that this value was likely an overestimation, as surveys with low response rates are less likely to be accepted for publication.9
While some studies have highlighted the concern caused by the declining response rates from GPs,3,10 others have focused on the reasons why GPs do not respond to questionnaires11–14 and on strategies to increase survey response rates in various populations.15–19 A 2002 systematic review15 found that response rates to postal questionnaires could substantially increase if the questionnaires were shorter.20 Indeed, nearly 20% of the respondents to the 1-page follow-up questionnaire sent to nonrespondents of the 2004 NPS indicated that they did not complete their original NPS survey that year because it was too long (16 pages).
The Internet is increasingly being considered an efficient means for conducting surveys,21,22 including for surveying physicians.23 Potential efficiencies include savings on time, postage, and printing costs.24 In populations that already use the Internet, electronic surveys have been found to be a useful means of conducting research.25–27 On the 2004 NPS, only about 1 in 5 (22%) physicians indicated that they did not have access to the Internet in their main patient care setting.28 Mixed mode strategies (offering both paper and on-line formats of questionnaires and other communications) have been recommended by other researchers.29,30 For the 2004 NPS, a greater percentage of physicians initially contacted by e-mail (compared with regular mail) completed the survey (40.1% versus 34.1%).
Kaner et al found that GPs were most likely to respond to postal surveys that had a high interest factor.11 Topical salience—the relevance and timeliness of the topics being studied according to the population receiving the surveys—is among the most important predictors of response.31,32 The ability of respondents to recognize a particular survey initiative as being one in which they wish to participate is also related to topical salience. Between 2004 and 2007, the NPS was increasingly branded through the widely publicized release and conveyance of the 2004 NPS results; therefore, the survey’s visibility and salience increased among physicians and the public in Canada as being a valid and useful source of Canadian physician work force information.
Follow-up contact with nonrespondents also increases response rates15,20,31–33; however, the optimal follow-up strategy for GPs has not been confirmed.5,11 Another way to increase response rates is personalization of the survey instrument and contacts with respondents.11,20 The well-known Dillman survey method1,29 recommends advance notification, follow-up, and personalization.
Edwards et al15 indicate that monetary incentives can achieve substantial increases in response rates to postal questionnaires16,34; they have also been found to positively affect response rates for GP-specific postal studies.35,36 The opportunity for respondents to be entered into a lottery can also increase physician response.37,38
Pooling the lessons learned from the literature and the 2004 NPS study, the 2007 NPS incorporated several methodologic strategies in an effort to increase the overall response rate. This paper reports on the following strategies and their results: first, the responses of Canadian physicians to the 2007 NPS survey, which utilized a shorter questionnaire, an increased percentage of respondents receiving e-mail communications, a more intensive marketing strategy, and enhanced topical salience compared with 2004; and second, the responses of Canadian PGY2s and MSs to their versions of the NPS when a lottery-based monetary incentive was introduced.
METHODS
Questionnaire design
A working group (including representatives from the CFPC, the CMA, the RCPSC, the Canadian Institute for Health Information, and other affiliated societies) reviewed and refined the 16-page 2004 physician NPS questionnaire and the student and resident questionnaires to create the 2007 NPS questionnaires. The refinement process included a call for content review and input from a range of stakeholders. The 2007 survey questions covered many key health care system issues for physicians in Canada and can be found at www.nationalphysiciansurvey.ca. The NPS questions were pilot-tested and finalized during the fall of 2006. The NPS received ethics approval from the University of British Columbia Behavioural Ethics Review Board.
Physician survey
The physician questionnaires were available in both paper and electronic versions. A core questionnaire (6 pages) and 2 versions of a more detailed questionnaire (12 pages each) were developed, one for family physicians and GPs and the other for all other specialists (with differences in clinical practice profile questions only). All core questionnaire content was also captured on the detailed questionnaires.
Second-year resident and MS surveys
For PGY2s, 2 on-line questionnaires were developed; the first was specific to family medicine residents and the second to residents in all other medical specialty programs. For the MSs, a separate on-line questionnaire was developed. Many of the PGY2 and MS questions were comparable to the core questions for the practising physicians.
Sample
Physician survey
The target population was all 62441 physicians (32891 family physicians and 29550 other specialists) licensed to practise in Canada. The list of physicians (the NPS Masterfile) was generated from the CMA membership system and the CFPC and RCPSC membership databases.
A new methodologic approach was used for NPS 2007, similar to that of the Census of Canada, which involved issuing a core questionnaire to a majority of recipients and a more detailed questionnaire to only a subset of recipients. Using a stratified systematic sample governed by operational and theoretical constraints39 (with a goal of maximizing response rate and statistical reliability while minimizing survey burden), physicians were assigned to strata based upon their province or territory of practice, broad specialty (family physician or other medical specialty), and sex. For provinces with large populations, (British Columbia, Alberta, Ontario, and Quebec), 1 in 3 physicians of each stratum received the detailed questionnaire and all others received the core questionnaire. For areas with smaller populations (the Atlantic provinces, Manitoba, Saskatchewan, and the territories), 2 out of 3 physicians in each stratum received the detailed questionnaire and all others received the core questionnaire.
Second-year resident and MS surveys
The targeted physician-in-training populations included all MSs (N = 9162) and all PGY2s (N = 2627) attending Canadian medical schools. As the length of family medicine residency training in Canada is usually 2 years, PGY2s were targeted to glean meaningful opinions about training and future intentions of residents in every medical specialty.
Data collection
Physician survey
Using a modified Dillman1 approach, the 2007 NPS physician survey applied a mixed methodology of e-mail and regular postal delivery to contact physicians. E-mail addresses were verified for 58% of all physicians (compared with 34% in 2004). The remaining 42% received all NPS communications by regular mail.
A survey identification number (unrelated to any existing organizational membership number) was assigned to each contact in the NPS Masterfile. These identification numbers ensured confidentiality of physician responses and enabled targeted follow-up with nonrespondents.
Beginning early in 2007, physicians were contacted multiple times according to the timeline in Figure 1,* with a change in format for the final contact. Any physicians with undeliverable e-mail addresses were moved to the regular mail group. All e-mails (including the first notification message) received by physicians in the e-mail group contained the URL link to the questionnaire, their unique identification number to access the questionnaire, and a link to an on-line “About the NPS” brochure. The survey package sent to the regular mail group contained a questionnaire, cover letter, print version of the brochure, and postage-paid return envelope. The e-mails and letters were addressed individually to each physician and were signed by the presidents of the CFPC, CMA, and RCPSC.
Communication was conducted either in French or English, depending on physicians’ stated preference. Physicians who were sent a paper questionnaire had the option of completing the questionnaire on-line via the URL link, using their unique ID number provided on the cover of the questionnaire. Conversely, physicians receiving the e-questionnaire could request a paper version. Incentives included maintenance of certification credits from the CFPC and the RCPSC for physicians who completed both the questionnaire and a related reflective exercise following the survey.
Second-year resident and MS surveys
From February to April 2007, PGY2s and MSs were contacted on 4 separate occasions, as illustrated in Figure 2,* and asked to complete their respective electronic questionnaires. E-mail invitations were forwarded by the undergraduate and postgraduate medical offices at each of the 17 medical schools, on behalf of the NPS, to ensure that all MSs and PGY2s were contacted and to protect confidentiality. These e-mail invitations also included a message from the presidents of the CFPC, CMA, and RCPSC, along with the questionnaire URL. All responses were captured confidentially and went directly into 2 separate databases, one for MSs and one for PGY2s.
Additional interventions
Marketing and communications strategy
The 2007 communications strategy included hiring a communications company from Ottawa, Ont, in the months leading up to the distribution of the 2007 survey. Promotional announcements began in the fall of 2006, with posters, brochures, and NPS-imprinted pencils being distributed at various meetings and conferences. More intense promotion began in January 2007, with advertisements and editorials in several publications and e-mail and website announcements shared among the members of the CFPC, CMA, and RCPSC.
The main strategy was to involve NPS champions (ie, influential peers) to encourage their colleagues to complete the NPS. More than a thousand champions (physicians in provincial or territorial medical associations, colleges, societies, and institutions) were asked to help promote the NPS using a promotional tool kit posted on the NPS website. It contained ready-to-use promotional articles, announcements, graphics, website buttons, and banners. Promotion among PGY2s and MSs was primarily through peer-to-peer encouragement using a similar tool kit.
Monetary incentive
The hypothesis that monetary incentives increase response rates was tested by including a considerable incentive for MSs and PGY2s. Upon confidential completion of their surveys, they were given the opportunity to submit their names and e-mail addresses, separately from their survey responses, and by so doing, their names would be entered into weekly draws for $100. The faster they completed their surveys, the sooner they could be eligible to win one of 30 cash prizes.
RESULTS
Response rates
The overall 2007 NPS response rates were compared with those from 2004 to assess the effects of the methodologic strategies outlined above.
Physician survey
Of the 62441 doctors originally identified as being in practice, a total of 1630 individuals were eliminated because they either had no known mailing address or were retired, in residency training, or working abroad. Therefore, the original study population was reduced to 60811 physicians. Of these, 19239 replied to the survey, for an overall 2007 NPS response rate of 31.64% (versus 35.85% in 2004).
Physician response rates by length of questionnaire
Thirty-eight percent of eligible respondents received the longer survey and 62% received the shorter survey, but there was no significant difference between either group’s response rates (31.7% and 31.6%, respectively), as seen in Table 1.
2007 National Physician Survey physician response rates by mode of completion and methologic strategy
Physician response rates by mode of contact
Fifty-eight percent of eligible respondents were contacted by e-mail, with a response rate of 29.9%; the 42% contacted by regular mail had a response rate of 34.1% (Table 1). The cumulative response rates by mode of contact (e-mail or paper), including which mode of contact respondents actually selected, are outlined in Figure 3. As noted in Table 1, only 2% of the physicians originally contacted by regular mail completed the questionnaire electronically. The response rate among physicians contacted in French was significantly higher than the response rate among physicians contacted in English (33% versus 31%; P <.0001).
2007 NPS physician cumulative response rates, by mode and timing of contact
Second-year resident and MS surveys
Among the MSs and PGY2s, the response rates were 30.8% and 27.9%, respectively (versus 31.2% and 35.6% in 2004).
DISCUSSION
The hypotheses that a shorter questionnaire and a greater percentage of the population being contacted electronically would increase the response rates for the 2007 NPS were not supported by our results. Additionally, the hypothesized overall response benefits of increased personalization, topical salience, and marketing of the surveys did not increase the 2007 NPS response rates. In the case of the MS and PGY2 surveys, the results clearly indicate that a monetary incentive did not increase the 2007 response rates compared with the 2004 rates of response, when no financial incentive was offered. These findings were quite unexpected. Another unexpected finding was the significantly higher response rate among physicians who received a survey in French compared with those who received one in English.
All of these results seem to contradict the findings of previous studies. For instance, the 2007 NPS used 7 of the 11 strategies for increasing response rates that Edwards et al15 found to have significant P values for heterogeneity. Further, the response rate among physicians who were contacted by e-mail in the 2004 NPS surpassed the response rate among those contacted by regular mail, yet this was not the case in 2007. It is possible that the volume of e-mails received by physicians might have increased in the 3-year period to the point where some e-mail messages are missed, filtered out, or ignored. Extensive e-mail address delivery verification was performed, so it is known that the 2007 NPS e-mails were delivered, but we cannot tell if they were immediately deleted or filtered by spam filters. However, an additional 2054 physicians did eventually reply to paper questionnaires after not replying to the e-mailed version (Figure 3).
With a 32% response rate, was the 2007 NPS a wasted effort? Not at all. While maximizing the response rate is an important aspect of survey design,1 there is no single acceptable response rate.40 Templeton et al concluded that a low response rate need not affect the validity of the data collected, as long as the nonresponse effects are documented, tested, and understood.7 The results of the 2007 NPS nonresponse analysis41 and the statistical weights applied to the responses provide reassurance that the 2007 NPS responses are generalizable to the total Canadian physician population.
Given the continuing decline in physician response rates, should the NPS abandon the self-reporting survey methodology? Barclay et al contend that well-designed surveys will remain an important part of primary care research and development,12 and in early 2008, Burns et al published a guide for the design and conduct of self-administered surveys of clinicians, indicating that they see a continuing need to survey clinicians.33
Instead of abandoning the long-standing survey methodology, other issues affecting surveys should be investigated. Accreditation or some form of incentive or payment for completing surveys might be necessary.11,36,42 The overarching national scope of the NPS initiative might also possibly limit the immediate salience9,11 of this survey for all practising physicians. Forty-eight percent of respondents to the 2004 NPS follow-up survey of nonrespondents indicated that they weren’t sure if they had received the original questionnaire, and 17% did not recall receiving it at all. To reduce the overall survey burden on physicians,3,11 perhaps all surveys targeting physicians in Canada could be coordinated, although this would likely prove impossible to implement. McAvoy et al, however, make a very interesting conclusion: “It needs to be said that the routine discarding of questionnaires without answering them may lead to a weakening of general practitioners’ power to influence service planning and provision.”3 There are many strategies that can (and should) be investigated to increase the NPS response rates (and physician response rates to surveys in general). Cost-effectiveness analyses are needed to investigate the feasibility of including a monetary incentive for such a large study population. In addition, although the NPS has been designed as a census survey for a number of important reasons that go beyond purely methodologic concerns, perhaps a well-designed sample survey with considerable follow-up of nonrespondents is a feasible alternative for the future.
Conclusion
While the objective of achieving an overall response rate of at least 50% for the 2007 NPS was not achieved, nearly 20 000 physicians across Canada, (and more than 700 second-year medical residents and 2700 medical students) did respond to the 2007 NPS. Measures were taken to ensure the validity and generalizability of these results to the entire Canadian physician population.
New methodologic strategies were incorporated into the 2007 NPS, which were indicated by other studies as ways to improve survey response rates—a shorter questionnaire, an electronic questionnaire and electronic modes of contact, implementing more promotional approaches (communications and marketing), and increasing topical salience. All of these strategies, particularly their combined effect, were expected to increase the 2007 NPS response rates but did not prove effective. The inclusion of a monetary incentive for medical students and residents also proved unsuccessful in increasing their response rates. The role of self-administered surveys in gathering information from physicians and physicians-in-training remains a methodologic option, but is both enigmatic and problematic. Researchers need to consider alternative strategies for achieving higher response rates to surveys of these populations.
Notes
EDITOR’S KEY POINTS
-
Because of its large sample size, error rates are low for the National Physician Survey (NPS), despite response rates of only 31.64% for practising physicians, 30.8% for medical students, and 27.9% for residents.
-
The 2007 NPS used several methodologic strategies to improve the response rate, including shortened surveys, e-mailed surveys, modest lottery incentives, and multiple marketing methods. Unfortunately despite these measures, the 2007 response rate did not improve over that of the 2004 survey.
-
Why did 2 out of 3 physicians fail to complete the survey in 2007? There are likely many reasons: survey fatigue; survey loss; time; general indifference.
-
Information from the NPS is used to develop health care policy. Do you want to have an influence on health care planning and delivery? Next time the NPS comes your way, fill it out.
POINTS DE REPÈRE DU RÉDACTEUR
-
En raison de la grande taille de l’échantillon, les taux d’erreur sont faibles dans le Sondage national des médecins (SNM), malgré des taux de réponse de seulement 31.64% pour les médecins en pratique, 30.8% pour les étudiants en médecine et 27.9% pour les résidents.
-
Le SNM 2007 a eu recours à plusieurs stratégies méthodologiques pour améliorer le taux de réponse, incluant des questionnaires plus courts, des enquête par courrier électronique, des incitatifs financierd modestes sous forme de loterie et plusieurs méthodes de marketing. Malheureusement, malgré ces mesures, le taux de réponse 2007 n’était pas meilleur que celui de 2004.
-
Pourquoi 2 médecins sur 3 n’ont pas répondu au sondage en 2007? Il y a vraisemblablement plusieurs raisons: trop de sondages; questionnaires égarés: contraintes de temps; indifférence générale.
-
L’information tirée du SNM sert à élaborer des politiques de soins de santé. Voulez-vous influencer la planification et la prestation des soins de santé? La prochaine fois que vous recevrez le SNM, répondez-y.
Footnotes
-
Contributors
Ms Grava-Gubins and Ms Scott contributed to concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.
-
Competing interests
None declared
-
↵* Figure 1 and Figure 2 are available at www.cfp.ca. Go to the full text of this article on-line, then click on CFPlus in the menu at the top right-hand side of the page.
-
This article has been peer reviewed.
- Copyright© the College of Family Physicians of Canada
References
In this issue
Jump to section
Related Articles
Cited By...
- Canadian national survey of family medicine residents on point-of-care ultrasound training
- Health-related behaviours and perceptions among physicians: results from a cross-sectional study in Israel
- Student acceptability of human patient simulators in undergraduate OSCEs
- Canadian physicians knowledge and counseling practices related to antibiotic use and antimicrobial resistance: Two-cycle national survey
- Statistical Application and Cost Saving in a Dental Survey
- Prevalence of Barriers and Facilitators to Enhancing Conservative Kidney Management for Older Adults in the Primary Care Setting
- Getting by with a little help from friends and colleagues: Testing how residents social support networks affect loneliness and burnout
- Osteoporosis guideline implementation in family medicine using electronic medical records: Survey of learning needs and barriers
- Medical end-of-life practices among Canadian physicians: a pilot study
- Peritoneal Dialysis University for Surgeons: A Peritoneal Access Training Program
- Prevalence and determinants of cannabinoid prescription for the management of chronic noncancer pain: a postal survey of physicians in the Abitibi-Temiscamingue region of Quebec
- Self-Perceptions of Cultural Competence Among Dental Students and Recent Graduates
- Pediatric Residents' Perspectives on Reducing Work Hours and Lengthening Residency: A National Survey
- Practice and payment preferences of newly practising family physicians in British Columbia
- Impact of PubMed search filters on the retrieval of evidence by physicians
- Attractiveness of family medicine for medical students: Influence of research and debt
- Listeriosis in pregnancy: Survey of British Columbia practitioners' knowledge of risk factors, counseling practices, and learning needs