Abstract
Objective To determine which screening tests family medicine residents order as part of preventive health care.
Design A cross-sectional survey.
Setting Alberta and Ontario.
Participants First- and second-year family medicine residents at the University of Alberta in Edmonton, the University of Calgary in Alberta, and McMaster University in Hamilton, Ont, during the 2011 to 2012 academic year.
Main outcome measures Demographic information, Likert scale ratings assessing ordering attitudes, and selections from a list of 38 possible tests that could be ordered for preventive health care for sample 38-year-old and 55-year-old female and male patients. Descriptive and comparative statistics were calculated.
Results A total of 318 of 482 residents (66%) completed the survey. Recommended or appropriate tests were ordered by 82% (for cervical cytology) to 95% (for fasting glucose measurement) of residents. Across the different sample patients, residents ordered an average of 3.3 to 5.7 inappropriate tests per patient, with 58% to 92% ordering at least 1 inappropriate test per patient. The estimated average excess costs varied from $38.39 for the 38-year-old man to $106.46 for the 55-year-old woman. More regular use of a periodic health examination screening template did not improve ordering (P = .88).
Conclusion In general, residents ordered appropriate preventive health tests reasonably well but also ordered an average of 3.3 to 5.7 inappropriate tests for each patient. Training programs need to provide better education for trainees around inappropriate screening and work hard to establish good ordering behaviour in preparation for entering practice.
Preventive health care (PHC) is an integral part of primary care practice. Unfortunately, previous studies have shown that physicians frequently do not meet recommendations for PHC.1–3 Failing to order appropriate tests could lead to missed opportunities to reduce morbidity and mortality, while ordering inappropriate tests could increase costs and potentially cause unintended harm. Therefore, learning and implementing appropriate PHC screening in training is essential in order to ingrain good PHC practice early.
A few studies have investigated residents’ recommendation of 2 to 3 appropriate PHC screening tests and found colon cancer, breast cancer, and lipid screening rates were generally between 80% and 90%.4,5 Another study examined residents’ routine ordering of 3 inappropriate tests and found approximately 5% ordered routine electrocardiograms, 12% ordered complete blood counts (CBCs), and 51% ordered urinalysis.6 We could not identify any studies that examined residents’ ordering of a broad range of tests for PHC.
While undoubtedly time requirements present a challenge to PHC,7 some evidence suggests that residents might lack understanding of some aspects of PHC, such as colon cancer screening.8 It is important that we begin to assess what family medicine residents believe to be appropriate test ordering for PHC so that we can determine if deficits exist and modify our training programs if necessary. This could in turn help residents improve their performance for PHC and enhance care for patients. The selective use of tests will not only improve screening but also prevent unnecessary testing and the potential consequences of those errors while reducing costs.
Our primary objective was to determine what tests residents thought were appropriate for PHC screening. Secondary objectives were to assess attitudes around the use of PHC screening templates in practice and any association between test ordering and demographic characteristics or attitudinal responses.
METHODS
Survey development
We used the Dillman total design method9 to develop and distribute a questionnaire. We collected demographic data on residency year (first or second), program location (rural or urban), university (McMaster University, University of Calgary, or University of Alberta), sex, and location of medical school graduation (Canadian or international).
To assess residents’ attitudes about test ordering and the use of PHC checklists or templates, we asked 5 questions:
-
When available, do you follow a template for routine health examinations?
-
Do you feel you are ordering less, the same, or more than a template (if available)?
-
Do you think the template you are using encourages you to order less, the same, or more?
-
Do you feel you are ordering less, the same, or more than your preceptor?
-
After graduation do you feel you would order less, the same, or more than as a resident?
Answers were provided on a 5-point Likert scale from 1 meaning “never” or “a lot less” to 5 meaning “always” or “a lot more.” We also asked residents if they would order any “baseline” screening tests and what they would be.
In order to more broadly assess residents’ PHC test ordering for adults we wanted sample middle-aged patients who would require little to no screening, as well as slightly older patients who would meet the criteria for most adult screening maneuvers. Our final group included 4 patients: female and male 38-year-old patients, as well as female and male 55-year-old patients. We also wanted to offer residents a large choice of tests to select from. The 38 investigations listed included 29 blood tests, 3 urine tests (urine analysis, microalbumin-creatinine ratio, and chlamydia and gonococcus screening), 3 imaging tests (chest x-ray scan, mammogram, and bone mineral density testing), electrocardiogram, cervical cytology, and fecal occult blood testing. We used the Preventive Care Checklist10,11 endorsed by the College of Family Physicians of Canada as our criterion standard for appropriate screening tests.
Together, D.F. and G.M.A. developed the survey. It was then distributed to the other clinician researchers (I.S., C.A.M., F.M.A., and A.K.) and modifications were made based on their suggestions. It was then reviewed by 3 practising clinicians to ensure clarity. The final distributed survey is available upon request.
Survey distribution
During the 2011 to 2012 academic year, we surveyed all family medicine residents from 3 university programs in Ontario and Alberta: McMaster University in Hamilton, Ont, the University of Calgary in Alberta, and the University of Alberta in Edmonton. The first distribution of the survey was done in person at a resident meeting day, such as an academic session or special program-wide training seminar. The local resident involved in the project (A.K., D.F.) performed the in-person distribution for their programs, except at McMaster where the faculty member (I.S.) filled in, as the resident (F.M.A.) was unavailable. Residents who did not respond to the first distribution received the survey up to 2 more times via e-mail at 6-week intervals. E-mail messages linked to a SurveyMonkey version of the questionnaire. To improve the response rate, residents completing the survey were entered into a draw for a chance to win 1 of 7 restaurant gift cards worth $25.
Ethics approval was obtained from the ethics boards of all 3 participating universities.
Data analysis
Descriptive statistics were performed for demographic characteristics, attitudinal questions, and ordering of tests. In order to broadly explore potential associations, we performed a large number of statistical comparisons (124 total). To reduce the risk of spurious results, we used an α value of 0.01 for statistical significance.
In general, Likert scale outcomes were analyzed using either the t test, for dichotomous comparisons, or ANOVA (analysis of variance), for comparisons with more than 2 categories. Fisher exact test was used to compare 2 dichotomous outcomes. The Cochran-Armitage trend test (for the single appropriate test in the 38-year-old female) and Jonckheere-Terpstra tests (all other comparisons) were used to assess if ratings on attitudinal Likert scales were associated with appropriate or inappropriate ordering in any of the 4 patients or in all 4 combined.
We also examined the costs of test ordering, both appropriate and inappropriate. We used full cost recovery pricing (including everything from technician time to tray fee to floor cleaning) attained from Alberta Health Services laboratory and the Alberta Health fee guide. Specific cost information is available from the authors on request.
In 2 cases, tests might have overlapped. A CBC with or without white blood cell (WBC) differential would include measurement of hemoglobin level, and electrolyte measurement would include sodium and potassium levels. We performed a sensitivity analysis of the number of tests ordered when hemoglobin was excluded if CBC with or without WBC differential was also ordered, and sodium and potassium were excluded when electrolyte measurement was ordered.
A post hoc analysis was performed to determine if one test (a sentinel) might identify residents more likely to order inappropriate tests. We compared the number of inappropriate tests ordered by residents ordering a CBC with or without WBC differential in a 38-year-old man with those who did not.
RESULTS
Of the potential 482 family medicine residents surveyed at the 3 universities, 318 (66%) responded. Demographic details of the residents are provided in Table 1.
For the 4 possible patients, residents ordered an average of 26.7 tests, 8.4 of which were appropriate and 18.3 of which were inappropriate. The proportions of appropriate tests ordered are provided in Figure 1. Depending on the test, 82% to 96% of residents ordered the appropriate tests.
For the 38-year-old female patient, residents on average ordered 4.8 tests, 4.0 of which were inappropriate. For the 38-year-old male patient, residents on average ordered 3.3 tests, all inappropriate. For the 55-year-old female patient, residents ordered 10.4 tests, 5.7 of which were inappropriate. For the 55-year-old male patient, residents ordered 8.2 tests, 5.3 of which were inappropriate. At least 1 inappropriate test was ordered by 73% of residents for the 38-year-old female patient, 58% of residents for the 38-year-old male patient, 92% of residents for the 55-year-old female patient, and 89% of residents for the 55-year-old male patient.
The most common inappropriate tests, by patient type, are provided in Table 2. A CBC with or without WBC differential and measurement of thyroid-stimulating hormone, creatinine, and electrolyte levels were consistently ordered excessively for all 4 patients. No demographic variable had a statistically significant association with the average number of inappropriate tests ordered for all 4 patients (Table 3). Demographic variables did not have a statistically significant association with the ordering of inappropriate tests for any of the 4 individual patients either (data not shown, all P values ≥ .03). In terms of appropriate tests, female residents were more likely than male residents (90% vs 74%) were to order cervical cytology (the only appropriate test) for the 38-year-old woman (P < .001). Otherwise, no demographic variable had a statistically significant association with the ordering of appropriate tests for any individual patient or all patients combined (data not shown, all P values ≥ .4).
Ordering inappropriate tests led to a mean (SD) excess cost of $66.53 ($66.76) per patient. Unnecessary laboratory costs accounted for a mean (SD) of $51.29 ($44.01) of the excess costs, with unnecessary radiology accounting for a mean (SD) of $15.23 ($44.30) of the remainder. By patient, residents’ mean (SD) excess cost from unnecessary testing was $57.34 ($51.93) for the 38-year-old woman, $38.39 ($42.03) for the 38-year-old man, $106.46 ($88.68) for the 55-year-old woman, and $63.93 ($55.30) for the 55-year-old man.
A few of the demographic characteristics seemed to influence the scores on attitudinal Likert rankings (Table 4). However, of the 4 statistically significant results, 3 involved a difference of less than 0.5 in the mean Likert rankings; while these differences are statistically significant, they are likely not meaningful differences. McMaster residents had an average Likert score for following a template of 4.16 compared with residents from the University of Calgary at 3.58 and the University of Alberta at 3.17 (P < .001).
Scores on attitudinal Likert rankings were not associated with appropriate ordering for individual patients or for all patients combined (data not shown, P ≥ .09). Inappropriate test ordering by attitudinal Likert rankings is provided in Table 5. Residents scoring higher for “Do you feel you are ordering less, the same, or more than a template?” were more likely to order more inappropriate tests than those who scored the question lower (P = .001 for all 4 patients together). Residents scoring higher for “Do you feel you are ordering less, the same, or more than your preceptor?” were more likely to order more inappropriate tests than those who scored the question lower (P = .01 for all 4 patients together).
In the sensitivity analysis excluding all potential duplicates, residents ordered 24.0 tests for all 4 patients; 8.4 were appropriate and 15.5 were inappropriate. Thus, duplicate ordering would have accounted for 2.8 inappropriate tests (18.3 minus 15.5) over 4 patients.
Residents who ordered a CBC with or without WBC differential for the 38-year-old man ordered a total of 28.3 additional inappropriate tests, which was significantly more than the 8.5 total inappropriate tests ordered by residents who did not order the CBC with or without WBC differential for the 38-year-old man (P < .001).
DISCUSSION
Approximately 90% of family medicine residents from 3 Canadian universities indicated they would order the appropriate screening tests for 38-year-old and 55-year-old female and male patients. Although 17% to 18% of residents did not indicate they would order cervical cytology for 38-year-old and 55-year-old women, it is possible that some did not because guidelines now recommend the test every 3 years (assuming no past positive results). There was no association between demographic characteristics or attitudinal rankings except that female residents were more likely than male residents were to order cervical cytology for the 38-year-old woman. Overall, it appears that most residents understand and order the appropriate, recommended screening tests.
Perhaps the more important result was the pervasive excess ordering of inappropriate and unnecessary tests. For the 38-year-old male patient, 58% of residents ordered at least 1 inappropriate test, and for the 55-year-old female patient, 92% ordered at least 1 inappropriate test. The mean number of inappropriate tests per patient varied from 3.3 to 5.7, which means additional unnecessary costs of $38.39 to $106.46. Taken over millions of patients per year, the inappropriate testing would result in multiple false-positive results, unnecessary worry, and obligatory follow-up testing, and would likely cost hundreds of millions of wasted dollars.
As templates and checklists, frequently integrated into electronic health records, become more common, we wondered if these might be having an effect on preventive health test ordering. That was not the case. Further, although McMaster residents reported using templates significantly more often (P < .001), this did not reduce inappropriate testing. In some cases, the templates themselves are generated by the clinic or clinician and are likely not following appropriate testing recommendations. Additionally, as shown in our study, clinicians might have recognized that they were ordering more than their colleagues were or than what was recommended by checklist templates but did so anyway. Residents who thought they were ordering more than templates recommended or preceptors ordered were ordering approximately 6 to 8 tests more (over 4 patients) than those who thought they were ordering fewer. Although residents in second year, those in rural programs, and those from the University of Alberta ordered more inappropriate tests, none of the differences was statistically or meaningfully different (average difference < 1 extra inappropriate test per patient).
Past research suggests residents might have lower adherence to preventive health maneuvers than nurse practitioners do,12 but in our study their adherence to recommended maneuvers was quite good. Zallman and colleagues6 found residents were at least as likely to perform preventive health maneuvers as staff physicians were.6 Our research suggests that residents have a pretty good grasp of appropriate test ordering for PHC, and only minor efforts would be needed to enhance uptake further.
The bigger issue by far is the excessive ordering of inappropriate tests. The reasons for inappropriate test ordering by residents likely begin with medical school education. However, residency training is certainly not improving things, as inappropriate ordering was the same or worse in second year. Different schools and different programs (rural vs urban) did not fare any differently, meaning no group is immune. Is this a problem with education within programs or with preceptors and faculty not serving as adequate role models? It is likely both.
Why might it be that physician teachers are not helping learners reduce inappropriate test ordering? The messaging around the inappropriate nature of tests like urinalysis and CBC has been clear for decades.
Despite this, recent research shows each of these tests is ordered in 18% to 56% of ambulatory visits for PHC.13,14 So, what drives clinicians to order tests that are inappropriate? There is a long list of factors contributing to inappropriate ordering, including patient requests,15 fear of missing something,16 being in solo practice,16 and fear of litigation.16 As the causes are multifactorial, likely the solutions are too. Multiple studies have shown moderate reduction in unnecessary testing with interventions like clinical outreach facilitation,17 decision support,18 and feedback.19 For those who have graduated and are in practice, sentinel criteria (like CBC with or without WBC differential) might help identify those most likely to order excessively and help create practice quality improvement initiatives. New initiatives like Choosing Wisely20 will provide further incentive to reduce inappropriate ordering. Regardless, as shown in our study and other studies,14 the costs and potential cost savings, particularly over the broad population, are huge. Further research into practising physicians’ preventive health screening habits nationally could help identify the scope and magnitude of overordering and the effects on the health care system and, more important, on patient care. This type of research is both lacking and needed.21
Strengths and limitations
Strengths of this project included a survey response rate of 66%. Also, residents across multiple residency training programs in a variety of settings (rural and urban) helped make the results more generalizable. Residents could choose from a broad range of tests, including radiologic investigations, for multiple patient types. Limitations included the fact that survey answers might not reflect actual ordering practices. Respondents’ comments showed that there was confusion as to how screening was defined, affecting tests ordered. Although a possible limitation, this indicates that further education on screening maneuvers versus diagnostic testing might be needed.
Conclusion
Family medicine residents order appropriate tests for preventive health screening quite well, but they also order additional inappropriate tests that can affect not only patient care but also cost to the health care system. Templates did not reduce inappropriate ordering and likely do not follow screening recommendations. Clearly, residency programs, teachers, and preceptors need to provide better education, and likely examples, for residents and learners.
Acknowledgments
We thank Sharon Nickel for her grammatical editing of the first draft. We also thank Dr Christopher Fung for his help identifying costs of imaging investigations and Valerie Taylor for help with laboratory costs.
Notes
EDITOR’S KEY POINTS
-
Preventive health care (PHC) is integral to primary care, but physicians frequently do not meet PHC recommendations. Failing to order appropriate tests can lead to missed opportunities to reduce morbidity and mortality, and ordering inappropriate tests can increase costs and potentially cause unintended harm. It is essential that residents learn appropriate PHC screening in training. This study aimed to assess residents’ PHC test ordering.
-
The authors found that family medicine residents order recommended screening tests appropriately but also excessively order unnecessary screening tests (average of 4.6 inappropriate tests per patient). The excess ordering leads to unnecessary costs of between $38.39 and $106.46 for sample 38-year-old and 55-year-old male and female patients.
-
The use of templates for the periodic health examination does not improve ordering. Teachers and preceptors should provide more education and positive role modeling to improve test ordering for PHC.
Footnotes
-
This article is eligible for Mainpro-M1 credits. To earn credits, go to www.cfp.ca and click on the Mainpro link.
-
La traduction en français de cet article se trouve à www.cfp.ca dans la table des matières du numéro de mars 2015 à la page e165.
-
Contributors
All authors made substantial contributions to the study design, writing, and editing. Dr Fung conceived the study, drafted the survey, worked on ethics approval, distributed the survey, organized the study, analyzed results, and edited the draft. Dr Schabort edited the survey, worked on ethics approval, distributed the survey, analyzed results, and edited the draft. Dr MacLean edited the survey, worked on ethics, analyzed results, and edited the draft. Dr Asrar edited the survey, distributed the survey, analyzed results, and edited the draft. Dr Khory edited the survey, distributed the survey, analyzed results, and edited the draft. Mr Vandermeer analyzed the results and edited the draft. Dr Allan drafted the survey, worked on ethics approval, organized the study, analyzed results, and wrote the first draft.
-
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
References
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.