Abstract
Objective To study the effect of electronic medical record (EMR) implementation on preventive services covered by Ontario’s pay-for-performance program.
Design Prospective double-cohort study.
Participants Twenty-seven community-based family physicians.
Setting Toronto, Ont.
Intervention Eighteen physicians implemented EMRs, while 9 physicians continued to use paper records.
Main outcome measure Provision of 4 preventive services affected by pay-for-performance incentives (Papanicolaou tests, screening mammograms, fecal occult blood testing, and influenza vaccinations) in the first 2 years of EMR implementation.
Results After adjustment, combined preventive services for the EMR group increased by 0.7%, a smaller increase than that seen in the non-EMR group (P = .55, 95% confidence interval −2.8 to 3.9).
Conclusion When compared with paper records, EMR implementation had no significant effect on the provision of the 4 preventive services studied.
Electronic medical records (EMRs) have been identified as critical to quality improvement efforts,1–4 and policies favouring the establishment of EMRs are being implemented in the United States and in most Canadian provinces.2,5,6 In 2005, the Ontario government offered a subsidy to selected primary care physicians for the purchase of EMRs.5 Physicians who were eligible were those who had chosen a blended capitation payment system based largely on the age and sex of patients enrolled in their practices. By the end of the program in 2009, most physicians offered the subsidy had purchased EMRs.7 This provided an opportunity to compare groups of physicians in the same communities implementing and not implementing EMRs.
Although EMRs invite hope for improvement,8 there is still much that is unknown about the effect of EMRs on quality and performance. A recent systematic review found that computerized decision-support systems improved practitioner performance, especially with regard to immunizations.9 However, these systems were not commercial, off-the-shelf EMRs like those commonly found in primary care practices. Several studies have found that EMRs might not improve care10–15 and might even be a source of errors.15–17 Studies of EMR implementation have often been descriptive, with very few evaluating measurable outcomes.16 At present, the effect of EMR implementation on the quality of care in small community-based family practices is unknown.
Electronic medical record systems are complex, and their implementation involves changes to many processes; only some changes can be implemented early on. In this study, we focused on preventive services targeted by Ontario’s pay-for-performance program as markers of early progress in EMR implementation. These services were attached to financial incentives and were perceived as representing good care.17 The pay-for-performance target numbers varied by physician and were based on the percentage of enrolled family practice patients being provided with Papanicolaou smears, mammograms, influenza vaccinations, fecal occult blood screening, and primary vaccinations (in children younger than 2 years).18,19
In order to be eligible for funding in Ontario, an EMR system had to include the ability to manage the provision of such preventive services through point-of-care alerts and reminder letters. Although physicians using paper records were also offered financial incentives and were expected to produce an increase in preventive services, those using EMRs had electronic tools that could lead to a greater increase.
Our objective was to compare the differences in provision of these preventive services between community-based family physicians implementing EMRs and those continuing to use paper-based records.
METHODS
Participants
We followed 2 cohorts of physicians: a group of 18 physicians implementing EMRs and a group of 9 physicians using paper records (the non-EMR cohort). Physicians in both cohorts were community-based, were affiliated with a local hospital, and were located in the Toronto area. All were members of the same local after-hours clinic. Physicians were signed out to the clinic and took turns providing walk-in care at a single location on evenings and weekends. No preventive services were offered at the clinic and no clinic data were included in the study.
The physicians in the EMR cohort had previously participated in a pay-for-performance study,20 so data on their characteristics were already available. They switched to the blended capitation model at the end of 2004 and began EMR implementation in the first 4 months of 2006 using the same EMR software. We studied the change in preventive services in the first 2 years of EMR implementation (2006 and 2007). The principal investigator was also a participant in this study (M.G.).
We recruited the non-EMR physicians through a letter of invitation and brief survey sent to all members of the after-hours clinic who were not part of the EMR cohort (125 physicians). These physicians were not eligible for the EMR subsidy. The recruitment process is shown in Figure 1.
Recruitment process for the non-EMR cohort
EMR—electronic medical record.
Data sources
The Ontario Ministry of Health and Long-Term Care provides family physicians with a list of patients in their practices who are eligible for each of the 4 preventive care services. We selected charts from these lists, using a random numbers table, from which we recorded the following: each patient’s age and sex, the presence or absence of a service within the required time period, and any record that a reminder letter had been sent to patients overdue for a service. We entered the data into an Epi Info database.21
We examined information on factors possibly associated with the provision of preventive services.22–32 Data were obtained through a questionnaire administered to all physicians. Following the methods of Glazier et al,33 practice-level data were also derived from linked administrative databases at the Institute for Clinical Evaluative Sciences (ICES). Income quintiles were derived from 2001 census data linked to patient postal codes. Neighbourhood income quintiles and recent immigration status were derived using published methods.33,34 Morbidity and comorbidity (adjusted diagnosis groups and resource utilization bands) were derived using the Johns Hopkins Adjusted Clinical Groups software,35 available from ICES, which has been described elsewhere.36–43 Numbers and percentages of patients with chronic diseases were obtained from linked ICES health databases.38–44
We audited paper charts for physicians in the non-EMR cohort and obtained data from EMR charts for physicians in the EMR cohort. When data were unavailable in the EMR, we retrieved and obtained data from the paper chart.
Five data auditors abstracted data from both paper charts and EMRs. The research coordinator initially audited 10 charts for each service in 2 practices and reviewed this with the principal investigator (M.G.). The coordinator then trained each data auditor, and reviewed at least 10 charts for each service from each auditor. The data were collected on paper forms and entered in the Epi Info database by 2 data entry clerks. Each clerk entered data from a training sample of at least 10 charts for each service. A randomly selected 10% data sample for each service, each year, and each physician was re-audited and entered in the database; we used the κ statistic to compare the 2 audits.
Outcome measures
The study’s end point was whether or not a preventive service was provided and documented within a required time period for each eligible patient. The target patient population consisted of all eligible patients enrolled in the participating practices. Documentation that the patient received the service through another health care provider was acceptable. Information on services is presented in Table 1.
Eligibility criteria, exclusion criteria, and required period for preventive service provision in a pay-for-performance family practice setting
We calculated a composite process score44 by using the total number of charts audited for eligible patients for each service per physician as the denominator and the total number of services documented in the audits as the numerator. We could not obtain lists of eligible children for 2006 in the non-EMR cohort. As a result, children’s vaccinations were excluded from the composite score.
We compared the change in the proportion of patients who received preventive services between the EMR and non-EMR cohorts during EMR implementation. A 5% increase was considered the minimum clinically important difference.
Sample size calculation
We calculated the sample size required to have an 80% power to detect a clinically important increase in service provision of 5% or higher20 in the first 2 years of EMR implementation (assuming relatively stable rates in the non-EMR cohort and with rates of influenza vaccination ranging from 83% to 88%) using an α level of .05, and determined that 40 charts per service per provider would be required. To further increase statistical power, we audited 50 charts per year per service per physician. Physicians who practise in groups might influence one another; this would reduce the variation in practice and might inflate the observed effect.45 Based on our previous pay-for-performance study,20 the estimated intraclass correlation coefficient46 due to clustering of physicians within practices was 0.01. If we assume that, on average, the recruited physicians were clustered within groups of 4 physicians, and that the intraclass correlation coefficient was 0.01, the inflation factor would be 1.03 ([1 + (4 - 1) × 0.01]), which should have a negligible effect on the sample size required.
Analysis
We compared unadjusted percentages using χ2 tests and compared adjusted percentages using multivariate logistic regression analysis. We used generalized estimating equations to adjust for the clustering structure of the data in regression models. When comparing the 2 groups, we adjusted for patient age for each individual patient,47 physician sex,20 years since each physician’s medical school graduation,22 and Certification in Family Medicine (ie, CCFP) status26 using multivariate logistic regression analysis. Administrative data could not be used for adjustment, as these data were not available to us at the patient level. Percentage of fecal occult blood tests provided might have been different from the other services provided; the provincial provision rate was low and there was a concurrent public health campaign in 2007.48,49 As a result, we re-analyzed the 2 cohorts with this screen excluded.
As the principal investigator was also a participant in the study, a sensitivity analysis was performed by excluding her practice data.
We performed analyses with SAS software, version 9.1. All tests were 2-sided using an α level of .05. The study was approved by the University of Toronto’s Research Ethics Board; the Sunnybrook Research Institute’s Research Ethics Board approved the use of ICES data. All participating physicians provided written informed consent.
RESULTS
Characteristics of the EMR and non-EMR physicians and their practice populations are presented in Table 2 and Table 3,22, 34–43 respectively. Physician characteristics between the 2 cohorts were similar. Patients in the EMR cohort were slightly younger and had lower levels of morbidity and comorbidity.
Characteristics of physicians in the EMR and non-EMR cohorts
Comparison of patient characteristics between the EMR and non-EMR cohort practice populations as of August 31, 2007
Results for individual services are presented in Table 4. Nine hundred charts were audited for each service for each year in the EMR cohort and 450 charts were audited in the non-EMR cohort. The number of Pap smears and mammograms increased more in the EMR cohort, while the rate of influenza vaccinations and fecal occult blood testing increased more in the non-EMR cohort. Composite process scores are shown in Table 5.
Service provision for individual services in the EMR and non-EMR cohorts before and after EMR implementation
Changes in composite process scores for the provision of preventive services between EMR and non-EMR cohorts
Female physician sex and younger patient age were positively correlated with the likelihood of receiving a service. There was no correlation with years since medical school graduation or Certification status. There was no significant difference in the change in service provision between the 2 groups; differences and confidence intervals were less than 5%. The results were not affected by the exclusion of fecal occult blood screening data.
Physicians in the EMR cohort mailed letters to 23 patients overdue for services in 2005, 265 patients in 2006, and 677 patients in 2007. Physicians in the non-EMR cohort mailed 1 letter in 2006 and none in 2007.
The results were unaffected by the exclusion of the principal investigator’s data. The κ statistic for the 10% of charts that were re-audited was 0.954, consistent with an acceptable level of agreement.
DISCUSSION
We found no statistically significant or clinically important difference in the change in preventive service provision between physicians implementing EMRs and those continuing to use paper records. Other studies have found no difference in the change in care of diabetes between EMR- and paper-based practices11 and no improvement when additional experience with EMR accrues over time.11,13,15
The EMR system we studied included elements that have been associated with an increase in the provision of preventive services, such as point-of-care alerts9,50,51 and the ability to generate reminder letters to patients.52 However, studies have shown that alerts or patient reminders for overdue services, even if available, might not be consistently implemented in practice.11,15,53,54 We found evidence of reminder letter mailings in the EMR cohort, but this did not appear to have had an effect on service uptake.
To frame and explain our results, a qualitative study presented in an accompanying article55 explores the context of this EMR implementation. Similar studies could help to determine what aspects of EMR implementation are associated with improved quality of care.
Limitations
This study was limited to a group of selected physicians in Toronto. However, all physicians in this study were practising in community-based settings, similar to most family physician settings in Ontario.56 Physicians in our cohorts were similar to their colleagues in capitated and reformed fee–for-service models in Ontario urban centres.33
Nineteen percent of physicians using paper records responded to the study invitation; we do not know if their characteristics differed from those of the nonresponders. The participating physicians provided preventive services to a large proportion of their patients. Increases might have been limited by ceiling effects. Also, we studied a single EMR system; results might differ for physicians using other EMR systems.
This was an observational cohort study, and is therefore subject to both measured and unmeasured confounders. In this study, we measured confounders that have been reported in the literature to affect preventive services22–32,47,57,58 and used statistical adjustments to adjust for these factors. There were differences between our 2 cohorts in terms of physician funding mechanisms. However, a recent study using administrative data did not find any consistent difference in the provision of preventive services between Ontario physicians in capitated or reformed fee-for-service groups.59
Conclusion
Among the family physician practices studied, the first 2 years of EMR implementation were not associated with an increase in the provision of preventive services targeted by Ontario’s pay-for-performance program when compared with the continued use of paper records. It should not be assumed that EMR implementation improves care.
Acknowledgments
This study was funded by a research grant from the Ontario Ministry of Health and Long-Term Care Health System Strategy Division (grant no. 020709) and was supported by the Institute for Clinical Evaluative Sciences. The opinions, results, and conclusions reported in this paper are those of the authors and do not imply endorsement by the Ministry of Health and Long-Term Care or the Institute for Clinical Evaluative Sciences.
Notes
EDITOR’S KEY POINTS
-
One of the drivers of electronic medical record (EMR) implementation is the hope that EMRs can improve quality of care for patients.
-
The authors examined preventive services with pay-for-performance incentives that should have improved with EMR use owing to electronic tools that assist in timely service provision (eg, alerts, automated reminders).
-
There was no difference in service provision between physicians using EMRs and those continuing to use paper records (in the first 2 years of EMR implementation).
-
It should not be assumed that EMR implementation improves care.
POINTS DE REPÈRE DU RÉDACTEUR
-
Une des raisons qui incitent à adopter les dossiers médicaux électroniques (DMÉ) est l’espoir qu’ils peuvent améliorer la qualité des soins.
-
Les auteurs ont examiné les services préventifs qui, avec une rémunération au rendement comme mesure incitative, auraient dû s’améliorer avec les DMÉ grâce aux outils électroniques qui facilitent la prestation de ces services en temps opportun (p. ex. signaux d’alerte, rappels automatisés).
-
Il n’y avait pas de différence de prestation des services entre les médecins utilisant les DMÉ et ceux qui continuaient d’utiliser les dossiers papiers (durant les 2 ans suivant l’introduction des DMÉ).
-
On ne devrait pas présumer que la mise en œuvre des DMÉ améliore les soins.
Footnotes
-
This article has been peer reviewed.
-
Cet article a fait l’objet d’une révision par des pairs.
-
Contributors
All authors contributed to the concept and design of the study. Drs Greiver and Barnsley contributed to the data gathering. All authors contributed to the analysis and interpretation of the results and the preparation of the manuscript for publication.
-
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
References
In this issue
Jump to section
Related Articles
Cited By...
- Demandes concurrentielles et possibilites en soins primaires
- Competing demands and opportunities in primary care
- Implementation of data management and effect on chronic disease coding in a primary care organisation: A parallel cohort observational study
- What if electronic medical records were unnecessary?
- Les dossiers medicaux electroniques ameliorent-ils la qualite des soins?: Non
- Do electronic medical records improve quality of care?: No
- Implementation of electronic medical records: Theory-informed qualitative study