Abstract
Objective To develop and test a variety of electronic medical record (EMR) search algorithms to allow clinicians to accurately identify their patients with asthma in order to enable improved care.
Design A retrospective chart analysis identified 5 relevant unique EMR information fields (electronic disease registry, cumulative patient profile, billing diagnostic code, medications, and chart notes); asthma-related search terms were designated for each field. The accuracy of each term was tested for its ability to identify the asthma patients among all patients whose charts were reviewed. Increasingly sophisticated search algorithms were then designed and evaluated by serially combining individual searches with Boolean operators.
Setting Two large academic primary care clinics in Hamilton, Ont.
Participants Charts for 600 randomly selected patients aged 16 years and older identified in an initial EMR search as likely having asthma (n = 150), chronic obstructive pulmonary disease (n = 150), other respiratory conditions (n = 150), or nonrespiratory conditions (n = 150) were reviewed until 100 patients per category were identified (or until all available names were exhausted). A total of 398 charts were reviewed in full and included.
Main outcome measures Sensitivity and specificity of each search for asthma diagnosis (against the reference standard of a physician chart review–based diagnosis).
Results Two physicians reviewed the charts identified in the initial EMR search using a standardized data collection form and ascribed the following diagnoses in 398 patients: 112 (28.1%) had asthma, 81 (20.4%) had chronic obstructive pulmonary disease, 104 (26.1%) had other respiratory conditions, and 101 (25.4%) had nonrespiratory conditions. Concordance between reviewers in chart abstraction diagnosis was high (κ = 0.89, 95% CI 0.80 to 0.97). Overall, the algorithm searching for patients who had asthma in their cumulative patient profiles or for whom an asthma billing code had been used was the most accurate (sensitivity of 90.2%, 95% CI 87.3% to 93.1%; specificity of 83.9%, 95% CI 80.3% to 87.5%).
Conclusion Usable, practical search algorithms that accurately identify patients with asthma in existing EMRs are presented. Clinicians can apply 1 of these algorithms to generate asthma registries for targeted quality improvement initiatives and outcome measurements. This methodology can be emulated for other diseases.
Asthma is the third most common chronic disease in adults in Canada. It affects 8.1% of the population, is increasing in prevalence,1,2 and carries an annual economic burden of $1.8 billion.3 Although several international bodies have produced evidence-based asthma diagnosis and management guidelines,4 care gaps in asthma management remain prevalent, with 53% of Canadian patients having poorly controlled disease according to guideline criteria.5 Key evidence-based care gaps responsible for this poor control are the underrecognition of suboptimal asthma control by both physicians and patients5–9; undertreatment of asthma5–7; and clinician failure to provide patients with a written asthma action plan.5,9,10
Given that most patients with asthma are seen in primary care,11 primary care–based quality improvement might bridge these gaps. Electronic medical record (EMR) systems play an increasingly vital role in primary care12,13 and have the potential to improve the safety and quality of care, reduce costs, and facilitate patient engagement. To realize these benefits, clinicians require the ability to compile valid and reliable disease-specific registries of patients to target for quality initiatives.14–16 Such registries could facilitate EMR-based quality improvement strategies. For example, decision support integrated into the charts of patients with asthma could prompt clinicians about poor asthma control, provide guideline-based medication recommendations, and automatically fill in an electronic asthma action plan. To date, accurate and practical methods for practitioners to identify patients with asthma from their EMRs have not been presented.
We sought to develop and determine the accuracy of EMR-based search algorithms that would enable clinicians to easily and reliably identify patients with asthma within their practices, to optimize their care.
METHODS
This was a retrospective chart analysis conducted at 2 academic primary care clinics in Hamilton, Ont, with a total of 33 staff physicians (with rotating McMaster University family medicine residents) working in a capitated payment model with 27 300 registered patients. Clinics used the open-source Oscar EMR system, which is used by 1500 clinicians in the care of 2 million patients across Canada (http://oscarcanada.org).17
The study was approved by research ethics boards at McMaster University and St Michael’s Hospital in Toronto, Ont. We retrieved charts of relevant patients aged 16 years and older, registered under any of 14 consenting physicians.
Search strategy design
Based on previous literature18,19 and our clinical expertise, we identified 5 unique EMR information fields that could be used to determine asthma disease status, and search parameters to identify patients with asthma within each field (Table 1). We defined additional parameters expected to exclude patients with chronic obstructive pulmonary disease (COPD) (Table 1).
The EMR search fields and parameters
To evaluate the accuracy of search algorithms in distinguishing asthma from other respiratory conditions, we identified patients who had a higher likelihood of having respiratory disease than the general population.20–22 We created approximately equal cohorts of patients likely to have each of the following conditions: asthma, COPD, other respiratory conditions (ie, not asthma or COPD), and nonrespiratory conditions. Patients with COPD and with other respiratory conditions were included to ensure that algorithms could differentiate asthma from these clinically similar conditions, whereas patients with nonrespiratory conditions acted as healthy controls. For simplicity, these patients were limited to those with hypertension or musculoskeletal disorders.21 We first identified patients with a high likelihood of carrying 1 of these 4 diagnoses. For possible COPD, other respiratory conditions, and nonrespiratory conditions, we identified patients with relevant diagnoses in the electronic disease registry section of the EMR or a relevant corresponding billing diagnostic code billed within the past 3 years. Diagnoses listed in the electronic disease registry and the Ontario Health Insurance Plan billing codes used to identify patients are available from CFPlus.* For possible asthma, in addition to the above strategies, we identified patients who had been prescribed an inhaled asthma medication within the past 12 months (available from CFPlus*), while excluding patients who had been prescribed tiotropium bromide or ipratropium bromide (medications used predominantly for COPD). Any patients who fulfilled criteria to be included in more than 1 category were placed in the category identified by the most recent relevant billing code or prescription.
After identifying all potential patients within each of these 4 diagnostic categories, we used a random number generator to choose 150 patients in each (600 total) for review, stratified by clinic site and by physician (Figure 1).
Chart identification and review methodology
COPD—chronic obstructive pulmonary disease.
*Excluding patients prescribed tiotropium bromide or ipratropium bromide.
Chart review
Chart analysis was performed in a random order with respect to diagnostic category by 2 physicians (N.X., R.W.) in the family medicine training program. Each reviewer completed a standardized data collection form to determine which of the 4 diagnostic categories each patient actually belonged to (reference standard), according to (in order of preference) pulmonary function (where available),23,24 non–family physician specialist opinion (where available), and clinical diagnosis. A random sample of 20.0% of charts was analyzed by both reviewers to determine interrater reliability.
Reviewers placed each patient into 1 of the 4 diagnostic categories, until 100 patients per category were identified (or until all available names were exhausted). Patients with asthma and/or COPD who also had other respiratory or nonrespiratory conditions were categorized as asthma, COPD, or asthma and COPD, and patients with both other respiratory and nonrespiratory conditions were categorized as other respiratory. Uncertain cases were resolved by a consensus committee consisting of the chart reviewers, a general practitioner (G.A.), and a respirologist (S.G.).
Algorithm testing
We searched for each of the previously identified asthma parameters (Table 1) in the EMR information fields of all reviewed charts. We repeated these searches with exclusion criteria designed to eliminate patients with COPD (Table 1). Next, we searched 2 fields at a time by connecting any 2 individual searches with an “or” operator, and tested all possible combinations of individual searches. Based on these results, we proceeded to serially combine previous searches, using “or” combinations to increase sensitivity, and/ or “and” combinations to increase specificity. Each algorithm was run separately by 2 study personnel to ensure identical results, and a random sample of 20 of the charts in each algorithm was verified to ensure that searches were accurate.
Analysis
We calculated the interrater reliability of chart abstractor diagnosis using a κ statistic. We compared patient characteristics using the Student t test for continuous variables and the χ2 test for categorical variables. Using chart abstractor diagnosis as the reference standard, we calculated true-positive, true-negative, false-positive, and false-negative rates; the sensitivity and specificity of each search; and the Youden index (J = sensitivity + specificity - 1). We also performed a discordance analysis (an in-depth chart review to identify reasons for misclassification in patients whose test results were falsely positive or falsely negative) in the algorithm with the highest Youden index. We calculated 95% CIs for test characteristics, assuming a binomial distribution. To maximize sensitivity, patients who had both asthma and COPD were categorized as asthma. We used SAS, version 9.3, for all analyses.
RESULTS
Study population
Reviewers assessed 460 charts, of which 41 (8.9%) had no available data, 21 (4.6%) were duplicates, and 398 (86.5%) were fully reviewed. These 398 patients consisted of 112 (28.1%) with asthma, 81 (20.4%) with COPD, 104 (26.1%) with other respiratory conditions (neither asthma nor COPD), and 101 (25.4%) with nonrespiratory conditions. Nine of 112 (8.0%) asthma patients had coexisting COPD. Concordance between the 2 reviewers in chart abstraction diagnosis was high (κ = 0.89, 95% CI 0.80 to 0.97) (Table 2).
Patient characteristics
Search algorithms
Results from each unique search query and the 5 algorithms with the best test characteristics (based on the Youden index) are presented in Table 3. All tested search algorithms as well as search characteristics by clinic are available upon request. True-positive and false-negative rates for these searches and for the most sensitive and most specific individual algorithms are represented in Figure 2.
Operating characteristics of individual search queries, and most specific, most sensitive, and most accurate search algorithms: Operating characteristics are represented for each individual search query and the 5 most accurate queries (as presented in Table 3). The most specific tested algorithm was already represented in Table 3 (asthma in the disease registry). The most sensitive tested algorithm was not represented in Table 3, and was added here for purposes of comparison (11). Error bars represent 95% CIs. Algorithms with a high true-positive rate and a low false-negative rate are favoured (upper left-hand corner).
Results from each unique search query and the 5 algorithms with the best test characteristics (based on the youden index)
Discordance analysis
The algorithm combining asthma in the cumulative patient profile (CPP) or use of billing code 493 (asthma or allergic bronchitis) had the highest Youden index (Table 3) and was used for the discordance analysis. There were 11 false-negative results and 46 false-positive results. Among the 11 false-negative results, 4 charts (36.4%) simply did not have asthma in the CPP and had not been billed for asthma despite clear chart documentation of asthma. Of the remaining 7 charts, 6 (85.7%) had been diagnosed with asthma by an outside specialist. In these cases, we suspect that clinicians might have been less likely to update the CPP because the diagnosis was made elsewhere. Also, 6 of these 7 had COPD in addition to asthma, and clinicians appeared to default to using COPD rather than asthma billing codes for respiratory-related visits in these patients. Among the 46 false-positive results, 14 (30.4%) had COPD as opposed to asthma, and clinicians might have confused this with asthma when completing the CPP or when billing. Another 13 (28.3%) were initially suspected of having had asthma, but later had negative objective test results for asthma. Five of these patients also saw other specialists and received the following alternate diagnoses: eosinophilic bronchitis (n = 2); bronchiectasis (n = 2); and gastroesophageal reflux disease (n = 1). Some of these charts had falsely positive results because the asthma billing code was used at the time of the initial diagnostic suspicion, and others because the CPP had not been updated in light of objective testing and non–family physician specialist results. Another 16 (34.8%) had an upper or lower respiratory tract infection that resulted in (usually) isolated use of an asthma billing code (with no other evidence of asthma in the chart). Finally, 3 (6.5%) had a single asthma billing code with no plausible explanation and no other evidence of asthma in the chart.
DISCUSSION
In this article, we present algorithms that will enable clinicians to accurately identify their patients with asthma by searching data routinely recorded in Canadian EMR systems. These will enable clinicians to create asthma registries that can be used for practice audits and to target and evaluate quality improvement measures.
A simple individual search for use of billing code 493 provided a reasonable balance of sensitivity (78.6%) and specificity (89.2%). However, combining individual searches into algorithms further improved their diagnostic yield, with the best overall accuracy achieved by the combination of asthma in the CPP or use of billing code 493. As expected, further excluding patients for whom a COPD code had been billed resulted in an algorithm with a higher specificity but a lower sensitivity owing to the fact that patients with a combination of asthma and COPD were no longer counted (Table 3).
Observed sensitivities and specificities of individual search strategies and findings of the discordance analysis offer insight into both care and charting patterns. Prescription of asthma medications was neither particularly sensitive (78.6%) nor specific (63.6%). Sensitivity was limited because medications are not always required in mild asthma (14.3% of asthma subjects had not been prescribed a medication within the past year) (Table 2). Specificity was reduced both by formulations of these drugs used for non-asthma conditions (eg, nasal steroid sprays) and because asthma medications are also used for COPD. Indeed, exclusion of patients prescribed medications used predominantly for COPD increased the specificity of this algorithm to 73.4%.
The discordance analysis demonstrated that clinicians do not consistently update the CPP in light of other specialist findings, and might confuse COPD and asthma, both of which contributed to false-negative and false-positive results. A tendency to confuse asthma with other conditions that might be associated with wheezing, including respiratory tract infections, was another source of false-positive results.
Previous research has focused on accurate identification of asthma for epidemiologic studies through prescription or health administrative databases. By linking data to a centralized Danish prescription registry, Moth and colleagues determined that at least 1 prescription for any asthma medication (with the exception of β2-agonists) in a 12-month period had a sensitivity of 96% and a specificity of 43% for a diagnosis of asthma in children.25 Other authors used a large health administrative database to propose an asthma diagnostic algorithm consisting of 2 or more ambulatory care visits or 1 or more hospitalizations for asthma in 2 years, with a sensitivity of 84% and a specificity of 76%.21 Afzal and colleagues described a novel machine-learning approach to derive a case-detection algorithm for childhood asthma, with a sensitivity of 96% and specificity of 90%, using both text and coded data in a large primary care database.26 However, applicability is limited to the specific database that it was developed for, and to researchers with access to the sophisticated search system used. These previous studies report diagnostic algorithms for use in population-based epidemiologic studies rather than practice-level programs, where the required data linkage and special tools would not be available to clinicians. In comparison, our algorithms can be applied by individual practitioners through searches within their own EMRs in order to leverage the unique data elements available in EMRs to yield stronger operating characteristics.
Clinicians and researchers seeking to identify a target population must choose carefully among the 35 algorithms that we tested, depending on their requirements for sensitivity, specificity, and the specific types of patients desired. For example, identifying patients through asthma medication prescriptions yielded 192 patients with asthma, whereas the more specific and less sensitive “asthma in CPP” search yielded only 93 patients with asthma. These differences would obviously also affect the results of quality of care assessments in the identified asthma population. For example, 93 of 192 (48.4%) of those identified by the medication prescription search had spirometry performed, whereas 40 of 93 (43.0%) patients identified by the CPP search had spirometry performed (data not shown).
Strengths and limitations
The strengths of our study include our inclusion of comparison groups consisting of patients with diseases similar to asthma, the exhaustive number of algorithms tested, the strong performance characteristics of leading algorithms, and their practical applicability in primary care. The main limitation of our study is that in intentionally targeting approximately equal proportions of patients in each disease category, we did not use a true population sample, and therefore could not calculate positive and negative predictive values for each algorithm. However, this approach was required to ensure that algorithms could accurately differentiate asthma from clinically similar conditions, and has been applied in previous studies for the same reason.20–22 A minority of our patients with asthma had undergone objective confirmatory testing, and our reference standard relied on clinical data in most cases. However, this reflects current real-world practice.27 Accordingly, our algorithms enable clinicians to accurately identify the patients in their practices with a clinical diagnosis of asthma. The study was conducted at 2 academic clinics, and we did not include non-teaching community settings. Algorithm operating characteristics were similar between the 2 clinics, despite their unique billing, prescribing, and charting cultures. However, our findings should be validated in a non-academic setting. Similarly, we tested our algorithms in a single EMR system; several different EMR vendors exist, and each system has a unique set of features and usability constraints. However, we limited our searches to data elements that are fundamental features of the patient encounter,28 are required fields for most EMRs,29 and have been searched successfully in other EMRs.19,30,31 Accordingly, our algorithms can be used across different EMR systems. It should be noted that we did not test algorithms based on a billing code, text string, or prescription occurring repeatedly over a period of time, as such searches are not feasible in most EMR systems. Although we did assess an array of algorithms from simple to complex, most are suitable for basic EMR packages with limited search capabilities. If required, individual search results could be exported to database software, where our more complex algorithms can be enacted through data manipulation.
Conclusion
Electronic medical records are a unique tool for primary care clinicians to efficiently measure local disease-specific outcomes and quality of care, and to target corresponding quality improvement. To achieve this, clinicians require the ability to generate accurate disease-specific registries without substantial cost or complexity. We present several EMR search algorithms that can easily be applied to generate accurate asthma registries. These registries can be used by clinicians and researchers alike for audit and feedback initiatives, asthma outcome monitoring, asthma epidemiology monitoring, targeting for asthma-related pay-for-performance incentives, and preventive or active care interventions, including point-of-care interventions.32 These methods can also be emulated to establish EMR identification algorithms for other chronic diseases.
Acknowledgments
This work was supported by the Canadian Institutes of Health Research and the Ontario Lung Association.
Notes
EDITOR’S KEY POINTS
Electronic medical record (EMR) systems are increasingly being promoted as tools that enable improved safety and quality of care, particularly for chronic diseases such as asthma.
Accurate registries of patients with asthma can be built through simple searches that can easily be performed in various primary care EMRs. Clinicians can use the described searches to accurately identify their patients with asthma for outcome and care monitoring or to target quality improvement initiatives.
The methods used to identify and to test the accuracy of EMR search algorithms can also be used to establish search algorithms for other chronic diseases.
POINTS DE REPÈRE DU RÉDACTEUR
On préconise de plus en plus l’utilisation de systèmes de dossiers médicaux électroniques (DME), qui sont susceptibles d’améliorer la sécurité et la qualité des soins, en particulier pour des maladies chroniques comme l’asthme.
On peut créer des registres précis pour les patients asthmatiques grâce à des recherches qui peuvent facilement être effectuées dans les DME de divers milieux de soins primaires. Les cliniciens peuvent utiliser les outils de recherche décrits dans cet article pour identifier de façon précise leurs patients asthmatiques dans le but de surveiller les issues et le traitement ou pour envisager des mesures d’amélioration de la qualité.
Les méthodes utilisées pour identifier les cas et vérifier la précision des algorithmes de recherche dans les DME peuvent aussi servir pour créer des algorithmes de recherche pour d’autres maladies chroniques.
Footnotes
↵* Diagnoses and billing codes used to identify patients and inhaled medications used to identify patients with possible asthma are available at www.cfp.ca. Go to the full text of the article online and click on CFPlus in the menu at the top right-hand side of the page.
This article has been peer reviewed.
Cet article fait l’objet d’une révision par des pairs.
Contributors
Dr Gupta conceived of the study and performed analyses and manuscript preparation; Dr Gershon helped to design the study and reviewed the manuscript; Drs Xi and Wallace performed data collection, analysis, and manuscript preparation; and Drs Agarwal and Chan helped with study design, analysis, and manuscript preparation.
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
