Abstract
Objective To understand the usefulness of audit and feedback among family physicians and examine the barriers to using it to improve quality of care.
Design Qualitative study using in-depth interviews.
Setting Family physicians across Ontario participating in audit and feedback initiatives describing the proportion of patients meeting quality targets for chronic disease.
Participants Purposive sampling was conducted to ensure variation in sex, years of experience, and baseline performance for quality metrics. All participants used electronic medical records and worked in multidisciplinary primary care practices.
Methods Semistructured interviews were conducted with family physicians. The interview guide and initial coding framework were adjusted iteratively in keeping with the constant comparative method. Sampling continued until saturation was reached. Interviews were analyzed using the framework approach.
Main findings Participants reported that the feedback increased their awareness of gaps between ideal and actual performance. This resulted mainly in efforts to “try harder” patient by patient. Key barriers to acting upon feedback in a systematic manner included a perceived discordance between population-level quality targets and patient-centred care, as well as competing priorities at both the patient and organizational levels. Although all participants had electronic medical records, participants reported a lack of quality improvement infrastructure in their practices.
Conclusion Family physicians were not highly motivated to achieve evidence-based population-level quality targets for diabetes; many competing organizational and clinical goals took priority. Additional human resources might be needed to translate data in feedback reports into systematic changes that could lead to sustained improvements in quality of care.
Audit and feedback, defined as a summary of the clinical performance of health care providers over a specified period,1 is a widely used quality improvement (QI) strategy.2 A recently updated Cochrane review of 140 randomized trials of audit and feedback conducted across many clinical conditions and settings found that it increases provider compliance with desired practices by a median of 4.3% (interquartile range 0.5% to 16%).1
Pressures to increase accountability in primary care often result in audit and feedback initiatives led by agencies external to family practices. However, previous research found that general practitioners perceived external quality programs to be an imposition, while internal QI was perceived to be a professional obligation.3 In this study, we sought to understand the perceived usefulness of externally generated feedback among family physicians and perceived barriers and facilitators to using audit and feedback to improve processes of care and patient outcomes. We also explored how to optimize the design of audit and feedback interventions to be most actionable for family physicians.
METHODS
This qualitative study was embedded within a pragmatic cluster-randomized trial in which all family physician participants received feedback reports. The protocol for the overall project has been previously reported.4 The study was approved by the Sunnybrook Research Ethics Board.
Setting and context
Ontario has a single-payer system in which there is no access fee for physician visits or hospitalizations. Pharmaceutical costs are covered for inpatients, patients on social assistance, and those older than age 65. During the past decade, Ontario has implemented substantial primary care reforms, with most family physicians moving from independent, fee-for-service models to group-based models with partial capitation-based payment, requiring physicians to “roster” patients to their practices.5 All participants in this study rostered their patients and also benefited from funds from the Ontario Ministry of Health to support allied health care providers in their clinics in models conceptually similar to the patient-centred medical home.6 The Ministry of Health has identified improvements in diabetes care as a key priority. In 2010, the Ministry of Health provided family physicians with reports summarizing the proportion of patients with diabetes receiving guideline-concordant care.7 New provincial legislation8 indicates that similar initiatives for other conditions will be developed in an attempt to improve accountability. However, most Ontario primary care practices had not experienced recurrent clinical audits or performance feedback at the time of this study.
Participants
All 54 family physicians in the overall study contributed data from their electronic medical records (EMRs) to the Electronic Medical Record Administrative Data Linked Database (EMRALD). The patient records in EMRALD are de-identified and patients with diabetes can be selected using validated algorithms.9 In this study, EMR records of patients with diabetes were mined for the presence of guideline-recommended processes or treatments.
Intervention
Participating family physicians received feedback reports from EMRALD detailing their scores for 9 different evidence-based quality targets. A sample feedback report is available from CFPlus.* An achievable benchmark of care representing the performance of the top 10% of peers was calculated for each quality indicator, based on research indicating that this comparator led to greater improvements in care than “mean” performance.10 These EMRALD feedback reports described only the overall proportions of patients meeting targets; no patient-specific data were included. The reports were delivered to each physician at their practices by courier from the investigators in envelopes marked “confidential,” along with reflection worksheets that could be completed for continuing medical education credits. Participants, like all other family physicians in Ontario, received feedback reports from the Ontario Ministry of Health regarding their patients with diabetes that included patient-specific data (ie, detailing whether each patient was overdue for a given test) but only for 3 process measures. A sample Ontario Ministry of Health feedback report is available from CFPlus.* At the time of this study, participants had each received 1 EMRALD report and 1 Ontario Ministry of Health report. This provided an opportunity to determine whether and how family physicians used the feedback reports and to explore preferences regarding design.
Data collection
Semistructured, in-depth, individual interviews were conducted by a single interviewer (N.I.) between October and December 2010. To inform question formation and sequencing, pilot interviews were conducted before recruitment. The interview guide started with questions to build rapport and to define the study context. After this, open-ended questions were asked about the feedback reports to elicit themes in a nonthreatening fashion. Probing questions followed to pursue areas of particular interest or issues brought up by the participant. The guide was iteratively adapted as the interviews were conducted. The questions in the interview guide were informed by the clinical and research experience of the multidisciplinary team of investigators and by relevant behavioural and psychological theories,11–17 which informed probing questions about likely barriers to the use of feedback in the clinical setting and moderating factors in design of the feedback.
Interviews were conducted at the time and place of the participants’ choosing and were recorded using a transcription service to produce verbatim electronic transcripts. We used “stratified purposeful sampling,”18 selecting participants with those features reported as relevant in previous studies, seeking informational rather than probabilistic representativeness. For instance, guideline adherence and quality of care might be related to years in practice19 and physician sex,20 so variety was sought in those factors. Additionally, we identified participants with various levels of baseline performance across all indicators, because this is an important predictor of feedback effectiveness.1 After themes were established based on the analysis of the first round of interviews (see Analysis), we used snowball sampling to seek out participants who might challenge our early findings. Specifically, we asked participants to recommend potential interviewees who were either highly involved in QI or who were particularly disinclined to participate in QI. Early findings were explored with these participants to search for disconfirming evidence and to crystallize interpretations. Although we sought variation in certain characteristics, the sample was similar in many other ways. All participants were EMR-using family physicians who worked in team-based practices with access to allied health care providers, and all consented to receiving feedback reports as part of the overall project. Given the targeted nature of the questions, we expected that saturation would be accomplished with approximately 12 interviews.21 To account for time away from patient care, we provided a $75 honorarium to each physician.
Analysis
We used the framework approach,22 aiming to accurately reflect the original accounts of the participants through the use of inductive techniques, directed by the a priori goals and objectives for the project. We tracked the identification of themes along with dates of interpretations to provide an audit trail documenting the analysis.23 NVivo software was used to assist with the data analysis.
We established an initial index of themes based on a priori–defined issues of interest and combined these with a data-driven coding framework developed after analyzing the first 3 interviews. For the next 4 interviews, 2 members of the research team (N.I. and J.B.) independently identified key findings arising from the data. The results were then discussed with a third investigator (M.Z.) to gain consensus on key initial findings. We pursued multiple coding in this way to provide reassurance that all possible themes were given consideration.24 In keeping with the constant comparison method, we revised the coding framework and the interview guide as the data collection proceeded. For example, we found that participants focused on the nature of the performance targets and their professional role and self-efficacy with respect to QI, rather than the specifics about how the feedback was designed or delivered. Thus, the final coding framework incorporated few of the a priori topics. After 7 interviews, consensus was reached regarding the descriptive codes. In the subsequent interviews we sought disconfirming evidence to both clarify the findings and ensure saturation.
Findings were coalesced into themes and then organized within 4 topic areas. To illustrate, the topic “personal barriers or facilitators” had 3 important sub-themes: competing priorities, perceived roles of patient and provider, and QI interest and expectations. Other topic areas included initial response and reaction, organizational barriers or facilitators, and feedback design preferences. As per the framework approach, key findings from each interview were placed into a matrix for each topic area (available upon request). Each participant had his or her own row with key quotes or summaries listed under the relevant subtheme. This facilitated the identification of central themes and patterns across participants, within and across the related topics. After multiple readings and discussion among the entire team of investigators, the results were ultimately grouped into 3 important topics: usefulness of feedback for systematic chronic disease management; reported barriers to QI efforts in response to feedback; and preference for intervention design to support QI.
FINDINGS
Participant characteristics
Data saturation was reached after 12 interviews. Interviews lasted a median of 50 minutes (range 37 to 70 minutes). The participants varied with respect to sex, years of experience, and location and size of practice (Table 1). Three participants were particularly high performers; 2 others had relatively low proportions of diabetes patients meeting targets. Snowball sampling led to inclusion of 1 participant who was highly sceptical of the benefits of any practice-based innovations and 2 participants highly engaged in practice-based QI activities.
Findings
Usefulness of the feedback for systematic chronic disease management
None of the participants reported that they found the feedback particularly useful. Participants commonly reported that they intended to improve performance by being more mindful of the relevant targets during patient encounters. However, no participants reported using the feedback to set specific goals for improvement or action plans for reaching these goals. Even when prompted, most participants could not envision ways for the practice to facilitate proactive chronic disease management (ie, as in the chronic care model25). A few proposed the concept of developing disease-based patient registries to check the data and then contacting those patients who need to take action or using reminders in the EMR. However, none had actually followed through on these ideas during the 8- to 12-week interval between receiving the feedback report and participating in the interview. Quotations from the interview participants regarding the usefulness of the feedback reports for QI are presented in Box 1.
Quotations regarding the usefulness of the feedback reports for quality improvement
It’s like getting a D and really, we’re all type A. Do you know what I do with bad report cards? I filed it away until you came today because I didn’t want to look at it again. (Interview 11)
It wanted me to sign that I was going to improve x, x, and x, over the next period of time and so on. I just said, “No way, I am not buying into this guilt trip.” I tried and I’m not perfect. I’m going to continue to try but I’m not going to be burdened with extra guilt. (Interview 9)
When I was seeing my diabetic patients, I spent more time, you know, making sure I was paying attention, you know, to the things I’m supposed to pay attention to, which I thought I did, but obviously could improve upon. (Interview 4)
Barriers to QI efforts in response to feedback
The most commonly reported barrier to using the feedback for QI was concern about the validity of the data used to generate the reports and the ability to leverage EMRs for QI. Providers did not trust the data although (or possibly because) it came from their own EMRs. To act upon reports providing only aggregate data and not individual patient names, physicians had to manually identify patients for whom action was needed. However, few were motivated or skilled enough to check the data or to generate lists by conducting EMR-based searches. Quotations about participants’ challenges in leveraging the EMRs for action are presented in Box 2.
Quotations regarding participants’ challenges leveraging the EMRs for action
A number of my patients with these diseases aren’t captured in the EMR. (Interview 5)
I’m the only one that learned how to create my own searches so I guess I have a greater appreciation for what programs to do, and as a result, also probably I’m among the most motivated for having thorough information in the patient profile because I know what it can get me. (Interview 10)
The problem is going back to try to verify [the data] is really time consuming. It just takes a lot of work to go and try to hunt down if that is really true and what is going on. (Interview 8)
EMR—electronic medical record.
Another important barrier to using the feedback reports for QI related to the tension between standardized targets for populations and patient-centred care. Many endorsed a desire to practise in a patient-centred fashion and believed that population-level targets or QI initiatives were in conflict with this ideal. They expressed professional pride in judiciously applying targets and guidelines and worried that standardizing care would result in disease-oriented rather than person-oriented decisions. Although the performance targets were based on well-established guidelines with high-level evidence, participants described concerns with the measurement of their performance based on these targets. Participants focused on patients that should be excluded because targets would not be appropriate (eg, elderly patients with comorbidities). A few participants expressed uncertainty about whether their (potential) efforts toward QI in response to feedback would translate into meaningful differences for their practices or for their patients. Concerns were also raised about being judged on outcomes beyond the physician’s control. Quotations from participants regarding the tension between population-level targets and individualized clinical decisions are presented in Box 3.
Quotations regarding the tension between population-level targets and individualized clinical decisions
It talks about whole populations as opposed to the one individual and I think my approach to this job is the one person at a time. (Interview 2)
We have to look at the whole person. What do we stand for as family doctors, you know? We are there to walk ... to make life’s journey as medical professionals with our patients and we, I think are going—I’m tearing up—we are doing them a big disservice by buying into this … it is just short-sighted. I want to explain myself, I guess, and say, “Listen, my numbers aren’t that good, but I’m a good doctor.” (Interview 3)
So, I mean, this tells us what we as physicians should do as technicians. It doesn’t tell us what we as physicians should do as motivators. I think this is great; I just think it’s incomplete. (Interview 4)
The last thing I want is to have one of my older patients become hypoglycemic, fall, break their hip because they’re on some [drug] which is really totally inappropriate, because some doctors try to meet an unrealistic target. So we have to be really careful of the clinical practice of medicine [that] is different from the guideline practice of medicine. It’s really important not to get caught up on “guideline-itis”—just treat your patients properly. (Interview 5)
They are all different, they all have their own financial supports and home supports and degrees of motivation, and it is a real tricky task. We are kind of stickhandling on what is the most important thing and you are trying to practise patient-centred medicine too. So you are trying to be ... they come to you and they are really worried about their daughter and [you cannot say], “Okay, we’ll talk about your daughter another day; let’s talk about your LDL.” (Interview 8)
LDL—low-density lipoprotein.
A third key barrier related to the challenges of priority setting in primary care. Participants expressed a sense of being overwhelmed and unable to fully balance demands on their time at the clinical, organizational, and personal levels. From a clinical perspective, the frequent presence of acute issues that interfered with chronic disease management was expressed as an inevitable problem. In such cases, the management of chronic disease was deemed to be clinically important, but other patient problems were the priority. A few of the participants who were more inclined toward QI discussed setting priorities at the practice level, noting other ongoing projects or programs that limited their ability to galvanize support or direct attention toward addressing the gaps identified by the externally produced feedback. Quotations from participants regarding challenges with priority setting in primary care are presented in Box 4.
Quotations regarding challenges with priority setting in primary care
How much time do you want your doctor devoting to that, because the more time I’m devoting to my computer extractions the less time I am [devoting] to phoning the patients or bringing the patients in and seeing them to meet these targets. You can’t have your cake and eat it too. (Interview 1)
Really, the time the 5 of us physicians here can get together, we’re dealing with contract issues and leaky roofs, paint on the wall, contract signing, nursing crisis, new staffing. The 5 of us are not at the point where we’re dealing with clinical stuff other than, you know, the corridor consult. (Interview 6)
For a while we’ve been trying to, at our business meetings, to set aside time to have a little data or a reminder about how to use one thing or another [in the electronic medical record] and then we haven’t been having that, it hasn’t been showing up on the agenda. (Interview 10)
Preferences for intervention design to support QI
The participants varied with respect to their stated preferences for other aspects of the intervention design. Despite being EMR users, most preferred paper-based reports; participants reported that they might not view feedback available on a website. In terms of frequency, the participants wanted enough time to improve the outcomes before the next report. However, a few participants foresaw a near future when reports for various diseases would provide more information than they could deeply reflect upon. To make the reports manageable, some requested only summary information, with the capacity to access more details as their time or interest level increased. Many suggested an emphasis on high-risk patients who were overdue for visits or who would clearly benefit from additional or more intensive care. Quotations from participants regarding their desire to focus feedback on higher-risk patients are presented in Box 5.
Quotations regarding the desire to focus feedback on higher-risk patients
[Feedback should be focused on] visits within the last 2 years or the last 6 months, so that you can see, “OK, these are the patients who are not coming in.” (Interview 3)
So if you had [hemoglobin] A1c greater than 8 and I saw how many patients or what percentage of my diabetic population was over 8 then you’d probably catch my attention and I’d be searching through … “You’re beyond this number, how far above [target] blood pressure are you?” (Interview 2)
You know what I would really like is the number of people, if there was—and this is a very different statistic to get—but the number … some way of showing a change has occurred over time from A1c somewhere in the 8s to the 7s or some number from the 9s to the 8s ... I wouldn’t want more people getting worse, that is for sure. (Interview 8)
There’s the group I would target despite what else is going on socially in their lives, most of the time. You know, if their A1c is over 8, in my mind, those are people I would target … but there’s the subtler differences of the LDL of 2.1 and 2.2 who, often, there’s going to be things at a higher priority. (Interview 6)
LDL—low-density lipoprotein.
Even participants inclined toward QI believed it was not the family doctor’s role (or that it would add too much work) to initiate changes to practice-based processes. Participants expressed a desire for complementary interventions that would provide support to take action, including both technical assistance for managing clinical information and administrative assistance to determine and implement the recommended QI activities. Many welcomed the idea of a follow-up telephone call by a supportive colleague to review the report and discuss explanations for the results and strategies for improving patient care. Quotations from participants regarding their desire for additional resources to manage chronic disease initiatives are presented in Box 6.
Quotations regarding the desire for additional resources to manage chronic disease initiatives
There probably should be like an information management person affiliated with every group-type practice. You need like an epidemiologist on-site whose job it is to generate information like this, provide feedback to patients and providers about how they are doing, and to track the data. (Interview 8)
We hired someone a couple half-days a week to help us mine data, but we discovered a lot more work associated with it, because once we mine the data and we need to communicate to our patients, then we need someone to call the patients as well. (Interview 12)
It becomes so overwhelming unless there are staff resources available to do that for me, to manage the data, mine the data, and then dedicated time to sit down with my team, with my nurse practitioner, with my pharmacist, with my dietitian and just talk about it. And maybe the mental health worker ... and say, “OK, tell me what the data look like, tell me where we’re not doing so well … how can you help me to do better, how can you help me to get better marks?” (Interview 7)
DISCUSSION
Increased awareness of suboptimal performance usually resulted in the intention to “try harder” to do more during each patient visit, rather than “work smarter” by implementing point-of-care reminders or initiating systems to identify and contact patients for reassessment. Participants reported that they welcomed the feedback, yet the reports often generated strong emotive responses wherein participants defended their position and their profession.
Such findings help to explain the small to moderate effects generally observed in randomized trials of audit and feedback.1 As noted in previous work in primary care, we found that family physicians struggle with integrating QI concepts into their practices.26 Previous studies in primary care also found that systematic implementation of QI occurs slowly.27,28 Similar to previous primary care studies,29 participants in this study reported discordance between patient-centred ideals (tailored, specific care) and QI interventions (systematic approach, population-level metrics). Our finding that not all quality targets or guideline recommendations are regarded as equally important by family physicians for measuring primary care performance echoes earlier work with Ontario family physicians.30 The salience of feedback might be increased if targets reflect the priorities of the family physician.31 For example, more holistic measures of quality in primary care are available that include indicators for access and patient-centredness32 and might be more fitting with the focus on the person (rather than the disease) that is subscribed to by family physicians.33 Ideally, all 6 Institute of Medicine quality domains would be covered: safety, effectiveness, patient centredness, timeliness, efficiency, and equity.34
The family physicians were challenged by multiple competing priorities. Goal conflict has been shown to be an important predictor of whether professionals follow through on accepted guideline recommendations,35,36 and consideration of competing demands provides insight into what is commonly deemed clinical inertia.37 One way to partially address legitimate competing priorities might be to heed our participants’ requests that feedback be tailored to focus on manageable numbers of higher-risk patients needing semiurgent or urgent action.38,39 One previous trial found that feedback reports stratified by risk levels were only marginally more successful for improving management of hypertension.39
Just as patients cannot focus on chronic disease management when they have unstable shelter, providers cannot focus on QI when their office resources are not established or available. For patient-centred medical homes to achieve their potential for providing community-oriented primary care,40 they require the human and technological resources to practise population health–type management.41,42 In our study, most physicians were unable to personally leverage their EMR data to facilitate QI. Indeed, most EMRs in use in primary care in the United States or in Canada are not yet truly functional for QI.43,44 Another qualitative study in Ontario found that even multidisciplinary primary care teams failed to take action upon receiving feedback reports owing to a lack of “performance management skill development.”45 While this expertise develops, external supports might be needed to leverage available data to identify important gaps in care and to work with primary care providers to identify changes that would help them to achieve their goals. Our findings also suggest that those designing audit and feedback interventions need to think carefully about precise behaviour they want the feedback to provoke in physicians and be sure that the intervention provides support to implement that behaviour.
Limitations
Participants in our study were relatively early adopters of EMRs and most practices were also involved throughout the study in extraneous QI initiatives. If these physicians and practices did not act upon the feedback owing to discomfort with the targets or to lack of resources, other primary care providers might be even less likely to act upon externally generated feedback. Nevertheless, we acknowledge that transferability of findings from this qualitative study of purposively sampled Ontario-based family physicians working in EMR-using multidisciplinary primary care practices is uncertain. It is also possible that repeated exposure to feedback over time could lead to different responses among the participants.
Conclusion
We found that family physicians did not readily act upon the feedback reports for a number of reasons (Table 2). For QI champions, this was generally owing to competing organizational-level priorities; these participants knew what would be necessary but were busy implementing other initiatives. The rest of the participants struggled with patient-level (and personal-level) priority setting and focused on potential flaws in the data or targets used in the feedback. Such participants perceived minimal utility in knowing the aggregate proportion of patients reaching guideline-based targets, believing that patients are unique and require tailored, patient-centred care. For audit and feedback interventions to lead to changes in the behaviour of family physicians, it is necessary for the content of the feedback to align with the patient-centred priorities of the family physician. Leveraging feedback to proactively identify and contact high-risk patients who might benefit from clinical assessment was considered desirable, but did not often occur owing to a lack of QI infrastructure. This includes both technical expertise and dedicated human resources committed to QI. Therefore, it is necessary to carefully consider the abilities and resources of the primary care practice; if adequate QI infrastructure does not exist, cointerventions should be delivered with the feedback to facilitate systematic, sustainable changes. Further research should methodically test how to best combine and implement interventions with audit and feedback.
Acknowledgments
Dr Ivers is supported by research fellowship awards from the Canadian Institutes of Health Research and the Department of Family and Community Medicine at the University of Toronto.
Notes
EDITOR’S KEY POINTS
Family physicians did not readily act upon the feedback reports they received for a number of reasons, including competing organizational-level priorities; difficulty with patient-level (and personal-level) priority setting; and concern about potential flaws in the data or targets used in the feedback.
For audit and feedback interventions to lead to changes in the behaviour of family physicians, it is necessary for the content of the feedback to align with the patient-centred priorities of the family physician.
It is necessary to carefully consider the abilities and resources of the primary care practice; if adequate quality improvement infrastructure does not exist, cointerventions should be delivered with the feedback to facilitate systematic, sustainable changes. Further research should methodically test how to best combine and implement interventions with audit and feedback.
POINTS DE REPÈRE DU RÉDACTEUR
Les médecins de famille ne réagissaient pas spontanément aux rapports de feedback qu’ils recevaient pour diverses raisons, incluant la présence de certains conflits avec des priorités d’ordre organisationnel; la difficulté d’établir les priorités pour leurs patients (et aussi pour eux-mêmes); et certains doutes concernant des lacunes possibles dans les données ou dans les cibles utilisées dans le feedback.
Si l’on veut que les interventions comme le feedback et le contrôle de la qualité entraînent des changements dans le comportement des médecins de famille, il faudra que le contenu du feedback tienne compte des priorités du médecin de famille, lesquelles sont centrées sur le patient.
On doit être très attentif aux capacités et aux ressources des établissements de soins primaires; en absence d’une infrastructure pour améliorer la qualité, des interventions devraient être offertes conjointement avec le feedback pour faciliter des changements systématiques durables. D’autres études devraient tenter d’établir la meilleure façon de combiner ces interventions avec le contrôle de qualité et le feedback, et de les mettre en place.
Footnotes
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
↵* Sample feedback reports with aggregated proportions of patients meeting 9 quality indicators and with patient-specific data from the Ontario Ministry of Health provided to all family physicians for 3 process measures are available at www.cfp.ca. Go to the full text of the article online and click on CFPlus in the menu at the top right-hand side of the page.
Contributors
Drs Ivers, Barnsley, and Tu contributed to the data gathering. Drs Ivers, Barnsley, and Zwarenstein contributed to the data analysis. All authors contributed to the concept and design of the study, interpretation, and preparing the manuscript for submission.
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada