ABSTRACT
OBJECTIVE To identify facilitators and barriers to implementing quality measurement in primary mental health care as part of a large Canadian study (Continuous Enhancement of Quality Measurement) to identify and select key performances measures for quality improvement in primary mental health care.
DATA SOURCES CINAHL, EMBASE, MEDLINE, and PsycINFO were searched, using various terms that represented the main concepts, for articles published in English between 1996 and 2005.
STUDY SELECTION In consultation with a health sciences research librarian, the initial list of identified references was reduced to 702 abstracts, which were assessed for relevance by 2 coders using predetermined selection criteria. Following a consensus process, 34 articles were selected for inclusion in the analysis. An additional 106 citations were identified in the references of these articles, 14 of which were deemed relevant to this study, for a total of 57 empirical articles identified for review. Most articles described implementation of health care innovations and clinical practice guidelines, 5 focused on quality indicators, and 1 examined mental health indicators.
SYNTHESIS Content analysis of the 57 articles identified 7 common categories of facilitators and barriers for implementing innovations, guidelines, and quality indicators: indicator characteristics, promotional strategies, implementation strategies, resources, individual-level factors, organizational-level factors, and external factors. Implementation studies in which these factors were addressed were more likely to achieve successful outcomes.
CONCLUSION The overlap in facilitators and barriers across implementation of mental health indicators, health care innovations, and practice guidelines is not surprising, as they are often related. The overlap strengthens the findings of the limited number of studies of quality indicators. The Continuous Enhancement of Quality Measurement process for identification and selection of indicators has attended to some of these issues by using a rigorous scientific approach and by engaging a range of stakeholders in selecting and prioritizing the indicators.
One in 5 Canadians will experience a mental illness during his or her lifetime.1 Most of those who use mental health services will seek mental health care in Canada’s primary health care system. A general population survey found that among patients who consulted health care professionals for mental health purposes, more than 35% saw FPs only, 25% saw FPs and other mental health care providers (eg, psychiatrists, psychologists, social workers), and 40% saw other mental health care providers only.2 From the perspective of FPs, 1 in 4 people visiting an FP has a clinically significant mental health condition.3,4 While there is a high demand for mental health services in primary care, there are considerable gaps between the quality of services provided and optimal care.5 The use of evidence-based measures (indicators) has been suggested as part of the process of quality improvement.6
Promotion of quality measurement activities related to primary mental health care has taken place in Australia,7 the United Kingdom,8 and the United States.9 In Canada, the Continuous Enhancement of Quality Measurement (CEQM) in Primary Mental Health Care: Closing the Implementation Loop project was launched in 2004. The goal of CEQM was to improve the quality of mental health care for all Canadians by fostering quality measurement in primary mental health care. It aimed to achieve this goal through building pan-Canadian consensus on a small set of quality measures. The set of health measures for Canadian primary care mental health services was to reflect a multistakeholder perspective and be suitable for facilitating quality improvement. A 3-stage process led to a final set of 30 consensus measures. The results and reports are available online at www.ceqm-acmq.com.
A systematic review of the current literature on the facilitators and barriers to implementing quality measures in primary care was conducted as a subproject of the CEQM. This paper presents the results of that review.
DATA SOURCES
The review focused on 4 main concepts: primary care, mental health, quality indicators, and innovation or change. Four electronic databases (CINAHL, EMBASE, MEDLINE, and PsycINFO) were searched, using various terms that represented the main concepts, for articles published in English between 1996 and 2005. The database search yielded 89 555 citations for all of the combinations of the 4 main concepts (Table 1). The number of citations was reduced to 75 063 citations by eliminating the searches that did not include quality indicators as a search term. A health sciences research librarian was consulted regarding a method for reducing the number of citations, and she recommended that searches with more than 300 articles be removed, because these searches were not successful in targeting a manageable number of articles. This resulted in 733 citations and, after removing duplicates, it was reduced to 702 citations.
Results for different combinations of search concepts by database
The abstracts of the 702 citations were printed and assessed for relevance by 2 coders with master’s-level research training (T.K. and S.D.) using predetermined selection criteria. The 2 coders rated 50 abstracts and compared their answers to assess whether they were applying the criteria in the same manner.10 They discussed the discrepancies until they agreed upon a rating. Some of the criteria were rewritten to increase clarity. To be selected as relevant, an abstract had to focus on primary health care (or primary mental health care) and refer to a quality improvement tool or the process of implementing quality measurement, quality indicators, or quality improvement. In the first round, coders rated the articles as yes, no, or unsure, and their ratings showed agreement for 533 abstracts (20 yes and 513 no) and did not agree or were rated unsure for 169 abstracts (κ = 0.540). The 169 abstracts were rated again by the same 2 people, but this time they had to make a forced choice of yes or no. After the second round, the assessment for relevance yielded 671 abstracts with identical ratings (62 yes and 609 no) and 31 abstracts with mixed ratings (κ = 0.775). A professor of psychiatry (D.A) with expertise in performance measurement took the role of a third coder for the 31 tied ratings, and the final result was 83 yes and 619 no.
The 83 agreed-upon articles were retrieved and read, and the 44 articles reporting findings of original research were selected for review. During the first reading, 3 types of articles were identified: those that specifically addressed quality indicators, those that addressed clinical practice guidelines, and those that addressed health care innovations in a broader sense. Quality ratings are a key step in systematic reviews10; however, with so few empirical articles on implementing quality indicators, we included all of them as long as they contained findings of original research. One of the researchers searched the reference lists of the most relevant articles for secondary references, uncovering 106 additional references. After 2 rounds of coding for relevance by the same 2 researchers using the original criteria, 34 abstracts were selected, the corresponding articles were read, and 14 additional empirical articles were added to the review.
Content analysis was used to abstract any text mentioning implementation facilitators or barriers from the selected articles.11 This process was completed by one of the master’s-trained researchers who consulted the second researcher as needed. A separate list of facilitators and barriers was created for the 3 groups of articles: quality indicators, clinical practice guidelines, and health care innovations. Two researchers independently examined each list of facilitators and barriers and grouped them by topic or recurring idea. The 2 researchers then compared their groupings and agreed upon broad categories to fit the data. Category development is a process of understanding and explaining the data.12
SYNTHESIS
The selected articles are summarized in Table 2.6,13–68 The authors of most of the articles were based in the United States (n = 29), discussed broad health care innovations (n = 32), and used solely qualitative research methods (n = 35). Twenty articles about implementing clinical practice guidelines were also found, even though the search was intended to focus on articles about implementing quality indicators and clinical practice guidelines was not a search term. There were only 5 empirical studies of the specific process of implementing quality indicators. The settings of 4 of these studies were clearly primary care, yet the indicators were not mental health–related, and 1 study implemented mental health quality indicators in a community-based mental health clinic. The fifth study was retained, even though the clinic might have offered both primary and secondary care, because we wanted to glean information about implementing mental health–specific indicators. The 5 articles included 1 quasi-experimental study, 1 case study, 1 retrospective audit, and 2 qualitative studies published between 2000 and 2004. Table 3 lists facilitators and barriers to implementing quality indicators, as listed in these specific articles, and Table 4 presents facilitators in the form of a checklist for readiness to implement clinical practice guidelines and other health care innovations.
Summary of the empirical articles reviewed: N = 57.
Facilitators and barriers to implementing quality indicators: N = 5 articles.
Readiness to implement quality measurement checklist*: Read each statement and indicate your response with a check mark. Complete the checklist based on your perspective in the organization. Try to respond to every statement.
The 2 master’s-trained researchers agreed upon 7 broad categories to represent the facilitators and barriers to implementing quality indicators. The categories that fit the quality indicator data were similar to the categories chosen for the clinical practice guideline and health innovation data. These similarities suggest that the same facilitators and barriers apply across quality indicators and clinical practice guidelines as 2 types of the broader class of health innovations. The 7 broad categories that represent the facilitators and barriers to implementing quality indicators include measure characteristics (key attributes), promotional strategies, implementation strategies, resources, individual-level factors, organizational-level factors, and factors external to the organization. The articles about implementing quality indicators are the focus of this paper. A number of articles reviewed provided both their results and rich, detailed information about the experience of implementation.13,15,16,69
DISCUSSION
The facilitators and barriers to implementing quality indicators in primary care are discussed by category and in relation to the innovation and change literature.
Characteristics
The stakeholders’ perceptions of the importance of what is being measured and the ease with which it can be measured are important considerations for selecting quality measures. Much of the variance in adoption rates for both processes and quality measures can be explained by a measure’s key attributes, as perceived by potential adopters. Attributes that are positively related to adoption include perceived benefit to patients, fit with existing skills and resources, ease of testing, face validity, and level of change required to implement the process and its measure. The perceived complexity of an innovation and its quality measure is negatively related to its adoption rate.70,71 Indicators were also more likely to be adopted if they reflected current knowledge, were evidence-based, covered important areas, used reliable and complete data, and represented an “open” versus a “hidden” agenda. An important barrier to implementing clinical practice guidelines in primary care, which reflects many of the above issues, was faced when the measures were originally developed for secondary or another level of care.
Promotion and implementation
Innovations are spread by influences that range from passive diffusion to active dissemination. Diffusion has been characterized as unplanned, informal, decentralized, and often mediated by peers, while dissemination is described as planned, formal, often centralized, and more likely to occur through vertical channels.71 Rogers described 5 main steps that take place before new measures are fully adopted: knowledge, persuasion, decision, implementation, and confirmation.70
Endorsement of measures by credible organizations, such as a government task force for quality indicators or publication in a respected journal, was shown to facilitate adoption. In contrast, the belief that quality measurement is a threat to professional autonomy or a tool to penalize bad performance is seen as a barrier. This highlights the importance of involving individuals in the early stages of planning the measurement process. Interpersonal communication between 2 or more similar individuals is more effective than mass communication in persuading someone to adopt an innovation.70 Linking quality indicator use to performance incentives was reported to be useful in some contexts. However, the use of financial penalties based on performance areas beyond the scope of professional control is a barrier. Ensuring that all stakeholders share the same perception of incentives is critical for successful implementation.
Resources
The PRECEDE-PROCEED model outlines the steps for planning, implementing, and evaluating innovations that are directed toward improving the health of individuals, populations, or organizations.72 Assessing the need for and availability of resources is vital to the implementation planning process. Dedicated resources such as time, funding, and skilled personnel are enabling factors that make desired change possible.71,72
Individual-level factors
The knowledge, attitudes, beliefs, values, and perceptions of individuals are predisposing factors that can facilitate or hinder motivation for change. In turn, the innovation decision is influenced by this motivation as well as by individuals’ needs, goals, skills, learning styles, and social networks.71,72 The implementation of quality indicators is facilitated when quality measurement is a personal interest or responsibility of a physician, staff member, or office manager.
Organizational-level and external factors
Organizational structures, culture, and resources are important for supporting the adoption of new processes and measures.71,72 In the context of primary care in Canada, there are obvious differences across provinces in the relationships between government funders, primary care providers, and specialty care providers. Relationships and funding arrangements that support information systems integration and collaboration between providers are more easily linked to quality measures. More specific factors within organizations that were reported to facilitate the implementation of indicators were multistakeholder involvement, board member support, and team agreement on the purpose, importance, and benefits of indicators. Reported implementation barriers included a perceived lack of time to plan and limited communication among professions. Generating indicators was difficult when some services were not documented in the medical record and the responsibility for data entry was not clear.73
Conclusion
The CEQM is an innovative Canadian project developing a consensus set of quality measures for primary mental health care. This review has demonstrated that successful implementation of quality measures can occur but will depend on the interaction of multiple factors, including measure characteristics, promotional messages, implementation strategies, resources, the intended adopters, and the intraorganizational and interorganizational contexts. As we undertake pilot projects to implement our quality measures, it will be important to gather data about the process and the outcomes.
Acknowledgments
The authors developed the checklist as a part of the Continuous Enhancement of Quality Measurement: Closing the Implementation Loop project, which is funded by Health Canada’s Primary Health Care Transition Fund.
Notes
EDITOR’S KEY POINTS
-
While there is a high demand for mental health services in primary care, there are considerable gaps between the quality of services provided and optimal care.
-
Promotion of quality measurement activities related to primary mental health care has taken place in many countries. In Canada, the Continuous Enhancement of Quality Measurement in Primary Mental Health Care project was launched in 2004. This paper presents the results of a systematic review of the current literature on the facilitators and barriers to implementing quality measures in primary care, conducted as part of this larger project.
-
The authors found that successful implementation of quality measures can occur, but that success depends on the interaction of multiple factors, including measure characteristics (key attributes), promotional messages, implementation strategies, resources, the intended adopters, and the intraorganizational and interorganizational contexts.
POINTS DE REPÈRE DU RÉDACTEUR
-
Alors qu’il existe une forte demande pour les services primaires en santé mentale, la qualité des services prodigués est loin d’être idéale.
-
Dans plusieurs pays, on a favorisé la mise en place de systèmes de mesure de la qualité des services primaires en santé mentale. Le projet Amélioration continue de la mesure de la qualité des soins et services de première ligne en santé mentale a été lancé au Canada en 2004. Cet article présente les résultats d’une revue systématique de la littérature effectuée à l’occasion de ce projet, revue portant sur les facteurs qui favorisent ou qui gênent la mise en place de mesures de la qualité des soins primaires.
-
Les auteurs ont observé qu’il est possible d’instaurer des mesures de la qualité, mais que le succès repose sur l’interaction entre plusieurs facteurs, incluant les caractéristiques de l’instrument de mesure (attribut clés), les messages promotionnels, la mise en place de stratégies, les ressources, les intervenants visés et les contextes intra et inter-organisationnels.
Footnotes
-
This article has been peer reviewed.
-
Cet article a fait l’objet d’une révision par des pairs.
-
Contributors
Ms Kyle and Ms Desai performed the literature search and coded the selected articles. All authors contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.
-
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
