Measuring what really matters

Why performance measures in screening are important

The initial framework for evaluation of quality of care was established by Avedis Donabedian with the publication of his classic paper in 1966.^5,6 This subsequently became the basis for the developing science of quality assurance and supported the emerging field of health sciences research.^1,7,8 Similarly, Barbara Starfield and colleagues outlined the importance of primary care in high-functioning health care systems. She found that primary care helps prevent illness and death and is associated with a more equitable distribution of health in a population compared with specialty care.⁹ More than 40 years ago, Edwards Deming introduced the concept of quality improvement to the United States. In subsequent years, his principles on management were widely adopted by health care organizations to improve quality and reduce costs.^10,11 More recently, Berwick and colleagues introduced the Triple Aim concept to health care. This proposed health care institutions pursue 3 dimensions of performance: improving the health of populations, enhancing the patient care experience, and reducing the per capita costs of health care.¹²

In the present era of managed health care, there is a push for external accountability around the cost and quality of care. This has led to the widespread adoption of quality improvement and quality assurance processes in primary care. A fundamental component of these processes is the development of performance measures. This has resulted in a plethora of primary care performance measurement frameworks by professional organizations and health authorities in Canada, the United States, and other Western countries.^13–20 These frameworks can be complex, consisting of many domains or dimensions, each of which contains many measures. For example, the Primary Care Performance Measurement Framework for Ontario contains 112 practice-level and 179 system-level measures.¹⁷ Screening is a main component of almost all of these frameworks.

There is increasing concern that traditional quality improvement processes might be inappropriate for primary care and that many measurable indicators of primary care performance might not be meaningful.^21–23 Other authors have pointed out that primary care practice is more complex than that of other specialties, including psychiatry, pediatrics, obstetrics and gynecology, or urology, and similar in complexity to internal medicine.²⁴ Primary care practices have been described as “complex adaptive systems” that require different metrics to measure quality beyond the traditional linear approach using a summation of single-disease–specific guidelines to describe the quality of work.^22,23 Commonly used performance measures of screening seldom reflect the nuances of primary care practice that include the importance of the patient-physician relationship, shared decision making (SDM) between patients and physicians, and the potential for different decisions from different patients owing to differences in values and preferences.^{21–23,25,26} Further, the emphasis on performance measures that evaluate the management of specific diseases coupled with pay-for-performance initiatives can have unintended consequences, such as fragmentation of care and exacerbation of health care disparities.^27,28

Many family physicians find it difficult to interpret the results of quality improvement performance measures of their practice. The complexity of practice also makes it challenging to implement strategies to improve or enhance screening. This paper expands on the role of quality in screening from a previous paper in the Prevention in Practice series on the quality of screening tests.²⁹ We consider the characteristics of screening performance measures in the context of the complex primary health care environment, the challenges related to implementation of performance measures, and potential approaches to quality improvement in the primary care environment.

Practice scenario

You are a family physician in a community-based practice comprising 7 members. Recently, as part of a quality improvement initiative in your province, you were provided with an annual report on your practice that includes the proportion of your patients who received selected screening maneuvers including mammography. In this report, breast cancer screening is reported as the percentage of eligible female patients who had received a mammogram. Owing to the potential for both harms and benefits from screening with mammography, you have been undertaking SDM with the use of decision aids to help guide your patients with their decision. Based on these discussions, some of your patients have decided not to participate in screening with mammography. You believe that these reports on screening activity in your practice do not reflect the potential for both harms and benefits from screening and the importance of individual patient values and preferences in decision making. You are concerned that these reports will be used by professional organizations and health authorities to evaluate the quality of your practice. You wonder whether there is a better approach to capturing the quality, complexity, and health outcomes of preventive screening in your practice.

Evaluating screening in your practice

Quality improvement, quality assurance, and audit and feedback are all processes that can be used in primary care to improve quality and support changes in clinical behaviour. Central to all these strategies is the development of performance measures, benchmarks, and targets that define quality.^13–20 Quality can be defined as “the degree to which health care services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”³⁰

Performance measurement is the process whereby an organization establishes the parameters by which programs and services are measured and determines whether desired outcomes are being achieved.¹⁷ A performance measure can be defined as “a measure of a primary care process or outcome that is useful at one or more levels of the health system (practice, organization, community, regional or province) to support planning, management or quality improvement.”¹⁷ Owing to the complexity and variability of primary care practice, family physicians need to consider the selection and development of performance measures suitable for their own practices.^21–26

Screening performance measures for primary care

Berwick, in his editorial “Era 3 for medicine and health care” on changes needed to improve health care systems, called for a reduction in the number of performance measures with a focus on measuring only what matters and mainly for learning.³¹ Choosing performance measures for primary care will require consideration of several issues if Berwick’s³¹ goals are to be achieved.^21–29,31 Existing screening performance measures are often narrowly focused on the uptake of individual preventive interventions and usually report percentages of patients who have received a screening test within a defined time period.^2,3,17–20 The Health Quality Ontario “MyPractice: primary care report technical appendix,” version 4, provides an example of a performance measure for screening with mammography. This was defined as the “percentage of screen eligible female patients aged 52 to 69 years who had a mammogram within the past two years.”¹⁸ However, for most screening maneuvers there is a narrow trade-off between the potential for benefit and the potential for harm.³² Often benefits are small and take years to become apparent, while harms related to overdiagnosis and false-positive results occur near the time of testing. Given these circumstances, implementing the process of SDM is most appropriate.^33–39 Shared decision making is a process whereby clinicians collaboratively help patients to reach evidence-informed and value-congruent medical decisions.^33,34 Patient values and preferences have been defined as the relative importance people place on health outcomes.^35,36 Complexity of both the practice environment and patient circumstances can also influence both physician and patient decision making on screening.^22–26 Understanding the entire scope of screening activity within a primary care practice will require consideration of all these issues.

Selecting screening performance measures for your practice

Owing to limitations in resources and availability of data, most primary care practices will be constrained in the number of performance measures that can reasonably be evaluated. A suggested goal would be to undertake the measurement of a maximum of 3 or 4 performance measures in a practice setting at any given time. Characteristics that could be used to select performance measures for screening are outlined in Table 1. Each individual performance measure would meet some of the characteristics outlined, while multiple performance measures when combined could meet as many as possible of the suggested characteristics.

Measuring SDM, patient values and preferences, and practice complexity

At present, measurement of SDM, patient values and preferences, and practice complexity is challenging.^40–42 There is also a paucity of instruments to measure practice complexity and patient values and preferences.^43,44 Further, many existing electronic medical records are poorly designed to collect patient data on these issues, making evaluation of screening activity in primary care practices more difficult and time-consuming. In contrast, definitions and methodology to measure the uptake of single-disease screening tests, such as patients who have received mammography, are well described in existing primary care performance measurement frameworks.^17–20 Table 2 provides an outline of measurement tools and instruments along with a description of measurement challenges.^23,43–53

Example of performance measurement with SDM in screening

Table 3 provides an example of how SDM could be included in a measure of the uptake of screening with mammography in a primary care practice.^3,54 Similar measures could be developed for other cancer screening interventions. Measurement of shared decisions across all cancer screening or other screening interventions could also be undertaken.

Performance targets and benchmarks

It is almost certain that family physicians will increasingly encounter performance targets and benchmarks related to screening applied to their practice by health care system payers and professional organizations. Targets are tools that are supposed to improve health and health system performance. Characteristics of targets include a commitment to achieving specific results in a defined time period, use of either quantitative or qualitative methods, and consideration of either health processes or outcomes.⁵⁵ Targets can be used in quality improvement processes, pay-for-performance schemes where physicians receive additional income for achieving specified levels of performance, and evaluation of the quality of care provided by individual physicians. Benchmarking has been described as an ongoing activity of comparing your organization’s processes, services, and products against best-known similar processes.⁵⁶ Target setting can be done by comparing performance either against other providers or against absolute predetermined levels (criterion base). Often target setting is done in an arbitrary fashion, designed to incentivize improvement and subject to change when performance levels are achieved.^57–59 Additional discussion of quality improvement approaches to improve screening in primary care will be provided in a future paper in the Prevention in Practice series.

Practice scenario part 2

You and your colleagues have had the opportunity to further discuss and reflect on performance measures provided in the annual provincial report on your practices for breast cancer screening. You decide that, because of the importance of SDM and the characteristics of your patients and practice setting, it is important to include performance measures that reflect outcomes related to the use of SDM in screening. You decide to further adapt and develop measures suitable for your practice environment to provide a more complete assessment of screening with mammography. One of your colleagues with an interest in quality improvement has made you aware of “quality circles,”^60–63 implemented in many European countries to improve quality in primary care. Your practice group decides to explore this concept further as a method of evaluating the results of your performance measurement strategy and improving the quality of screening in your practice. You also decide to ensure that knowledge translation tools on screening with mammography are easily available to your patients before or during their visits when this issue is discussed.

Suggestions for improving the quality of screening in primary care practices

Family physicians can consider some of the following suggested strategies to improve the quality of screening in their practices.

• Individual or practice groups of family physicians should carefully select, adapt, or develop screening performance measures that reflect the needs of their practices and patients. These measures can differ between practices and can evolve as quality improvement processes develop and mature in each practice setting.

• Family physicians should develop skills in the use of SDM in circumstances where there is a tradeoff between the harms and benefits of screening. Performance measures should reflect the use of SDM and the potential for no screening decisions.

• Before or at the time of the patient visit, knowledge translation tools on screening maneuvers should be readily available to assist in SDM on screening.

• Electronic medical records should be updated to allow collection of data on the use of SDM.

• Persons tasked with the development of performance measures need to consider the importance of SDM and the complexity of the primary care practice environment in the development of screening performance measures.