|
 |
|
|||||||||
|
|||||||||||
Vol. 54, No. 10, October 2008, pp.1436 - 1437.e5
Copyright © 2008 by The College of Family Physicians of Canada
Research |
Can you use a sequential sample of patients as a substitute for a full practice audit?
Study of mammography screening rates in 20 family practices in Ontario
Graham Swanson, MSc MD CCFP FCFPPart-time Associate Professor in the Department of Family Medicine at McMaster University in Hamilton, Ont
Janusz Kaczorowski, MA PhD
Professor and Research Director in the Department of Family Practice at the University of British Columbia in Vancouver
Correspondence: Dr Graham Swanson, 2238 Caroline St, Burlington, ON L7R 1M6; e-mailswansong{at}mcmaster.ca
Quality assurance activities in family medicine are recommended by the College of Family Physicians of Canada.1 Quality assurance involves reviewing and reflecting on the way we practice and trying to improve it. Several authors have studied various ways to measure family physicians’ performance of selected preventive maneuvers and to provide feedback to improve the quality of care.2,3 There are at least 2 recurring problems that deter implementation of quality assurance in family practices: getting physicians involved in the quality assurance activities and the amount of time and energy it takes to perform data extraction.2 This last is particularly a factor for practices that do not use fully electronic medical records.
In an attempt to make data extraction easier and less time-consuming, 20 physicians participated in an audit of mammography screening rates in family practice using an approach designed to overcome these problems. Instead of conducting a full practice audit or an audit based on a random sample of patients representative of the practice, a sample was generated from consecutive eligible patients who were booked for appointments during a predefined data-collection period.4 As these women’s charts were already pulled for the appointments, the relevant information was easily available and was collected by the physicians or their office staff.
Once collected, the data were faxed to a research assistant who collated them weekly. Anonymous individual and group summaries were prepared and shared with all participants weekly.
This very inexpensive process stimulated an enthusiastic response from participants as they collected their own data and compared them with the data of their peers. Before further studies of the effect of using a sequential sampling technique to collect data for an audit were to be undertaken, it was thought important to assess how well the data generated using this approach represented data from the whole practice. This paper reports on a study in which the results of an audit of mammography screening rates among all women in a practice eligible for screening were compared with the results of an audit of a sample of eligible women from the same practice chosen by a sequential sampling technique.
METHODS
Twenty family physicians in south-central Ontario volunteered to participate in the study. Seventeen of these physicians had been involved in a previous audit that had been part of a faculty development project in the Department of Family Medicine at McMaster University in Hamilton, Ont. The other 3 were colleagues of 1 of the authors (G.S.) who recruited them for the study.
Data were collected using 2 sampling strategies. One approach was based on a full practice audit of all eligible patients in a practice, and the other collected data from a sample of eligible women from the same practice who booked appointments during the 8-week study period. Eligible patients were defined as women 52 to 71 years old who had had at least 1 visit to the office during the past 3 years. The age group chosen corresponded to the recommended age group for screening with 2 years added to allow for the first mammograms to be done within the recommended period (50 to 69 years old).5 All women were included even if mammography was ordered for diagnostic rather than screening purposes.
Each practice was responsible for generating a list of eligible patients for the complete audit. The same research assistant visited each practice and reviewed the charts of all patients on the list. Physicians or office staff were responsible for collecting data from the sequential sample of eligible patients who had appointments during the study period. Data sheets were faxed to the research assistant who entered the data into an Excel spreadsheet. The practices were assessed in 4 blocks of 5 physicians to allow the full and sequential audits to occur simultaneously.
The research assistant collected data on age, date of most recent mammogram, and whether patients had participated in the Ontario Breast Screening Program. Questionnaires were sent to physicians before and after the audit, asking about practice profiles and the effect of the processes of the study on office function. In 1 practice, a second reviewer audited a random sample of the charts of eligible patients to ensure validity of the extraction process.
We hypothesized that there would be a 5% to 10% difference between rates measured by the 2 audit techniques with the sequential approach yielding a higher rate of mammography than the full practice approach.
Data were analyzed using SPSS software for Macintosh (version 9.1). Descriptive statistics were calculated, and correlational analyses were conducted using the difference between the mammography rates based on the 2 sampling strategies as the dependent variable to examine associations with physician and practice variables. All statistical tests were 2-sided and a P value of > .05 was considered statistically significant. Ethics approval was received from the Ethics Review Board of Hamilton Health Sciences.
RESULTS
The audits were carried out between September 2003 and June 2004. Seventeen of the family physicians were affiliated with McMaster University Department of Family Medicine (part-time clinical faculty) as part of a group of community physicians who teach family practice residents in their offices; 3 family physicians were community physicians with no university affiliation (Table 16). Table 2 shows a comparison of the sociodemographic characteristics of eligible patients by sampling method. Mammography rates for each data sampling method are compared in Figure 1. Data were collected by physicians in 10 practices, by nurses in 6 practices, and by other members of staff in 3 practices. The mean time-appropriate mammography rate for the full practice audit was 58.8% and for the sequential sample was 66.4%. The sequential sample approach resulted in a higher rate in all but 4 practices. Data were analyzed at various intervals using a grace period of 6 or 12 months after the time-appropriate 2-year mark. In the full practice audit, the rate of mammography increased to 70.0% during the study, while in the sequential sample, the rate rose to 76.2% by the end of the 12-month grace period.
|
|
|
Figure 2 shows a scatter diagram as well as the regression equation of full audit versus sequential sampling where the cutoff date was 24 months from date of data collection to date of most recent mammogram. Regression analysis, using ordinary least squares linear regression, indicated a positive and significant correlation between mammography rates using the 2 different sampling techniques (r2 = 0.50). Regression analysis indicated that the rate of mammography in the sequential group could be used as a reasonable approximation of the actual rate of mammography based on the full practice audit by employing the following equation: y (actual mammography rate) = 14% + (sequential rate) x 0.66. For example, for a practice with a mammography rate of 50% based on sequential sampling, the full practice rate would be estimated to be 47% (14% + 50% x 0.66 = 47%) and for a practice with a 65% rate, the full practice rate would be estimated to be 56.9%.
|
There was no significant difference in mammography rates for women younger than 60 years and women older than 60 years by either sampling technique. Figure 3 summarizes physicians’ perceptions of the effect of the study on their practices.
|
DISCUSSION
The mean rate of mammography screening was 7.6% higher for the sequential sample than for the full audit. Although the difference in the mean rate was less than 10%, as originally postulated, the amount of variability was greater than expected as was the direction of the difference, which ranged from –6.5% to 24.9%. In 11 practices, mammography rates based on the 2 approaches differed by less than 10%. The anticipated pattern of sequential mammography rates being systematically higher than rates based on a full practice audit was confirmed.
While a rate based on a sequential sample seems to reasonably approximate the actual mammography rate, similar audits need to replicate this study design at other sites and cover other topics to assess its validity and reliability. The fact that the mammography rate based on the sequential sample was higher was anticipated because patients who have office visits are more likely to have more frequent visits and thus are more likely than those who do not have office visits to have mammograms ordered. The regression equation describes a mathematical relationship between mammography rates based on the sequential sample and rates based on a full audit. For physicians who do not have easy access to a full practice audit or who do not have fully computerized medical records, the sequential sampling strategy offers an alternative method of monitoring the care they provide.
Mammography rates based on sequential sampling run the risk of missing women who do not make appointments during the study period. The sequential sampling approach was intended to generate an approximate rate of mammography for a practice, not to find women who had not had time-appropriate mammograms.
In Ontario, incremental financial bonuses are offered as performance incentives to physicians practising under certain new payment models (rostered practices with prescribed rates). The mammography rates we found were similar among practices whether physicians were eligible through rostering for these incentives or not.
Exploratory analysis of factors thought to influence mammography screening rates, such as practice size, age of physician, sex of physician, availability of electronic records, method of payment, and percentage of eligible patients in the practice, did not reveal any notable patterns. One practice identified only 16 eligible patients during the entire study period. This was considerably fewer that the 6 to 10 patients expected to be eligible in each practice each week. As a result, the estimated rate for that practice had a very wide margin of error. The effect of the small numbers was not likely to account for the large difference in rates. Most physicians did not find the sequential audit technique an onerous experience or one that interfered with routine office function. One physician, who did find data collection onerous, had computerized medical records so found a manual data collection process relatively cumbersome. The process generated little anxiety among physicians or their staff, and almost half of them said they were willing to undertake another audit.
Changes in practice and in rates of mammography, while anticipated, were not measured in this study. Some participants commented that it had been a good experience. One said, "I was alerted on how history-keeping on the front of the chart was extremely important," and another commented, "Recording at our end needed change." Others said, the audit was "Worthwhile, surprising to me and adjustments gladly made," and, "[It] provided me with an easy way to be aware of women in my practice needing mammogram."
Limitations
There are several limitations to this study. In the sequential approach, data collection was self-recorded and, as such, was subject to selection bias. Collection of data by physicians’ staff, although not reported as an onerous task, might have been forgotten some days, thus reducing the number of patients in the sequential sample. Lists for the full audit were generated by the practices, which could also make them subject to selection bias. Without a computerized registry or roster, these lists might not truly represent the full number of eligible patients. These biases could affect estimates of mammography rates in the sequential sampling approach. As patients who have appointments are more likely to have mammograms ordered, sampling such patients is likely to produce a higher rate of mammography. All the physicians were practising in south-central Ontario, and most were affiliated with McMaster University so they are not likely to be representative of Canadian family physicians in general. This chart audit used the presence of a report in the chart as evidence of a mammogram being done. It did not capture mammograms that were ordered but not done. Although a doctor’s endorsement of mammography helps encourage women to have screening, some women prefer not to have mammograms.7,8
Conclusion
A mammography rate based on a sample generated through a sequential approach can create a snapshot of a whole practice for physicians. Having peers collecting similar data creates a credible standard of care against which participating physicians can measure their performance without feeling uncomfortable. Further study needs to be done on other audit topics to see how well rates generated through a sequential approach approximate actual rates that would be found in a full practice audit. Questions about how practice might change after an audit need to be explored with participating physicians.
| EDITOR’S KEY POINTS
|
| POINTS DE REPÈRE DU RÉDACTEUR
|
Acknowledgments
This project was supported by a grant from Physicians’ Services Incorporated.
Footnotes
Dr Swanson, principal investigator and author, had the original idea and developed it with Dr Kaczorowski, co-investigator and statistician. Dr Swanson drafted the article, and Dr Kaczorowski reviewed, revised, and edited it. Both the authors agreed on the version submitted for publication.
None declared
*Full text is available in English at www.cfp.ca.
This article has been peer reviewed.
References
- College of Family Physicians of Canada. Standards for accreditation of residency training programs. Mississauga, ON: College of Family Physicians of Canada; 2006. Available from: www.cfpc.ca/local/files/Education. Accessed 2008 Jun 5.
- Lemelin J, Hogg W, Baskerville N. Evidence to action: a tailored multifaceted approach to changing family physician practice patterns and improving preventive care. CMAJ 2001;164(6):757–63.
[Abstract/Free Full Text] - Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2003;(3):CD000259.
- Swanson JG, Edey L, Kaczorowski J. Quality assurance through reflective change (QuARC): involving family medicine community-based preceptors in a continuous quality improvement programme. Qual Prim Care 2006;14(3):155–60.
- Morrison BJ; Canadian Task Force on the Periodic Health Examination, Screening for breast cancer. Canadian guide to clinical preventive health care. Ottawa, ON: Health Canada; 1994. p. 788–95.
- College of Family Physicians of Canada, Canadian Medical Association, Royal College of Physicians and Surgeons of Canada. 2004 National Physician Survey. Mississauga, ON: College of Family Physicians of Canada; 2004. Available from: www.nationalphysiciansurvey.ca/nps/2004_Survey/2004nps-e.asp. Accessed 2008 Jun 5.
- Andrews FJ. Pain during mammography: implications for breast screening programmes. Australas Radiol 2001;45(2):113–7.[Medline]
- Miller D, Martin I, Herbison P. Interventions for relieving the pain and discomfort of screening mammography. Cochrane Database Syst Rev 2002;(4):CD002942.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | CONTACT US | FEEDBACK | SUBSCRIPTIONS | CURRENT ISSUE | PAST ISSUES | SEARCH |