ABSTRACT
OBJECTIVE To develop a survey instrument with good internal consistency and test-retest reliability to explore the level of knowledge among Nova Scotia family physicians concerning the risk factors, signs and symptoms, and treatment of otitis media and the use of pneumatic otoscopy.
DESIGN Prospective cohort design.
SETTING Fee-for-service family practices in Nova Scotia.
PARTICIPANTS A convenience sample of 25 family physicians.
MAIN OUTCOME MEASURES Test-retest reliability and internal consistency of the survey.
RESULTS The constructs including “signs and symptoms of otitis media with effusion” and “comprehensive knowledge scores” showed excellent internal consistency with Kuder-Richardson 20 scores greater than 0.7 whereas the construct “signs and symptoms of acute otitis media” has a Kuder-Richardson 20 score of 0.54 after deletion of several items. The Cohen κ and Spearman ρ tests showed the survey has very good test-retest reliability.
CONCLUSION The questionnaire that we developed proved to have very good internal consistency and test-retest reliability. We hope to use this questionnaire to explore the practice patterns of family physicians in managing otitis media disease.
Otitis media (OM) is an infectious disease of the middle ear that presents with or without acute symptoms.1 It is the most common diagnosis among children presenting at family physicians’ offices in the United States and the second most common diagnosis overall.2 It is also the most common childhood infectious disease for which antibiotics are prescribed.3 The 2 main categories are acute otitis media (AOM) and otitis media with effusion (OME). Acute otitis media is characterized by the presence of fluid in the middle ear accompanied by signs and symptoms of acute infection. Otitis media with effusion is characterized by the presence of fluid in the middle ear without evidence of acute infection.4 It is clinically important to distinguish these 2 types because antimicrobial therapy is seldom required in the management of OME.5
Because OM is so common, the World Health Organization has designated OM management skills a priority for primary care providers to develop.6 For optimal accuracy, the diagnosis is based on clinical symptoms combined with visual examination of the tympanic membrane using pneumatic otoscopy.2,7 In AOM there is rapid onset, a bulging eardrum with poor mobility, and usually a reddish or yellowish discolouration due to the presence of pus in the middle ear space. Patients with AOM might appear acutely ill with fever and irritability. In OME the eardrum tends to be normally positioned, to be retracted, or to have reduced mobility, but there is no acute inflammation.8
Although pneumatic otoscopy enhances the accuracy of diagnosis, many physicians do not use this technique.9 Improved diagnostic accuracy would lead to improved management of AOM and OME, a more judicious use of antibiotics, less antibiotic resistance, and ultimately an improvement in the health of children with OM. A more prudent use of antibiotics would also lower the economic burden of this disease. In 1994 it was estimated that the total cost of OM in Canada was $611 million, or approximately 0.08% of the gross domestic product. Increased use of pneumatic otoscopy could contribute to reducing these costs.10–12
Assessment of physicians’ knowledge, attitudes, and behaviour is a necessary first step toward understanding why gaps appear between evidence and practice. Such gaps are related to factors that prevent (or fail to promote) behavioural change. These factors were categorized by Green and Kreuter into 3 types: predisposing factors, enabling factors, and reinforcing factors.13 This was part of a behavioural framework known as the precede-proceed model. This model provides a strategy for health promotion whereby behavioural change is best achieved by targeting the barriers that are prevalent in the population. Knowledge of these barriers is critical for planning an effective intervention strategy. For optimal diagnosis and management of OM, the predisposing factors would be related physician beliefs and training; the enabling factors would be equipment availability, patient cooperation, and practice facilitation; and the reinforcing factors would be collegial support and positive parental feedback.
In this paper we report our experience with developing and pilot-testing a survey instrument to assess family physicians’ knowledge of first episodes of OM (AOM and OME) in children aged 2 to 6 years; measure the prevalence of the use of pneumatic otoscopy; and determine barriers to optimal diagnosis and management of this disease, in particular the factors that influence family physicians’ use of pneumatic otoscopy.
Ethics approval to develop and test the instrument was obtained from Dalhousie University Health Sciences Research Ethics Board in Halifax, NS.
METHODS
Survey design and pilot test
An initial draft of a self-reported, mailed questionnaire was designed by the authors. The first part contained questions relating to physicians’ practice settings, training, and demographic characteristics. The second part contained questions about the epidemiology, diagnosis, and management of OM. Some of the questions were specific to AOM, while others were specific to OME. A convenience sample of 25 family physicians who practise in the Halifax regional municipality were selected to test the questionnaire. All 25 participants were registered with the College of Physicians and Surgeons of Nova Scotia in 2003. We did not include physicians who were either in training or not currently in active practice, as determined by their status in the College of Physicians and Surgeons of Nova Scotia register, or who were practising in an area of medicine that does not generally involve assessment of OM, such as occupational medicine. Each participant received a package containing a cover letter, a questionnaire, and a consent form. Each was informed that responses would be held in confidence and that only aggregate data would be reported. The completed consent forms and questionnaires were returned by prepaid registered mail. The statistical analyses were performed using SAS and SPSS software under licence to Dalhousie University.
Data analysis
Three knowledge scores (or scales) were defined. One was based on the signs and symptoms of AOM. Another was based on the signs and symptoms of OME. The third was a comprehensive score that included both of the above and questions about agents, risk factors, appropriateness of antimicrobial therapy, and effectiveness of amoxicillin. Signs and symptoms were combined because clinicians often describe a disease in terms of these 2 entities and only rarely are the signs (objective indicators) separated from the symptoms (subjective indicators). All of the items were binary (true-false, yes-no, present-absent). To test the homogeneity of the items in each knowledge scale, the Kuder-Richardson 20 (KR20) statistic was used. The KR20 measures the internal consistency of the scale with regard to a specific construct, which in this case is a particular area of knowledge. High values of KR20 indicate that the items are strongly correlated, suggesting that some are redundant and could be omitted. Low values of KR20 indicate that the items are poorly correlated. If KR20 decreases markedly when a particular item is added, the item does not measure the same construct and should perhaps be dropped or considered separately. Each item’s effect on the value of KR20 was examined separately. Aday recommends aiming for values of KR20 between 0.7 and 0.9.14
The test-retest reliability of the responses was examined by asking the 25 physicians to complete the survey instrument a second time 14 days later. An interval of 14 days has frequently been used in studies of test-retest reliability.15 Two statistics were used to measure test-retest reliability. For items with binary responses, we used Cohen κ statistic. For items with ordinal responses (coded 1, 2, 3, 4, 5) we used Spearman correlation coefficient. However, not all variables were analyzed. Those that were deemed on visual inspection to have a very high level of agreement were excluded and only those test-retest responses that appeared to be somewhat discordant were analyzed. A value of 0.4 was chosen as a cutoff for discriminating between adequate and inadequate reliability.
RESULTS
All 25 participants completed both the test questionnaire and the retest questionnaire. Table 1 presents the internal consistency (KR20) values for the knowledge scales. The knowledge scale for “signs and symptoms of OME” had a KR20 of 0.90. The knowledge scale for “signs and symptoms of AOM” had a value of 0.54, even after 6 of the original items were removed. The comprehensive knowledge score had a value of 0.75, probably buoyed by the high KR20 in the OME scale.
Measures of internal consistency using the KR20 statistic
Table 2 presents the levels of agreement between test and retest using Cohen κ statistic. Most of the items had good agreement, and we concluded that the relevant knowledge had been reliably measured. There were some exceptions. Surprisingly, male sex and family history of OM as risk factors for AOM had low κ scores. Diminution of light reflex as a sign of the disease had poor agreement in both the AOM and OME scales.
Not all variables were analyzed. Those that were deemed on visual inspection to have a very high level of agreement were excluded and only those test-retest responses that appeared discordant were analyzed.
Table 3 lists the Spearman correlation coefficients for the attitudinal constructs, comprehensive knowledge score, duration of antimicrobial therapy, interval between initial examination and reexamination, and duration of follow-up before referral for ventilation tube placement. Almost all variables had a high correlation between test and retest. One exception was the item “I find pneumatic otoscopy too technically difficult to perform” (ρ = 0.32). This could reflect a change in the physicians’ perspectives between test and retest, prompted by reflection on this issue after their attention had been drawn to it in the test.
Test-retest Spearman correlation for items with an ordinal response
DISCUSSION
We have developed an instrument that explores physicians’ knowledge and beliefs in the diagnosis of OM. Overall it has good internal consistency and good test-retest reliability. We deleted certain items from the “signs and symptoms of AOM” that would have caused a low internal consistency in that scale. The deleted items were related to diarrhea, stomach pain, emesis, location of the light reflex, a retracted eardrum, and presence of the light reflex. The first 3 are known to be associated with AOM, but they are nonspecific and can be linked to other common pediatric diseases as well, such as infectious gastroenteritis. Also, a retracted tympanic membrane is a sign of OME and not AOM. The presence or absence of a light reflex is a relatively nonspecific sign.
Studies have shown that combining the clinical information about colour, position, and mobility of the tympanic membrane significantly enhances the diagnostic power. A Finnish study16 reported a prevalence of middle ear effusion ranging from 69.1% to 84.9%, as diagnosed by the criterion standard of myringotomy and aspiration. Furthermore, cloudiness combined with bulging and distinct immobility was associated with a 98.8% probability of AOM. On the other hand, when the tympanic membrane showed normal mobility, the probability fell to below 40%. These results provide clear evidence that the diagnostic accuracy is enhanced when the tympanic membrane is well visualized and its mobility is directly observed.16
With regard to test-retest reliability, it was surprising to find that male sex as a risk factor for AOM achieved a κ score of only 0.117. This could be because physicians believe sex does not affect the diagnosis of OM. The poor κ score for family history of OM can be explained in a similar way.
Limitations
We followed recommendations of ways to increase response rates, including the use of monetary and non-monetary incentives, use of self-addressed, stamped envelopes, an assurance of confidentiality, and university sponsorship.17 In our literature review, we were unable to obtain meaningful guidance on the optimum number of subjects used for pretesting. Indeed, the number often becomes a matter of personal judgment.18 Hence, we recruited 25 physicians for our endeavour because we thought that a small number would reduce the risk of nonresponse—and all of them responded to our survey.
Our study employed a convenience sample that might differ substantially from the general physician population that we intend to sample at a later date with our pretested survey. Therefore, the effect of selection bias remains unknown.
In addition, statistics cannot be solely relied upon to ascertain the validity of a survey. As noted, although male sex and family history as risk factors for OM had low test-retest reliability scores, we believe that they should not be omitted from the survey because they have been shown elsewhere to be valid.19
Conclusion
Otitis media is a common medical diagnosis. We are unaware of any published survey that examines physician knowledge, beliefs, and practices regarding the diagnosis and management of AOM and OME. We used a combination of statistical analysis and clinical judgment to remove items from our instrument that did not contribute to its robustness. We intend to administer this instrument to a more representative sample of family physicians later to determine the knowledge, beliefs, and practices of Canadian family physicians in the management of this important childhood disease.
Notes
EDITOR’S KEY POINTS
-
Otitis media (OM) is the most common medical diagnosis for children in North America, yet there is much debate concerning its management. Improving diagnostic accuracy in management of OM could increase appropriate use and, more important, decrease inappropriate use of antimicrobial therapy for this condition, which is recommended by guidelines for the diagnosis of OM.
-
Assessment of physicians’ knowledge, attitudes, and behaviour is a necessary first step toward understanding why gaps appear between evidence and practice. The authors set out to design and test a survey instrument to assess these gaps in the management of OM. The authors used a combination of statistical analysis and clinical judgment to remove items from the instrument that did not contribute to its robustness.
-
To pilot-test the survey the study employed a convenience sample that might differ substantially from the general physician population that the instrument is meant to study, so although the survey’s reliability and consistency were found to be high, the effects of selection bias remain unknown.
POINTS DE REPÈRE DU RÉDACTEUR
-
L’otite moyenne (OM) est le diagnostic médical le plus fréquent chez les enfants d’Amérique du Nord et pourtant son traitement est l’objet de beaucoup de controverse. Un diagnostic plus précis de l’OM pourrait favoriser une utilisation plus judicieuse des antibiotiques et, ce qui est plus important, réduire leur utilisation inappropriée, conformément aux directives sur le diagnostic de l’OM.
-
Une évaluation des connaissances, attitudes et comportements des médecins constitue une étape initiale essentielle à la compréhension des raisons de ces écarts entre les données probantes et la pratique. Les auteurs ont entrepris d’élaborer et de tester un instrument d’enquête pour évaluer ces écarts dans le traitement de l’OM. Ils ont utilisé une combinaison d’analyses statistiques et de jugement clinique pour éliminer de l’instrument les éléments qui ne contribuaient pas à sa robustesse.
-
Pour faire un essai pilote de l’enquête, l’étude a utilisé un échantillon arbitraire qui pourrait différer considérablement de la population générale des médecins que l’instrument veut étudier, de sorte que même si la fiabilité et la cohérence de l’enquête se sont révélées élevées, on ignore quels peuvent être les effets de biais de sélection.
Footnotes
-
Contributors
Drs Lee, Flowerdew, and Delaney contributed to study design, acquisition and interpretation of data, and review of the manuscript.
-
Competing interests
None declared
-
*Full text is available in English at www.cfp.ca.
-
This article has been peer reviewed.
- Copyright© the College of Family Physicians of Canada
In this issue
Jump to section
Related Articles
Cited By...
- No citing articles found.