Family physicians have access to numerous evidence-based electronic knowledge resources (EKRs) for retrieving treatment recommendations when needed. In addition, resources received via e-mail, such as e-Therapeutics Highlights, help FPs stay up to date. The Highlights are selected from the content of e-Therapeutics+, a Canadian primary care reference written, updated, and reviewed by Canadian physicians and pharmacists. Accessing EKRs contributes to continuing medical education (CME) and knowledge translation.1 In 2 recent longitudinal studies,2,3 one of which appears in this issue of Canadian Family Physician (page e258),2 FPs anticipated health benefits for their patients arising from the use of research-based information received by e-mail. The results suggest FPs use e-mailed evidence in patient encounters at a later time (ie, put evidence into practice), which represents 1-way knowledge translation.
This commentary presents our work on 2-way knowledge translation, which we believe can lead to “better-than-best” evidence (Table 1). We define 2-way knowledge translation as a continuous interactional process between information providers who update and deliver the “best” available evidence, and information users who assess this evidence and submit constructive feedback. In turn, information providers might use this feedback to optimize their evidence, which is then made available online for further retrieval by information users when needed. In other words, research-based information delivered by e-mail can be enhanced by experience-based information from health professionals.
Comparison of 1- versus 2-way knowledge translation
Background
As shown in Figure 1, 2-way knowledge translation assumes that information users, such as practising FPs, have relevant expertise. To paraphrase Richard Smith, former BMJ editor, most FPs are not scientists.4 Most FPs do not have graduate research training, and they are rarely involved in research or research synthesis; however, they have “special technical expertise in virtue of experience,”5 which legitimates their discourse about research-based information. Research-based information involves people with 3 types of expertise: experiential, contributory, and interactive. People with experiential expertise (eg, FPs) have practical experience in the topic covered by the research-based information. Research scientists, who have “enough expertise to contribute to the science of the field,”5 have contributory expertise. People with interactive expertise (eg, editors) have “enough expertise to interact interestingly”5 with researchers, and can carry out research synthesis. These 3 types of expertise are not mutually exclusive “ideal types.” For instance, someone might combine all 3 types of expertise, and be concomitantly researcher, practitioner, and clinical editor. For a successful 2-way knowledge translation process, our experience suggests information providers must recognize information users as technically qualified experts who bring suggestions to clarify or add content, as well as editorial attention to the most recent developments in the field, contradictory evidence, or regional differences in recommendations.
Recognizing 3 types of expertise for 2-way knowledge translation
Longitudinal evaluation of a CME program
We explored whether stimulated feedback from FPs could further optimize research-based information delivered via e-mail. Three organizations took part in a prospective longitudinal evaluation of a CME program: the Canadian Pharmacists Association (CPhA) in Ottawa, Ont; the College of Family Physicians of Canada (CFPC) in Mississauga, Ont; and McGill University in Montreal, Que. Ethics approval was obtained from the Institutional Review Board of the Faculty of Medicine at McGill University. The CPhA publishes e-Therapeutics+, an EKR that provides updated evidence-based therapeutic recommendations. Fifty-one information Highlights were selected from the content of e-Therapeutics+, and delivered by e-mail to about 17 000 CFPC members in 2010.2
Physician feedback was stimulated by and collected with the Information Assessment Method (IAM) questionnaire. The IAM questionnaire is a validated method for assessing the value of information in terms of its relevance, cognitive impact, use, and expected health benefits.6 Participants were asked to rate each Highlight using the IAM questionnaire, and could also submit free-text comments. For each rated Highlight, respondents received 0.1 Mainpro-M1 credits. In 2010, 5346 CFPC members rated at least 1 Highlight using the IAM questionnaire, which made it one of the most popular CME programs of its kind in Canada.2,3
All free-text comments were reviewed in order to select “potentially constructive feedback” (ie, comments that could contribute to improving e-Therapeutics+). The selection of potentially constructive feedback comments was based on criteria developed in a partnership between McGill researchers and CPhA editors. These comments were then sent to CPhA editors, who investigated further and worked with CPhA’s expert authors and reviewers as needed to revise the content of e-Therapeutics+.
Constructive feedback comments were defined as comments that actually led to a change in content. These comments were coded to 4 types of constructive feedback by 2 independent raters using a coding scheme (D.L.T., P.P.). The coding scheme was previously developed in 3 steps: using a sample of comments, 2 researchers proposed initial codes, which were refined by researchers and CPhA editors, then tested and revised by 2 researchers (P.P., R.G., C.R., B.J., D.L., D.L.T., V.G.). Interrater reliability scores (Cohen κ statistics) were calculated using SPSS software, version 19. Reliability scores were interpreted as indicative of substantial agreement (0.61 to 0.80) or perfect agreement (0.81 to 1.00).7
Constructive feedback comments received
Of the 31 429 completed questionnaires, 4166 (13.3%) contained free-text comments. A total of 682 (2.2%) comments were identified as potentially constructive feedback and reported to the CPhA. Overall, 126 (0.4%) comments were confirmed as constructive feedback comments that led to changes in e-Therapeutics+. In total, 116 FPs submitted constructive feedback comments on 31 Highlights. They submitted a mean of 1.1 constructive feedback comments (minimum 1; maximum 3). Each of the 31 Highlights was associated with a mean of 4.1 constructive feedback comments (minimum 1; maximum 25). We categorized all 126 constructive feedback comments into the 4 following types, with Cohen κ statistics calculated for each type: additional content; reservations or disagreement; contradictory evidence; and need for clarification. There were 79 suggestions for additional content (62.7% of comments; κ = 0.82), 26 comments that expressed reservations or disagreed with the content (20.6% of comments; κ = 0.74), 24 suggestions to consider contradictory evidence (19.0% of comments; κ = 1.00), and 9 comments regarding the need for clarification of content (7.1% of comments; κ = 0.65). Given that 12 constructive feedback comments were coded to 2 types, the cumulative percentage is greater than 100%. Based on the κ statistics, there was substantial or perfect agreement between raters; thus, the proposed coding scheme to categorize constructive feedback comments is reliable, and is currently used to manually select potentially constructive feedback for the CPhA.
Conclusion
Constructive feedback represented 3.0% (126 of 4166) of all written comments. The information provider, CPhA, valued the systematic collection and integration of user feedback in their editorial process. This 2-way knowledge translation process appears to be unique with regard to management of EKRs.8 We reviewed the literature and could not find other work examining how health professionals’ feedback could be used to improve EKRs.
Before using the IAM, the CPhA occasionally received user comments (2 or 3 per week). The IAM questionnaire stimulated and collected user feedback in response to e-Therapeutics Highlights and allowed for 2-way knowledge translation, which is beneficial to both the information provider and the end user. From the CPhA point of view, the editorial process and the content of their knowledge resource was optimized by user feedback. As for the knowledge users, CFPC members, their voices were heard and they contributed to creating “better-than-best” evidence. In line with the literature on relational marketing,9 being open to user feedback and handling such feedback in a timely manner can improve knowledge resources and aid information providers in sustaining relationships with the users by valuing their expertise.
Acknowledgments
Dr Pluye holds a New Investigator Award from the Canadian Institutes of Health Research. This work is derived from a project supported by the Canadian Institutes of Health Research End-of-Grant Knowledge Translation program. The Canadian Pharmacists Association editors who contributed to this study are Angela Ross, Jo-Anne Hutsul, Geoff Lewis, Hélène Perrier, and Marc Riachi.
Footnotes
This article has been peer reviewed.
Cet article se trouve aussi en français à la page 425.
Competing interests
None declared
The opinions expressed in commentaries are those of the authors. Publication does not imply endorsement by the College of Family Physicians of Canada.
- Copyright© the College of Family Physicians of Canada
