Abstract
Problem addressed Although e-learning programs are popular and access to electronic knowledge resources has improved, raising awareness about updated therapeutic recommendations in practice continues to be a challenge.
Objective of program To raise awareness about and document the use of therapeutic recommendations.
Program description In 2010, family physicians evaluated e-Therapeutics (e-T) Highlights with a Web-based tool called the Information Assessment Method (IAM). The e-T Highlights consisted of information found in the primary care reference e-Therapeutics+. Each week, family physicians received an e-mail containing a link to 1 Highlight from a different chapter of e-Therapeutics+. Family physicians received continuing medical education credits for each Highlight they rated with the IAM. Of the 5346 participants, 85% of them were full-time or part-time practitioners. A total of 31 429 Highlights ratings were received in 2010 (median of 2 ratings per participant, range 1 to 49). Among participants who rated more than 2 Highlights, the median number of ratings was 7 (mean 11.9). The relevance of the information from individual Highlights varied widely; however, for 90% of the rated Highlights participants indicated total or partial relevance of the information for at least 1 patient. For 41% of rated Highlights, participants expected patient health benefits to result from implementing the recommendation, such as avoiding an unnecessary or inappropriate treatment, or a preventive intervention.
Conclusion This continuing medical education program stimulated family physicians to rate therapeutic recommendations that were delivered weekly via e-mail. The process of rating e-T Highlights with the IAM raised awareness about treatment recommendations and documented self-reported use of this information in practice.
With a national continuing medical education (CME) program, we addressed a long-standing problem for clinical medicine: raising awareness about updated treatment recommendations. While e-learning programs are popular and access to electronic knowledge resources has improved, we know little about strategies to optimize the application of clinical information such as the use of treatment recommendations in practice.1,2
Research on the effect of e-mail alerts and electronic knowledge resources led us to develop and validate a tool called the Information Assessment Method (IAM).3 The IAM contains a brief questionnaire linked to 1 specific object of clinical information (eg, a synopsis) delivered via e-mail. Health professionals use the IAM questionnaire to evaluate clinical information in relation to 4 constructs:
cognitive impact of the information (eg, learning something new),
relevance of the information for at least 1 patient,
intended use of the information for a specific patient (eg, modifying treatment), and
patient benefits (eg, avoiding unnecessary treatment).
Since 2006, physicians have been using the IAM to document reflective learning in e-learning programs that provide Mainpro credits. In this paper, we describe a program designed to raise awareness about and document the use of clinical information from e-Therapeutics (e-T) Highlights, an online resource containing therapeutic recommendations. The program recruited members of the College of Family Physicians of Canada (CFPC) to evaluate therapeutic information delivered via e-mail.
Program description
This CME program was implemented in the context of a funded study that used a participatory research approach with the Canadian Pharmacists Association and the CFPC. Beginning in early 2010, the Canadian Pharmacists Association delivered weekly Highlights via e-mail to CFPC members. Information for Highlights was gathered from e-Therapeutics+,4 the online version of the book Therapeutic Choices5 published by the Canadian Pharmacists Association. On viewing the Highlights e-mail, participants had access to the chapter of e-Therapeutics+ from which the information was derived. Chapter access was granted for 2 months from the date the Highlights e-mail was sent. Highlights were linked to the IAM so that participants could document their reflective learning.
The e-T Highlights were chosen for this program by clinical editors at the Canadian Pharmacists Association and vetted by 1 CFPC member (co-author B.M.) before dissemination. Beginning January 20, 2010, all CFPC members with valid e-mail addresses received a weekly invitation to participate in rating that week’s Highlight. Members of the CFPC who accepted the invitation completed a short demographic survey after clicking on the Earn Mainpro Credits button adjacent to the first Highlight they read. This was followed by their first IAM questionnaire, which asked participants to indicate whether that particular Highlight had an effect on them or their practice (yes, no, possibly) and whether that Highlight was relevant for at least 1 of their patients. With each subsequent Highlight that was read, only the IAM questionnaire was presented. For each submitted IAM questionnaire, family physicians earned 0.1 Mainpro-M1 credits, following a process similar to that used for other e-learning programs.6
One Highlight was delivered weekly to 17 000 e-mail addresses (each address belonging to 1 CFPC member). To read the Highlight embedded within its e-Therapeutics+ chapter, participants had to open the e-mail and click on the hyperlinked Highlight title. Data collection ran from January 20 to December 31, 2010. Data from the demographic characteristics questionnaire and the Highlights ratings completed with the IAM were analyzed descriptively.
A total of 5346 CFPC members completed the demographic characteristics questionnaire and submitted at least 1 rating of a Highlight. The participation rate was 31% (5346 of 17 000). Ninety-one percent of participants reported they were in family or general practice, while 85% were in full-time or part-time practice. Slightly more than half of participants were women (53%), and 56% of participants were CFPC Certificants.
There were 31 429 ratings of 49 Highlights in 2010, for an average of 667 ratings per Highlight (range 415 to 1176). The median number of Highlight ratings per participant was 2 (range 1 to 49). Among CFPC members who submitted more than 2 ratings, the median number of ratings per participant was 7 (mean 11.9).
Participants believed the clinical information from e-T Highlights was totally or partially relevant for at least 1 patient 90% of the time. To understand which Highlights were most or least relevant according to CFPC members, we calculated a relevance statistic that ranged from 0 to 1, called the clinical relevance of information index. The clinical relevance of information index value ranged from 0.458 to 0.899 per Highlight (mean 0.713). The following 2 Highlights had the lowest and the highest relevance rating, respectively:
Tizanidine is a good first-line treatment, or it can be combined with baclofen, as they have different sites of action.
Penicillin is the drug of choice for streptococcal sore throat. Although cephalosporins are effective, they should not replace penicillin as the drug of choice.
With regard to knowledge translation, for 59% of the rated Highlights, participants said the information would be used for at least 1 patient. For 41% of the rated Highlights, participants expected patient health benefits owing to Highlight recommendations, such as avoiding an unnecessary or inappropriate treatment, or a preventive intervention (Figure 1).
Value of the information from e-Therapeutics Highlights
Discussion
The e-T Highlights, which were written and reviewed by expert physicians and pharmacists, were rated by more than 5000 CFPC members in 2010. In addition to ratings of relevance and cognitive impact, participants often reported that therapeutic recommendations from Highlights would be used or applied in practice. While this led some participants to expect health benefits for their patients, our review of the literature shows studies have not yet demonstrated that these subjective physician experiences translate to action in practice.7
A strength of this program was the rating of Highlights using a validated method to document reflective learning. In addition, Highlights were delivered more slowly compared with other programs, at the rate of 1 Highlight per week. While we do not know the optimal speed for e-mail delivery of brief educational content, spaced online education has been shown to improve knowledge retention in medical students.8 Concurrent to this Highlights program, nearly 3000 family physicians used the IAM to rate Patient-Oriented Evidence that Matters (POEMs) synopses in 2010. Given the differences between family physicians who rated POEMs and those who rated Highlights (Table 1), a comparison of ratings of these 2 types of clinical information must be interpreted with caution. Furthermore, POEMs are different from Highlights, as the former are synopses of recently published original research articles with a focus not limited to therapeutics. However, we provide Figure 2 to simply illustrate the results that might be obtained, for example, in randomized trials of e-mail alerting services for a variety of health professionals. In Figure 2, the clinical relevance of POEMs is distributed over a wider range than the clinical relevance of Highlights is. In addition, the clinical relevance of POEMs delivered in 2010 was generally lower than that of Highlights. This finding is not surprising considering that Highlights are therapeutic recommendations for practice, authored by and for Canadian physicians.
Distribution of relevance ratings for 2 types of clinical information delivered to FPs and GPs: Each point represents a single e-mail alert rated in 2010.
POEMs—Patient-Oriented Evidence that Matters.
Characteristics of the FPs and GPs who rated POEMs and e-Therapeutics Highlights in 2010
Beyond increasing the awareness of Highlights, this e-learning program improved access to therapeutic recommendations, as before the program, physicians could access recommendations from Therapeutic Choices only by obtaining a print copy. By having the IAM link within the e-T Highlights that were delivered via e-mail, participants had the opportunity to comment on what they had read. Their comments, such as suggestions for additional content, were relayed to the editor-in-chief and clinical editors. Consequently, the editors were able to bring further improvements to e-Therapeutics+.9,10
Limitations
Our data are self-reported. As such, the health benefits for specific patients were “expected” and not objectively documented. Whether patient health outcomes are influenced by the push of clinical information from one electronic knowledge resource is an open question for future research. In addition, we did not measure how frequently Highlights were read and not rated. If Highlights are more likely to be rated by family physicians when they are relevant for at least 1 patient, then our data overestimate the clinical relevance of each Highlight.
Continuing medical education programs do not randomly select participants. Therefore, physicians in this program were more motivated to receive e-mail alerts, as well as read and rate Highlights. Nevertheless, knowledge that the Highlights program might lead to health benefits for specific patients is potentially important to improving the outcomes of clinical practice. Given these expected benefits for patients, the Highlights e-learning program has been sustained by the Canadian Pharmacists Association and the CFPC. In addition, a new program for Canadian pharmacists—mirroring the current program for family physicians—was launched by the Canadian Pharmacists Association in 2012. The continuation of this program opens the door to further research, as ratings are subjective physician experiences that might be objectively confirmed.
Conclusion
This CME program stimulated family physicians to rate therapeutic recommendations delivered to them weekly via e-mail. The process of rating Highlights raised awareness of treatment recommendations and documented self-reported use of this information in practice. For participants, the IAM helped to document their reflective learning for Mainpro. For information providers, the IAM can reveal the clinical relevance of 1 specific information object (eg, 1 Highlight), as well as the overall clinical relevance of CME programs.
Acknowledgments
This continuing medical education program was implemented in the context of a study funded by the Canadian Institutes of Health Research. Opinions are those of the authors and do not imply endorsement by the College of Family Physicians of Canada.
Notes
EDITOR’S KEY POINTS
In this program, family physicians evaluated therapeutic information from the online resource e-Therapeutics Highlights with the Information Assessment Method (IAM).
By rating the information from Highlights, participants earned continuing medical education credits for each completed IAM questionnaire. The IAM questionnaire asked participants whether the information from Highlights had an effect on them or their practices and whether it was relevant for at least 1 of their patients.
The strengths of this program include raising awareness of and improving access to therapeutic recommendations, as well as documenting self-reported use of this information in practice.
POINTS DE REPÈRE DU RÉDACTEUR
Dans ce programme, des médecins de famille ont utilisé la Méthode d’évaluation des informations (IAM) pour évaluer l’information thérapeutique contenue dans l’e-Therapeutics Highlight, une ressource sur le Web.
En évaluant l’information contenue dans les Highlights, les participants ont obtenu des crédits de formation médicale continue pour chaque questionnaire de l’IAM complété. Dans ces questionnaires, on demandait aux participants si l’information des Highlights les affectait ou si elle affectait leur pratique, et si elle était pertinente pour au moins un de leurs patients.
Parmi les principaux avantages de ce programme, mentionnons le fait de sensibiliser les participants aux recommandations thérapeutiques et de rendre ces informations plus accessibles, mais aussi de documenter les déclarations des participants sur l’usage qu’ils en font dans leur pratique.
Footnotes
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
Contributors
Dr Grad, Dr Pluye, Ms Repchinsky, Ms Jovaisas, Dr Marlow, Mr Shulha, and Dr de Gaspé Bonar contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission. Drs Ricarte and Barbosa Galvão contributed to the analysis and preparation of the manuscript.
Competing interests
Drs Grad and Pluye receive unrestricted research grants from the Canadian Pharmacists Association and the Canadian Medical Association. These funds provide bursaries in support of graduate student research at McGill University. At the time of writing, Ms Repchinsky was Editor-in-Chief, Ms Jovaisas was Clinical Editor, and Dr de Gaspé Bonar is Senior Director of Digital Publishing Solutions, all at the Canadian Pharmacists Association.
- Copyright© the College of Family Physicians of Canada
