Skip to main content

Main menu

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums

User menu

  • My alerts

Search

  • Advanced search
The College of Family Physicians of Canada
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums
  • My alerts
The College of Family Physicians of Canada

Advanced Search

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • RSS feeds
  • Follow cfp Template on Twitter
  • LinkedIn
  • Instagram
Research ArticleProgram Description

Physician assessments of the value of therapeutic information delivered via e-mail

Roland Grad, Pierre Pluye, Carol Repchinsky, Barbara Jovaisas, Bernard Marlow, Ivan L. Marques Ricarte, Maria Cristiane Barbosa Galvão, Michael Shulha and James de Gaspé Bonar
Canadian Family Physician May 2014; 60 (5) e258-e262;
Roland Grad
Associate professor in the Department of Family Medicine at McGill University in Montreal, Que
MD CM MSc CCFP FCFP
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: roland.grad@mcgill.ca
Pierre Pluye
Associate professors in the Department of Family Medicine at McGill University in Montreal, Que.
MD PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Carol Repchinsky
At the time of writing, Editor-in-Chief, at the Canadian Pharmacists Association in Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Barbara Jovaisas
At the time of writing, Clinical Editor, at the Canadian Pharmacist Association in Ottawa, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bernard Marlow
Director of Continuing Professional Development at the College of Family Physicians of Canada in Mississauga, Ont, at the time of this study.
MD CCFP FCFP FACME
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ivan L. Marques Ricarte
Associate Professor at the University of Campinas in São Paulo, Brazil.
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Maria Cristiane Barbosa Galvão
Professor at the University of São Paulo in Brazil.
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Shulha
Research Associate at the Herzl Family Practice Centre in Montreal.
MLIS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
James de Gaspé Bonar
Senior Director of Digital Publishing Solutions at the Canadian Pharmacists Association.
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF
Loading

Abstract

Problem addressed Although e-learning programs are popular and access to electronic knowledge resources has improved, raising awareness about updated therapeutic recommendations in practice continues to be a challenge.

Objective of program To raise awareness about and document the use of therapeutic recommendations.

Program description In 2010, family physicians evaluated e-Therapeutics (e-T) Highlights with a Web-based tool called the Information Assessment Method (IAM). The e-T Highlights consisted of information found in the primary care reference e-Therapeutics+. Each week, family physicians received an e-mail containing a link to 1 Highlight from a different chapter of e-Therapeutics+. Family physicians received continuing medical education credits for each Highlight they rated with the IAM. Of the 5346 participants, 85% of them were full-time or part-time practitioners. A total of 31 429 Highlights ratings were received in 2010 (median of 2 ratings per participant, range 1 to 49). Among participants who rated more than 2 Highlights, the median number of ratings was 7 (mean 11.9). The relevance of the information from individual Highlights varied widely; however, for 90% of the rated Highlights participants indicated total or partial relevance of the information for at least 1 patient. For 41% of rated Highlights, participants expected patient health benefits to result from implementing the recommendation, such as avoiding an unnecessary or inappropriate treatment, or a preventive intervention.

Conclusion This continuing medical education program stimulated family physicians to rate therapeutic recommendations that were delivered weekly via e-mail. The process of rating e-T Highlights with the IAM raised awareness about treatment recommendations and documented self-reported use of this information in practice.

With a national continuing medical education (CME) program, we addressed a long-standing problem for clinical medicine: raising awareness about updated treatment recommendations. While e-learning programs are popular and access to electronic knowledge resources has improved, we know little about strategies to optimize the application of clinical information such as the use of treatment recommendations in practice.1,2

Research on the effect of e-mail alerts and electronic knowledge resources led us to develop and validate a tool called the Information Assessment Method (IAM).3 The IAM contains a brief questionnaire linked to 1 specific object of clinical information (eg, a synopsis) delivered via e-mail. Health professionals use the IAM questionnaire to evaluate clinical information in relation to 4 constructs:

  • cognitive impact of the information (eg, learning something new),

  • relevance of the information for at least 1 patient,

  • intended use of the information for a specific patient (eg, modifying treatment), and

  • patient benefits (eg, avoiding unnecessary treatment).

Since 2006, physicians have been using the IAM to document reflective learning in e-learning programs that provide Mainpro credits. In this paper, we describe a program designed to raise awareness about and document the use of clinical information from e-Therapeutics (e-T) Highlights, an online resource containing therapeutic recommendations. The program recruited members of the College of Family Physicians of Canada (CFPC) to evaluate therapeutic information delivered via e-mail.

Program description

This CME program was implemented in the context of a funded study that used a participatory research approach with the Canadian Pharmacists Association and the CFPC. Beginning in early 2010, the Canadian Pharmacists Association delivered weekly Highlights via e-mail to CFPC members. Information for Highlights was gathered from e-Therapeutics+,4 the online version of the book Therapeutic Choices5 published by the Canadian Pharmacists Association. On viewing the Highlights e-mail, participants had access to the chapter of e-Therapeutics+ from which the information was derived. Chapter access was granted for 2 months from the date the Highlights e-mail was sent. Highlights were linked to the IAM so that participants could document their reflective learning.

The e-T Highlights were chosen for this program by clinical editors at the Canadian Pharmacists Association and vetted by 1 CFPC member (co-author B.M.) before dissemination. Beginning January 20, 2010, all CFPC members with valid e-mail addresses received a weekly invitation to participate in rating that week’s Highlight. Members of the CFPC who accepted the invitation completed a short demographic survey after clicking on the Earn Mainpro Credits button adjacent to the first Highlight they read. This was followed by their first IAM questionnaire, which asked participants to indicate whether that particular Highlight had an effect on them or their practice (yes, no, possibly) and whether that Highlight was relevant for at least 1 of their patients. With each subsequent Highlight that was read, only the IAM questionnaire was presented. For each submitted IAM questionnaire, family physicians earned 0.1 Mainpro-M1 credits, following a process similar to that used for other e-learning programs.6

One Highlight was delivered weekly to 17 000 e-mail addresses (each address belonging to 1 CFPC member). To read the Highlight embedded within its e-Therapeutics+ chapter, participants had to open the e-mail and click on the hyperlinked Highlight title. Data collection ran from January 20 to December 31, 2010. Data from the demographic characteristics questionnaire and the Highlights ratings completed with the IAM were analyzed descriptively.

A total of 5346 CFPC members completed the demographic characteristics questionnaire and submitted at least 1 rating of a Highlight. The participation rate was 31% (5346 of 17 000). Ninety-one percent of participants reported they were in family or general practice, while 85% were in full-time or part-time practice. Slightly more than half of participants were women (53%), and 56% of participants were CFPC Certificants.

There were 31 429 ratings of 49 Highlights in 2010, for an average of 667 ratings per Highlight (range 415 to 1176). The median number of Highlight ratings per participant was 2 (range 1 to 49). Among CFPC members who submitted more than 2 ratings, the median number of ratings per participant was 7 (mean 11.9).

Participants believed the clinical information from e-T Highlights was totally or partially relevant for at least 1 patient 90% of the time. To understand which Highlights were most or least relevant according to CFPC members, we calculated a relevance statistic that ranged from 0 to 1, called the clinical relevance of information index. The clinical relevance of information index value ranged from 0.458 to 0.899 per Highlight (mean 0.713). The following 2 Highlights had the lowest and the highest relevance rating, respectively:

  • Tizanidine is a good first-line treatment, or it can be combined with baclofen, as they have different sites of action.

  • Penicillin is the drug of choice for streptococcal sore throat. Although cephalosporins are effective, they should not replace penicillin as the drug of choice.

With regard to knowledge translation, for 59% of the rated Highlights, participants said the information would be used for at least 1 patient. For 41% of the rated Highlights, participants expected patient health benefits owing to Highlight recommendations, such as avoiding an unnecessary or inappropriate treatment, or a preventive intervention (Figure 1).

Figure 1.
  • Download figure
  • Open in new tab
Figure 1.

Value of the information from e-Therapeutics Highlights

Discussion

The e-T Highlights, which were written and reviewed by expert physicians and pharmacists, were rated by more than 5000 CFPC members in 2010. In addition to ratings of relevance and cognitive impact, participants often reported that therapeutic recommendations from Highlights would be used or applied in practice. While this led some participants to expect health benefits for their patients, our review of the literature shows studies have not yet demonstrated that these subjective physician experiences translate to action in practice.7

A strength of this program was the rating of Highlights using a validated method to document reflective learning. In addition, Highlights were delivered more slowly compared with other programs, at the rate of 1 Highlight per week. While we do not know the optimal speed for e-mail delivery of brief educational content, spaced online education has been shown to improve knowledge retention in medical students.8 Concurrent to this Highlights program, nearly 3000 family physicians used the IAM to rate Patient-Oriented Evidence that Matters (POEMs) synopses in 2010. Given the differences between family physicians who rated POEMs and those who rated Highlights (Table 1), a comparison of ratings of these 2 types of clinical information must be interpreted with caution. Furthermore, POEMs are different from Highlights, as the former are synopses of recently published original research articles with a focus not limited to therapeutics. However, we provide Figure 2 to simply illustrate the results that might be obtained, for example, in randomized trials of e-mail alerting services for a variety of health professionals. In Figure 2, the clinical relevance of POEMs is distributed over a wider range than the clinical relevance of Highlights is. In addition, the clinical relevance of POEMs delivered in 2010 was generally lower than that of Highlights. This finding is not surprising considering that Highlights are therapeutic recommendations for practice, authored by and for Canadian physicians.

Figure 2.
  • Download figure
  • Open in new tab
Figure 2.

Distribution of relevance ratings for 2 types of clinical information delivered to FPs and GPs: Each point represents a single e-mail alert rated in 2010.

POEMs—Patient-Oriented Evidence that Matters.

View this table:
  • View inline
  • View popup
Table 1.

Characteristics of the FPs and GPs who rated POEMs and e-Therapeutics Highlights in 2010

Beyond increasing the awareness of Highlights, this e-learning program improved access to therapeutic recommendations, as before the program, physicians could access recommendations from Therapeutic Choices only by obtaining a print copy. By having the IAM link within the e-T Highlights that were delivered via e-mail, participants had the opportunity to comment on what they had read. Their comments, such as suggestions for additional content, were relayed to the editor-in-chief and clinical editors. Consequently, the editors were able to bring further improvements to e-Therapeutics+.9,10

Limitations

Our data are self-reported. As such, the health benefits for specific patients were “expected” and not objectively documented. Whether patient health outcomes are influenced by the push of clinical information from one electronic knowledge resource is an open question for future research. In addition, we did not measure how frequently Highlights were read and not rated. If Highlights are more likely to be rated by family physicians when they are relevant for at least 1 patient, then our data overestimate the clinical relevance of each Highlight.

Continuing medical education programs do not randomly select participants. Therefore, physicians in this program were more motivated to receive e-mail alerts, as well as read and rate Highlights. Nevertheless, knowledge that the Highlights program might lead to health benefits for specific patients is potentially important to improving the outcomes of clinical practice. Given these expected benefits for patients, the Highlights e-learning program has been sustained by the Canadian Pharmacists Association and the CFPC. In addition, a new program for Canadian pharmacists—mirroring the current program for family physicians—was launched by the Canadian Pharmacists Association in 2012. The continuation of this program opens the door to further research, as ratings are subjective physician experiences that might be objectively confirmed.

Conclusion

This CME program stimulated family physicians to rate therapeutic recommendations delivered to them weekly via e-mail. The process of rating Highlights raised awareness of treatment recommendations and documented self-reported use of this information in practice. For participants, the IAM helped to document their reflective learning for Mainpro. For information providers, the IAM can reveal the clinical relevance of 1 specific information object (eg, 1 Highlight), as well as the overall clinical relevance of CME programs.

Acknowledgments

This continuing medical education program was implemented in the context of a study funded by the Canadian Institutes of Health Research. Opinions are those of the authors and do not imply endorsement by the College of Family Physicians of Canada.

Notes

EDITOR’S KEY POINTS

  • In this program, family physicians evaluated therapeutic information from the online resource e-Therapeutics Highlights with the Information Assessment Method (IAM).

  • By rating the information from Highlights, participants earned continuing medical education credits for each completed IAM questionnaire. The IAM questionnaire asked participants whether the information from Highlights had an effect on them or their practices and whether it was relevant for at least 1 of their patients.

  • The strengths of this program include raising awareness of and improving access to therapeutic recommendations, as well as documenting self-reported use of this information in practice.

POINTS DE REPÈRE DU RÉDACTEUR

  • Dans ce programme, des médecins de famille ont utilisé la Méthode d’évaluation des informations (IAM) pour évaluer l’information thérapeutique contenue dans l’e-Therapeutics Highlight, une ressource sur le Web.

  • En évaluant l’information contenue dans les Highlights, les participants ont obtenu des crédits de formation médicale continue pour chaque questionnaire de l’IAM complété. Dans ces questionnaires, on demandait aux participants si l’information des Highlights les affectait ou si elle affectait leur pratique, et si elle était pertinente pour au moins un de leurs patients.

  • Parmi les principaux avantages de ce programme, mentionnons le fait de sensibiliser les participants aux recommandations thérapeutiques et de rendre ces informations plus accessibles, mais aussi de documenter les déclarations des participants sur l’usage qu’ils en font dans leur pratique.

Footnotes

  • This article has been peer reviewed.

  • Cet article a fait l’objet d’une révision par des pairs.

  • Contributors

    Dr Grad, Dr Pluye, Ms Repchinsky, Ms Jovaisas, Dr Marlow, Mr Shulha, and Dr de Gaspé Bonar contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission. Drs Ricarte and Barbosa Galvão contributed to the analysis and preparation of the manuscript.

  • Competing interests

    Drs Grad and Pluye receive unrestricted research grants from the Canadian Pharmacists Association and the Canadian Medical Association. These funds provide bursaries in support of graduate student research at McGill University. At the time of writing, Ms Repchinsky was Editor-in-Chief, Ms Jovaisas was Clinical Editor, and Dr de Gaspé Bonar is Senior Director of Digital Publishing Solutions, all at the Canadian Pharmacists Association.

  • Copyright© the College of Family Physicians of Canada

References

  1. 1.↵
    1. Pluye P,
    2. Grad RM,
    3. Granikov V,
    4. Jagosh J,
    5. Leung K
    . Evaluation of email alerts in practice: part 1. Review of the literature on clinical emailing channels. J Eval Clin Pract 2010;16(6):1227-35. Epub 2010 Aug 15.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Grimshaw JM,
    2. Thomas RE,
    3. MacLennan G,
    4. Fraser C,
    5. Ramsay CR,
    6. Vale L,
    7. et al
    . Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess 2004;8(6):iii-iv, 1-72.
    OpenUrlCrossRefPubMed
  3. 3.↵
    1. Pluye P,
    2. Grad RM,
    3. Johnson-Lafleur J,
    4. Bambrick T,
    5. Burnand B,
    6. Mercer J,
    7. et al
    . Evaluation of email alerts in practice: part 2. Validation of the information assessment method (IAM). J Eval Clin Pract 2010;16(6):1236-43. Epub 2010 Aug 15.
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Canadian Pharmacists Association
    . e-Therapeutics+. Ottawa, ON: Canadian Pharmacists Association; 2014. Available from: www.e-therapeutics.ca. Accessed 2014 Apr 1.
  5. 5.↵
    1. Canadian Pharmacists Association
    . Therapeutic choices. 6th ed. Ottawa, ON: Canadian Pharmacists Association; 2011.
  6. 6.↵
    1. Grad RM,
    2. Pluye P,
    3. Mercer J,
    4. Marlow B,
    5. Beauchamp ME,
    6. Shulha M,
    7. et al
    . Impact of research-based synopses delivered as daily e-mail: a prospective observational study. J Am Med Inform Assoc 2008;15(2):240-5. Epub 2007 Dec 20.
    OpenUrlCrossRefPubMed
  7. 7.↵
    1. Pluye P,
    2. Grad R,
    3. Granikov V,
    4. Theriault G,
    5. Frémont P,
    6. Burnand B,
    7. et al
    . Feasibility of a knowledge translation CME program: Courriels Cochrane. J Contin Educ Health Prof 2012;32(2):134-41.
    OpenUrlCrossRefPubMed
  8. 8.↵
    1. Kerfoot BP,
    2. DeWolf WC,
    3. Masser BA,
    4. Church PA,
    5. Federman DD
    . Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Med Educ 2007;41(1):23-31.
    OpenUrlCrossRefPubMed
  9. 9.↵
    1. Pluye P,
    2. Grad RM,
    3. Repchinsky C,
    4. Jovaisas B,
    5. Lewis D,
    6. Tang DL,
    7. et al
    . Better than “best” evidence? The information assessment method can help information providers to use feedback from family physicians; Poster presented at: Family Medicine Forum; 2011 Nov 2; Montreal, QC.
  10. 10.↵
    1. Pluye P,
    2. Grad R,
    3. Repchinsky C,
    4. Jovaisas B,
    5. Lewis D,
    6. Tang DL,
    7. et al
    . “Better-than-best” evidence? Using family physicians’ feedback for 2-way knowledge translation. Can Fam Physician 2014;60:415-7. (Eng), 425–7 (Fr).
    OpenUrlFREE Full Text
PreviousNext
Back to top

In this issue

Canadian Family Physician: 60 (5)
Canadian Family Physician
Vol. 60, Issue 5
1 May 2014
  • Table of Contents
  • About the Cover
  • Index by author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on The College of Family Physicians of Canada.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Physician assessments of the value of therapeutic information delivered via e-mail
(Your Name) has sent you a message from The College of Family Physicians of Canada
(Your Name) thought you would like to see the The College of Family Physicians of Canada web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Physician assessments of the value of therapeutic information delivered via e-mail
Roland Grad, Pierre Pluye, Carol Repchinsky, Barbara Jovaisas, Bernard Marlow, Ivan L. Marques Ricarte, Maria Cristiane Barbosa Galvão, Michael Shulha, James de Gaspé Bonar
Canadian Family Physician May 2014, 60 (5) e258-e262;

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Respond to this article
Share
Physician assessments of the value of therapeutic information delivered via e-mail
Roland Grad, Pierre Pluye, Carol Repchinsky, Barbara Jovaisas, Bernard Marlow, Ivan L. Marques Ricarte, Maria Cristiane Barbosa Galvão, Michael Shulha, James de Gaspé Bonar
Canadian Family Physician May 2014, 60 (5) e258-e262;
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Program description
    • Discussion
    • Limitations
    • Conclusion
    • Acknowledgments
    • Notes
    • Footnotes
    • References
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF

Related Articles

  • “Better-than-best” evidence?
  • Données probantes « meilleures que les meilleures »?
  • PubMed
  • Google Scholar

Cited By...

  • Correction
  • Donnees probantes << meilleures que les meilleures >>?: Transposition a 2 sens du savoir grace a la retroaction des medecins de famille
  • "Better-than-best" evidence?: Using family physicians' feedback for 2-way knowledge translation
  • Google Scholar

More in this TOC Section

  • The SAFE (Social Accountability as the Framework for Engagement) for Health Institutions project
  • Chronic noncancer pain management
  • Home-based primary care in Canada
Show more Program Description

Similar Articles

Navigate

  • Home
  • Current Issue
  • Archive
  • Collections - English
  • Collections - Française

For Authors

  • Authors and Reviewers
  • Submit a Manuscript
  • Permissions
  • Terms of Use

General Information

  • About CFP
  • About the CFPC
  • Advertisers
  • Careers & Locums
  • Editorial Advisory Board
  • Subscribers

Journal Services

  • Email Alerts
  • Twitter
  • LinkedIn
  • Instagram
  • RSS Feeds

Copyright © 2025 by The College of Family Physicians of Canada

Powered by HighWire