Skip to main content

Main menu

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums

User menu

  • My alerts

Search

  • Advanced search
The College of Family Physicians of Canada
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums
  • My alerts
The College of Family Physicians of Canada

Advanced Search

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • RSS feeds
  • Follow cfp Template on Twitter
OtherCollege

Impact of family medicine research

Bibliometrics and beyond

Lynn G. Dunikowski and Thomas R. Freeman
Canadian Family Physician March 2016, 62 (3) 266-268;
Lynn G. Dunikowski
Was Director of Library Services for the College of Family Physicians of Canada in Mississauga, Ont.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Thomas R. Freeman
Professor in the Department of Family Medicine of the Schulich School of Medicine and Dentistry at Western University in London, Ont, and former Chair of the department.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF
Loading

A discipline is defined by an active area of research and a codified body of knowledge1,2 that provides legitimacy for the work it does and leads to innovation. Academic family medicine has existed for almost 50 years. Family medicine research has made great strides,3 with a steady increase in research articles published since the 1950s. What has been the impact of all this activity?

Meaning of impact

Research impact has been defined as “the value and benefit associated with using the knowledge produced through research, and being involved in conducting research.”4 Its meaning and measurement differ among groups and individuals. The meaning of impact for funders and policy makers might be conceived of as influence on policy, health service delivery, and population health outcomes. Academic institutions are more interested in how a discipline is represented in the academic milieu. For researchers, impact might mean the influence of their work on other researchers and practitioners in the field. For clinicians, the focus will be on the effect on daily practice.

Measuring impact

Funders and policy makers use methods such as benchmarking, case studies, peer review, economic rate of return, logic modeling, and bibliometrics.5 The recent allocation of $3 billion in annual research funding in the United Kingdom, using graded case study “stories,” illustrates a trend toward using more qualitative measures of impact as well as hard data.6 Academic faculties traditionally rely heavily on bibliometrics to allocate funds, make promotion and tenure decisions, and benchmark their research. Full-time and clinician researchers in family medicine are largely held to the measures used by their institutions. They want to demonstrate the worth of their research to funders and policy makers, their employers, and their colleagues. Full-time practising family physicians will perceive research to have impact to the extent of its practical application. They might be sceptical about the value of research received from experts7 and might not be very interested in academic measures of impact.

The Canadian Academy of Health Sciences has proposed an impact framework and a preferred menu of nearly 70 indicators and metrics of impact that can be used for evaluating the return on investment in health research.8 The Canadian Academy of Health Sciences reports and others emphasize that there is no one simple set of metrics for all.5,8 This article focuses on a subset of measures that deal with the published body of knowledge, commonly called bibliometrics, and a set of newer “alternative” publication metrics.

Bibliometrics

Examples of simple measures are number of articles published in peer-reviewed journals or number of times an article has been cited. More sophisticated measures include the h index,9 which quantifies the impact of a scholar who has published h papers, each of which has been cited in other papers at least h times. Measures of journal influence such as the journal impact factor (JIF)10 and the SCImago Journal Rank indicator11 are calculated from data in bibliographic databases. The JIF is calculated by dividing the number of citations to a given journal in Thomson Reuters’ Journal Citation Reports by the total number of articles published in the 2 previous years. A JIF of 1.0 means that, on average, the articles published 1 or 2 years ago have been cited 1 time. The JIF is popularly recognized if not well understood, and the inappropriate use of a journal-level metric to measure the impact of an individual researcher or article has been discussed.10,12

Citations to 2 articles from The Seven Wonders of Family Medicine Research,13 a list of 7 influential family medicine research articles, demonstrate that bibliometrics are not absolutes—they depend on the source from which they are derived. Searches conducted on the same day found that articles from the list had quite different numbers of citations in Web of Science, Scopus, and Google Scholar (Table 1).14,15 Note that the 2 measures of impact are not well correlated—these 2 articles, both judged to have high impact by peers, received widely different numbers of citations.

View this table:
  • View inline
  • View popup
Table 1

Number of times articles were cited by source consulted

Alternative metrics

Recently a wave of alternative metrics or altmetrics has appeared.16 These include statistics such as the number of times an article has been downloaded, viewed, or shared on social media platforms such as Twitter. These altmetrics give a more immediate view of impact than, for example, the number of times an article is cited, a measure that might take several years to develop. Altmetrics use reflects the reality of researchers who are increasingly working in an online environment and publishing their research in nontraditional venues and formats—like YouTube and SlideShare, blogs, and institutional repositories. Currently, online users can see altmetrics displayed beside articles in databases such as Scopus and in online journals such as BMJ and JAMA. Alternative metrics offer new ways to measure the social impact of research, the importance of which is increasingly acknowledged.12,17 However, alternative metrics are not without problems, such as the ease with which they can be manipulated or gamed.16

Challenges

Why is it difficult to determine the impact of family medicine research with publication metrics? One reason is that family medicine research articles are scattered among family medicine journals—and even more widely scattered among non–family medicine journals. For example, a Scopus search conducted in 2012 found that the 250 most highly cited family medicine research articles were published in 71 different journals, with 47 journals publishing 1 article each (Figure 1). Only 5 of the 71 journals could be described as “family medicine” journals.18 Only 3 articles in the list of 7 influential family medicine papers previously mentioned were published in family medicine journals.13 This scattering of research articles makes finding relevant articles more problematic for researchers and authors, and impact measures might be affected. The scatter of articles is a problem for most family physicians, who do not have time to scan multiple journals for items of interest.

Figure 1
  • Download figure
  • Open in new tab
Figure 1

Journal contributions to a list of the 250 most-cited family medicine articles

Data from a Scopus search in April 2012.

Establishing the influence of publications on the behaviour of individual clinicians is a complex and challenging task.17 The time lag between research findings and when benefits occur is a challenge. The longer the time, the more difficult it is to track impact and attribute it to the project. One reason is that researchers do not necessarily record all their dissemination activities or maintain contact with users of their research. Another reason is that impact can occur at any of the different stages of the research cycle. One research team that undertook this tracking challenge reported extensive outputs beyond the expected peer-reviewed publications, including impact on processes and policies, producing new knowledge, and building capacity. In their study, the number of peer-reviewed publications was not always a good indicator of impact—one of the projects with the highest impact had no peer-reviewed publications.19

How does one measure the impact of seminal thinkers such as Dr Ian McWhinney? Certainly his articulation of the principles of family medicine has contributed to the conceptual basis of the discipline, but when this is actuated in a new clinical method then it has a direct impact on practice. The inadequacies of simple bibliometrics and altmetrics illustrate the adage that not everything that counts can be counted.

Notes

Hypothesis is a quarterly series in Canadian Family Physician, coordinated by the Section of Researchers of the College of Family Physicians of Canada. The goal is to explore clinically relevant research concepts for all CFP readers. Submissions are invited from researchers and nonresearchers. Ideas or submissions can be submitted online at http://mc.manuscriptcentral.com/cfp or through the CFP website www.cfp.ca under “Authors and Reviewers.”

Footnotes

  • Competing interests

    None declared

  • Copyright© the College of Family Physicians of Canada

References

  1. 1.↵
    1. McWhinney IR
    . General practice as an academic discipline: reflections after a visit to the United States. Lancet 1966;1(7434):419-23.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Beaulieu MD,
    2. Rioux M,
    3. Rocher G,
    4. Samson L,
    5. Boucher L
    . Family practice: professional identity in transition. A case study of family medicine in Canada. Soc Sci Med 2008;67(7):1153-63.
    OpenUrlCrossRefPubMed
  3. 3.↵
    1. Pimlott N,
    2. Upshur RE
    . From clinical observation to clinical discovery. The challenge for family medicine research. Can Fam Physician 2014;60:14-6. (Eng), 27–9 (Fr).
    OpenUrlFREE Full Text
  4. 4.↵
    1. Beacham B,
    2. Kalucy L,
    3. McIntyre E
    . Understanding & measuring research impact. FOCUS on 2005;(2):1-11. Available from: www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3236.pdf. Accessed 2016 Feb 2.
  5. 5.↵
    1. Canadian Health Services Research Foundation
    . Measuring the impact of research: what do we know? (Part I). Insight Action 2008;46:1-4. Available from: www.cfhi-fcass.ca/SearchResultsNews/08-10-01/eb43c553-38c6-48c9-980b-ef3061fa1987.aspx. Accessed 2016 Feb 2.
    OpenUrl
  6. 6.↵
    Telling stories. Nature 2015;518(7538):137.
    OpenUrl
  7. 7.↵
    1. Allan GM,
    2. Kraut R,
    3. Crawshay A,
    4. Korownyk C,
    5. Vandermeer B,
    6. Kolber MR
    . Contributors to primary care guidelines. What are their professions and how many of them have conflicts of interest? Can Fam Physician 2015;61:52-8.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    Making an impact: a preferred framework and indicators to measure returns on investment in health research. Ottawa ON: Canadian Academy of Health Sciences; 2009. Panel on Return on Investment in Health Research. Available from: http://cahs-acss.ca/making-an-impact. Accessed 2016 Feb 2.
  9. 9.↵
    1. Hirsch JE
    . An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A 2005;102(46):16569-72.
    OpenUrlAbstract/FREE Full Text
  10. 10.↵
    1. McVeigh ME,
    2. Mann SJ
    . The journal impact factor denominator: defining citable (counted) items. JAMA 2009;302(10):1107-9.
    OpenUrlCrossRefPubMed
  11. 11.↵
    SJR SCImago Journal & Country Rank [website]. SCImago; 2007. Available from: www.scimagojr.com. Accessed 2016 Feb 2.
  12. 12.↵
    1. Van Driel ML,
    2. Maier M,
    3. De Maeseneer J
    . Measuring the impact of family medicine research: scientific citations or societal impact? Fam Pract 2007;24(5):401-2.
    OpenUrlFREE Full Text
  13. 13.↵
    1. College of Family Physicians of Canada
    . The seven wonders of family medicine research. Mississauga ON: College of Family Physicians of Canada; 2014. Available from: www.cfpc.ca/uploadedFiles/Directories/Sections/7%20Wonders%20of%20FM%20Research%20-%20WEB%20EN.pdf. Accessed 2016 Feb 2.
  14. 14.↵
    1. Stewart M,
    2. Brown JB,
    3. Donner A,
    4. McWhinney IR,
    5. Oates J,
    6. Weston WW,
    7. et al
    . The impact of patient-centered care on outcomes. J Fam Pract 2000;49(9):796-804.
    OpenUrlPubMed
  15. 15.↵
    1. Klein MC,
    2. Gauthier RJ,
    3. Jorgensen SH,
    4. Robbins JM,
    5. Kaczorowski J,
    6. Johnson B,
    7. et al
    . Does episiotomy prevent perineal trauma and pelvic floor relaxation? First North American trial of episiotomy. Online J Curr Clin Trials 1992 Jul 1;1. (Doc no. 10).
  16. 16.↵
    1. Bornmann L
    . Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. J Informetrics 2014;8(4):895-903.
    OpenUrl
  17. 17.↵
    Quantifying the social impact of research and medical journals. Lancet 2014;384(9943):557.
    OpenUrlPubMed
  18. 18.↵
    1. Freeman T,
    2. Dunikowski LG
    . Academic family medicine: a retrospective look at the body of knowledge. Poster presented at: Family Medicine Forum; 2012 Nov 15–17; Toronto, ON.
  19. 19.↵
    1. Kalucy L,
    2. McIntyre E,
    3. Jackson Bowers E
    . Primary health care research impact project. Final report stage 1. Adelaide, Aust: Primary Health Care Research & Information Service; 2007. Available from: www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3338.pdf. Accessed 2016 Feb 2.
PreviousNext
Back to top

In this issue

Canadian Family Physician: 62 (3)
Canadian Family Physician
Vol. 62, Issue 3
1 Mar 2016
  • Table of Contents
  • About the Cover
  • Index by author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on The College of Family Physicians of Canada.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Impact of family medicine research
(Your Name) has sent you a message from The College of Family Physicians of Canada
(Your Name) thought you would like to see the The College of Family Physicians of Canada web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Impact of family medicine research
Lynn G. Dunikowski, Thomas R. Freeman
Canadian Family Physician Mar 2016, 62 (3) 266-268;

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Respond to this article
Share
Impact of family medicine research
Lynn G. Dunikowski, Thomas R. Freeman
Canadian Family Physician Mar 2016, 62 (3) 266-268;
Reddit logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Meaning of impact
    • Measuring impact
    • Bibliometrics
    • Alternative metrics
    • Challenges
    • Notes
    • Footnotes
    • References
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • Bibliometric Analysis of Manuscript Characteristics that Influence Citations: A Comparison of Six Major Family Medicine Journals
  • Population vulnerability to COVID-19 in Europe: a burden of disease analysis
  • Google Scholar

More in this TOC Section

College

  • Collective effort: our key to success
  • L’effort collectif : la clé de notre réussite
  • La collégialité : le remède à la crise actuelle
Show more College

Hypothesis

  • Utility of hashing and salting algorithms in quality improvement studies
  • Recognition of inherent biases in administrative data
  • The Canadian Primary Care Information Network
Show more Hypothesis

Section of Researchers

  • Utility of hashing and salting algorithms in quality improvement studies
  • Recognition of inherent biases in administrative data
  • Impact of patient partner co-design on survey development in primary care research
Show more Section of Researchers

Similar Articles

Navigate

  • Home
  • Current Issue
  • Archive
  • Collections - English
  • Collections - Française

For Authors

  • Authors and Reviewers
  • Submit a Manuscript
  • Permissions
  • Terms of Use

General Information

  • About CFP
  • About the CFPC
  • Advertisers
  • Careers & Locums
  • Editorial Advisory Board
  • Subscribers

Journal Services

  • Email Alerts
  • Twitter
  • RSS Feeds

Copyright © 2023 by The College of Family Physicians of Canada

Powered by HighWire