Skip to main content

Main menu

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums

User menu

  • My alerts

Search

  • Advanced search
The College of Family Physicians of Canada
  • Other Publications
    • http://www.cfpc.ca/Canadianfamilyphysician/
    • https://www.cfpc.ca/Login/
    • Careers and Locums
  • My alerts
The College of Family Physicians of Canada

Advanced Search

  • Home
  • Articles
    • Current
    • Published Ahead of Print
    • Archive
    • Supplemental Issues
    • Collections - French
    • Collections - English
  • Info for
    • Authors & Reviewers
    • Submit a Manuscript
    • Advertisers
    • Careers & Locums
    • Subscribers
    • Permissions
  • About CFP
    • About CFP
    • About the CFPC
    • Editorial Advisory Board
    • Terms of Use
    • Contact Us
  • Feedback
    • Feedback
    • Rapid Responses
    • Most Read
    • Most Cited
    • Email Alerts
  • Blogs
    • Latest Blogs
    • Blog Guidelines
    • Directives pour les blogues
  • Mainpro+ Credits
    • About Mainpro+
    • Member Login
    • Instructions
  • RSS feeds
  • Follow cfp Template on Twitter
  • LinkedIn
  • Instagram
Research ArticleCollege

Patient self-report and medical records

Measuring agreement for binary data

Angela M. Barbara, Mark Loeb, Lisa Dolovich, Kevin Brazil and Margaret L. Russell
Canadian Family Physician June 2011; 57 (6) 737-738;
Angela M. Barbara
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: barbara@mcmaster.ca
Mark Loeb
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Lisa Dolovich
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kevin Brazil
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Margaret L. Russell
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF
Loading

Research data in family medicine often comes from 2 sources: self-report and medical record review. Frequently, the quality of these data sources is assumed to be high, but measuring the reproducibility of these data is essential to evaluating the quality of the information collected. In ideal circumstances, data obtained from either data source would be equivalent. However, no source of data is without error. In cases of low agreement between data sources, research findings differ depending on the method of data collection used1 and leave the researcher with questions about which estimate is correct. Comparing data from different sources can give family medicine researchers insight into which data source is most appropriate to answer a specific research question or can direct efforts to improve the collection and recording of health data.2

Imagine that we are interested in the prevalence of fever or cough in outpatients over the past influenza season. Neither the medical record nor patient self-report is considered the true criterion standard for symptoms. We are not assessing the accuracy of one data source compared with another; rather, we are examining agreement between the sources of data. The presence or absence of patient symptoms is considered a binary variable—a categorical variable in which there are 2 possible conditions (eg, yes or no, positive or negative). This paper describes indicators for determining agreement between binary variables: total agreement, κ, and positive and negative agreement.

Interpreting the value of κ

Table 1 displays data from the Hutterite Influenza Prevention Study in 2 × 2 contingency tables.3 Symptoms reported by Hutterite community members were compared with documentation in the medical records. Total agreement is the number of concordant pairs divided by the total sample. In Table 1A, total agreement is 74%, which is the number of concordant yes’s for fever (18) plus the concordant no’s (112) divided by 176 participants. However, this simple measure does not take into account that a certain amount of agreement between medical charts and self-report is expected by chance alone4; assessment of κ, on the other hand, measures the strength of agreement beyond what we expect solely by chance. The calculation for κ is as follows: κ=total agreement−chance agreement1−chance agreemen

View this table:
  • View inline
  • View popup
Table 1

Contingency tables of data from the Hutterite Influenza Prevention Study3: A) fever, B) earache, C) cough, and D) chills.

The answer falls on a scale of −1 to 1, where 0 equals chance agreement and 1 equals perfect agreement. In 1977, Landis and Koch proposed the following guidelines for understanding κ values: less than 0 equals no agreement, 0.01 to 0.20 equals slight agreement, 0.21 to 0.40 equals fair agreement, 0.41 to 0.60 equals moderate agreement, 0.61 to 0.80 equals substantial agreement, and 0.81 to 1.0 equals almost perfect agreement.5 While these guidelines are widely used and cited, the cutoffs are not universally accepted and have been criticized for being arbitrary divisions based on personal opinion rather than evidence.6,7

The value of κ is not simple to interpret because it is influenced by the prevalence of the variable being measured.8 Table 1A and 1C have similar total agreements (as do 1B and 1D), but κ values differ according to distributions. The κ value represents the proportion of total variance that is not attributable to chance or random error. Because total variance is minimal in a uniform (homogeneous) population where there is a relatively high (or low) prevalence, κ will be low even though total agreement might be high (Table 1D). Because chance agreement is smallest in a mixed (heterogeneous) population, κ will be higher when prevalence is closer to 50% (Table 1B and 1C). This makes it difficult to compare κ values between patient symptoms or other variables and different prevalences.9

Calculation of κ is also influenced by bias or the disagreement in the proportion of positive or negative cases (number of discordant responses)6; that is, the mismatch of positive or negative cases or disagreements are not random but go in one direction rather than another,8,10 which tends to happen when the prevalence of a symptom is high or low. This might result in a low κ value even though agreement is substantial (Table 1A and 1D); the value of κ is higher when there is a large bias and lowest when bias is absent.11

The κ value does not distinguish between various types and sources of agreement and disagreement.6,8,12,13 The aim of measuring agreement is to discover the bases of differences and reduce them if possible, rather than, for example, simply quantifying the degree of disagreement.9 In fact, it might be that no single agreement statistic can adequately capture agreement.11

Calculating positive and negative agreement

To help interpret κ values, calculating both positive and negative agreement has been recommended.11,14 The formula for calculating positive agreement is as follows: 2×concordant positives(positive pair+positive pair) Negative agreement is calculated as follows: 2×concordant negatives(negative pair+negative pair) Using these indices also provides insight into the agreement and imbalance in the proportion of positive or negative responses. This information is useful in determining where the focus should be to improve data quality depending on what is most important, which would be missed by calculating solely the κ value and total agreement.2,11,14 Low positive agreement indicates there is poor concordance between both sources in reporting the presence of the symptom (Table 1D), whereas high negative agreement means there is good concordance between both sources in identifying that the symptom was not experienced14 (Table 1A, 1B, 1D).

Family medicine practitioners should consider these concepts when evaluating various aspects of clinical care, such as data collection for a new practice quality assurance process. Although total agreement and the value of κ are commonly reported in agreement studies, we recommend the additional calculation of positive agreement and negative agreement.

Notes

Hypothesis is a quarterly series in Canadian Family Physician, coordinated by the Section of Researchers of the College of Family Physicians of Canada. The goal is to explore clinically relevant research concepts for all CFP readers. Submissions are invited from researchers and nonresearchers. Ideas or submissions can be submitted online at http://mc.manuscriptcentral.com/cfp or through the CFP website www.cfp.ca under “Authors.”

Footnotes

  • Competing interests

    None declared

  • Copyright© the College of Family Physicians of Canada

References

  1. ↵
    1. Zhu K,
    2. McKnight B,
    3. Stergachis A,
    4. Daling JR,
    5. Levine RS
    . Comparison of self-report data and medical records data: results from a case-control study on prostate cancer. Int J Epidemiol 1999;28(3):409-17.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Westbrook JI,
    2. McIntosh JH,
    3. Rushworth RL,
    4. Berry G,
    5. Duggan JM
    . Agreement between medical record data and patients’ accounts of their medical history and treatment for dyspepsia. J Clin Epidemiol 1998;51(3):237-44.
    OpenUrlCrossRefPubMed
  3. ↵
    1. Loeb M,
    2. Russell ML,
    3. Moss L,
    4. Fonseca K,
    5. Fox J,
    6. Earn DJ,
    7. et al
    . Effect of influenza vaccination of children on infection rates in Hutterite communities: a randomized trial. JAMA 2010;303(10):943-50.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Banerjee M,
    2. Capozzoli M,
    3. McSweeney L,
    4. Sinha D
    . Beyond kappa: a review of interrater agreement measures. Can J Stat 1999;27(1):3-23.
    OpenUrlCrossRef
  5. ↵
    1. Landis JR,
    2. Koch GG
    . The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159-74.
    OpenUrlCrossRefPubMed
  6. ↵
    1. Sim J,
    2. Wright CC
    . The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005;85(3):257-68.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Brennan P,
    2. Silman A
    . Statistical methods for assessing observer variability in clinical measures. BMJ 1992;304(6840):1491-4.
    OpenUrlFREE Full Text
  8. ↵
    1. Feinstein AR,
    2. Cicchetti DV
    . High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43(6):543-9.
    OpenUrlCrossRefPubMed
  9. ↵
    1. Sargeant JM,
    2. Martin SW
    . The dependence of kappa on attribute prevalence when assessing the repeatability of questionnaire data. Prev Vet Med 1998;34(2–3):115-23.
    OpenUrlCrossRefPubMed
  10. ↵
    1. Byrt T,
    2. Bishop J,
    3. Carlin JB
    . Bias, prevalence and kappa. J Clin Epidemiol 1993;46(5):423-9.
    OpenUrlCrossRefPubMed
  11. ↵
    1. Chen G,
    2. Faris P,
    3. Hemmelgarn B,
    4. Walker RL,
    5. Quan H
    . Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol 2009;9:5.
    OpenUrlCrossRefPubMed
  12. ↵
    1. Thompson WD,
    2. Walter SD
    . A reappraisal of the kappa coefficient. J Clin Epidemiol 1988;41(10):949-58.
    OpenUrlCrossRefPubMed
  13. ↵
    1. Lantz CA,
    2. Nebenzahl E
    . Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J Clin Epidemiol 1996;49(4):431-4.
    OpenUrlCrossRefPubMed
  14. ↵
    1. Cicchetti DV,
    2. Feinstein AR
    . High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990;43(6):551-8.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top

In this issue

Canadian Family Physician: 57 (6)
Canadian Family Physician
Vol. 57, Issue 6
1 Jun 2011
  • Table of Contents
  • About the Cover
  • Index by author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on The College of Family Physicians of Canada.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Patient self-report and medical records
(Your Name) has sent you a message from The College of Family Physicians of Canada
(Your Name) thought you would like to see the The College of Family Physicians of Canada web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Patient self-report and medical records
Angela M. Barbara, Mark Loeb, Lisa Dolovich, Kevin Brazil, Margaret L. Russell
Canadian Family Physician Jun 2011, 57 (6) 737-738;

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Respond to this article
Share
Patient self-report and medical records
Angela M. Barbara, Mark Loeb, Lisa Dolovich, Kevin Brazil, Margaret L. Russell
Canadian Family Physician Jun 2011, 57 (6) 737-738;
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Interpreting the value of κ
    • Calculating positive and negative agreement
    • Notes
    • Footnotes
    • References
  • Figures & Data
  • eLetters
  • Info & Metrics
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

College

  • Servant leadership
  • Parlons d’argent : la situation financière du CMFC
  • Le leadership au service d’autrui
Show more College

Hypothesis

  • State of traditional healing from the perspective of cultural support
  • Strength in family medicine research
  • Utility of hashing and salting algorithms in quality improvement studies
Show more Hypothesis

Similar Articles

Navigate

  • Home
  • Current Issue
  • Archive
  • Collections - English
  • Collections - Française

For Authors

  • Authors and Reviewers
  • Submit a Manuscript
  • Permissions
  • Terms of Use

General Information

  • About CFP
  • About the CFPC
  • Advertisers
  • Careers & Locums
  • Editorial Advisory Board
  • Subscribers

Journal Services

  • Email Alerts
  • Twitter
  • LinkedIn
  • Instagram
  • RSS Feeds

Copyright © 2025 by The College of Family Physicians of Canada

Powered by HighWire