Patient self-report and medical records

Interpreting the value of κ

Table 1 displays data from the Hutterite Influenza Prevention Study in 2 × 2 contingency tables.³ Symptoms reported by Hutterite community members were compared with documentation in the medical records. Total agreement is the number of concordant pairs divided by the total sample. In Table 1A, total agreement is 74%, which is the number of concordant yes’s for fever (18) plus the concordant no’s (112) divided by 176 participants. However, this simple measure does not take into account that a certain amount of agreement between medical charts and self-report is expected by chance alone⁴; assessment of κ, on the other hand, measures the strength of agreement beyond what we expect solely by chance. The calculation for κ is as follows: κ=total agreement−chance agreement1−chance agreemen

The answer falls on a scale of −1 to 1, where 0 equals chance agreement and 1 equals perfect agreement. In 1977, Landis and Koch proposed the following guidelines for understanding κ values: less than 0 equals no agreement, 0.01 to 0.20 equals slight agreement, 0.21 to 0.40 equals fair agreement, 0.41 to 0.60 equals moderate agreement, 0.61 to 0.80 equals substantial agreement, and 0.81 to 1.0 equals almost perfect agreement.⁵ While these guidelines are widely used and cited, the cutoffs are not universally accepted and have been criticized for being arbitrary divisions based on personal opinion rather than evidence.^6,7

The value of κ is not simple to interpret because it is influenced by the prevalence of the variable being measured.⁸ Table 1A and 1C have similar total agreements (as do 1B and 1D), but κ values differ according to distributions. The κ value represents the proportion of total variance that is not attributable to chance or random error. Because total variance is minimal in a uniform (homogeneous) population where there is a relatively high (or low) prevalence, κ will be low even though total agreement might be high (Table 1D). Because chance agreement is smallest in a mixed (heterogeneous) population, κ will be higher when prevalence is closer to 50% (Table 1B and 1C). This makes it difficult to compare κ values between patient symptoms or other variables and different prevalences.⁹

Calculation of κ is also influenced by bias or the disagreement in the proportion of positive or negative cases (number of discordant responses)⁶; that is, the mismatch of positive or negative cases or disagreements are not random but go in one direction rather than another,^8,10 which tends to happen when the prevalence of a symptom is high or low. This might result in a low κ value even though agreement is substantial (Table 1A and 1D); the value of κ is higher when there is a large bias and lowest when bias is absent.¹¹

The κ value does not distinguish between various types and sources of agreement and disagreement.^6,8,12,13 The aim of measuring agreement is to discover the bases of differences and reduce them if possible, rather than, for example, simply quantifying the degree of disagreement.⁹ In fact, it might be that no single agreement statistic can adequately capture agreement.¹¹

Calculating positive and negative agreement

To help interpret κ values, calculating both positive and negative agreement has been recommended.^11,14 The formula for calculating positive agreement is as follows: 2×concordant positives(positive pair+positive pair) Negative agreement is calculated as follows: 2×concordant negatives(negative pair+negative pair) Using these indices also provides insight into the agreement and imbalance in the proportion of positive or negative responses. This information is useful in determining where the focus should be to improve data quality depending on what is most important, which would be missed by calculating solely the κ value and total agreement.^2,11,14 Low positive agreement indicates there is poor concordance between both sources in reporting the presence of the symptom (Table 1D), whereas high negative agreement means there is good concordance between both sources in identifying that the symptom was not experienced¹⁴ (Table 1A, 1B, 1D).

Family medicine practitioners should consider these concepts when evaluating various aspects of clinical care, such as data collection for a new practice quality assurance process. Although total agreement and the value of κ are commonly reported in agreement studies, we recommend the additional calculation of positive agreement and negative agreement.