Failure to predict on-road results

We read with interest the article “Accuracy of the DriveABLE cognitive assessment to determine cognitive fitness to drive” by Dr Dobbs,¹ which appeared in the March issue of Canadian Family Physician, and disagree with his interpretation of the findings. Dr Dobbs’ conclusion that the DriveABLE In-Office cognitive assessment is highly accurate in identifying drivers with suspected or confirmed cognitive impairment who would pass or fail the DriveABLE On-Road Evaluation is based, incorrectly, on overall cell percentages. In his article, he stated the following: “For the total sample, only 1.7% of the patients who received an In-Office pass outcome received a mismatching DORE [DriveABLE On-Road Evaluation] outcome of fail. The errors for the fail outcome were somewhat higher but still low (5.6%).”¹

Rather than presenting overall percentages, Dr Dobbs should have reported the actual cell counts and the row percentages, as these are far more relevant (a revised version of Table 1 is available from the corresponding author). The row percentages show that 62 of the 504 individuals who passed the In-Office assessment (12.3%) failed the On-Road test; and 204 of the 1474 who failed the In-Office assessment (13.8%) passed the On-Road test. These numbers are very different than the 1.7% and 5.6% presented in the article.

Finally, the overall raw agreement between the In-Office assessment and the On-Road test is only 50.4%. The 2 approaches would agree by chance alone 33% of the time, and a statistic to denote “chance-corrected” agreement should have been presented. We calculated such a statistic (Cohen weighted κ with quadratic weights) and it was also far from impressive. The κ value for these data is 0.432 (95% CI 0.406 to 0.459), which, based on accepted guidelines, represents only fair-to-moderate agreement—well below the required minimum value to support Dr Dobbs’ conclusions.^2,3 Landis and Koch suggested that κ values of 0.61 to 0.80 represent substantial agreement, while values of 0.81 to 1 designate almost perfect agreement.² Fleiss characterized κ values greater than 0.75 as excellent.³

Therefore, the correct interpretation of the data is that there is only fair-to-moderate agreement between the In-Office and On-Road outcomes. We do not agree with Dobbs’ conclusion that these “findings provide the evidence physicians need to be confident in using the recommendations from the DriveABLE In-Office cognitive evaluation to assist them in making accurate, evidence-based decisions about their patients’ fitness to drive.”¹ Dobbs’ interpretation and the editor’s key points need to be revised to accurately reflect the results.

• Copyright© the College of Family Physicians of Canada