Accuracy of the DriveABLE cognitive assessment to determine cognitive fitness to drive

Allen R. Dobbs

Compose eLetter

Title *

Author Information

Contributors
First Name and Middle Initial * First or given name, e.g. 'Peter'. Last Name * Your last, or family, name, e.g. 'MacMoody'. Email Address * Your email address, e.g. higgs-boson@gmail.com Occupation * Your role and/or occupation, e.g. 'Orthopedic Surgeon'. Affiliation * Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.

Statement of Competing Interests

Competing interests? *

Yes

Please describe the competing interests

CAPTCHA

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

What code is in the image? *

Enter the characters shown in the image.

Vertical Tabs

Jump to comment:

In-office DriveABLE cognitive assessments fail to predict on-road results with high accuracy: A re-analysis
Michel Bedard

Published on: 30 May 2013
Study does NOT show that DriveABLE assessments are highly accurate
Robert Gifford

Published on: 12 May 2013

Published on: (30 May 2013)
Page navigation anchor for In-office DriveABLE cognitive assessments fail to predict on-road results with high accuracy: A re-analysis
In-office DriveABLE cognitive assessments fail to predict on-road results with high accuracy: A re-analysis
Michel Bedard, Director
Other Contributors:
We read with interest the article published by Dr. Dobbs (2013;53:e156-61) and disagree with his interpretation of the findings. Dr. Dobbs' conclusion that the DriveABLE in-office cognitive assessment is highly accurate in identifying drivers with suspected or confirmed cognitive impairment who would pass or fail the DriveABLE On-Road Evaluation (DORE) is based, incorrectly, on overall cell percentages. To quote him, "...
Show More

We read with interest the article published by Dr. Dobbs (2013;53:e156-61) and disagree with his interpretation of the findings. Dr. Dobbs' conclusion that the DriveABLE in-office cognitive assessment is highly accurate in identifying drivers with suspected or confirmed cognitive impairment who would pass or fail the DriveABLE On-Road Evaluation (DORE) is based, incorrectly, on overall cell percentages. To quote him, "For the total sample, only 1.7% of the patients who received an In-Office pass outcome received a mismatching DORE outcome of fail. The errors for the fail outcome were somewhat higher but still low (5.6%)." (p. e160)
Rather than presenting overall percentages, Dr. Dobbs should have reported the actual cell counts and the row percentages, as these are far more relevant (a revised version of Table 1 is available from the corresponding author). The row percentages show that 62 of the 504 individuals who passed the in-office assessment (12.3%) failed the on-road test; and 204 of the 1474 who failed the in-office assessment (13.8%) passed the on-road test. These numbers are very different than the 1.7% and 5.6% presented in the article.
Finally, the overall raw agreement between the in-office assessment and on-road tests is only 50.4%. The two approaches would agree by chance alone 33% of the time and a statistic to denote "chance-corrected" agreement should have been presented. We calculated such a statistic (Cohen's weighted Kappa with quadratic weights) and it is also far from impressive. The Kappa value for these data is .432 (95% CI: .406 to .459), which based on accepted guidelines represents only fair to moderate agreement, well below the required minimum value to support Dr. Dobbs' conclusions.(1,2) Landis suggests that Kappa values of 0.61-0.80 represent substantial agreement, while values of 0.81-1 designate almost perfect agreement.(1) Fleiss characterizes kappa values over 0.75 as excellent.(2)
Therefore, the correct interpretation of the data is that there is only fair to moderate agreement between the in-office and on-road outcomes. We do not agree with Dobbs' conclusion that these "findings provide the evidence physicians need to be confident in using the recommendations from the DriveABLE In-Office cognitive evaluation to assist them in making accurate, evidence-based decisions about their patients' fitness to drive." (p. e161) The author's interpretation and the Editor's Key Points need to be revised to accurately reflect the results.
1. Landis, J.R.; & Koch, G.G. (1977). "The measurement of observer agreement for categorical data". Biometrics 33 (1): 159-174.
2. Fleiss, J.L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley.
Conflict of Interest:
The authors are members of Candrive, a CIHR-funded research team to support safe driving in older adults. One of the team's major objectives is to develop a tool to aid clinicians in assessing older drivers' fitness-to-drive.
Show Less

Competing Interests: None declared.
Published on: (12 May 2013)
Page navigation anchor for Study does NOT show that DriveABLE assessments are highly accurate
Study does NOT show that DriveABLE assessments are highly accurate
Robert Gifford, Professor
In my opinion, the conflict of interest in this study appears to have coloured the interpretation of the results to an unacceptable degree, and the manuscript's conclusions should have been totally revised, or the manuscript rejected. Table 1 clearly shows that the in-office test had about 69% accuracy when it gave drivers a "pass," about 75% accurate when it gave a "fail" and about 24% accurate when it said a driver was...
Show More

In my opinion, the conflict of interest in this study appears to have coloured the interpretation of the results to an unacceptable degree, and the manuscript's conclusions should have been totally revised, or the manuscript rejected. Table 1 clearly shows that the in-office test had about 69% accuracy when it gave drivers a "pass," about 75% accurate when it gave a "fail" and about 24% accurate when it said a driver was "indeterminate." Using the diagonal percents as the measure of accuracy across all cases, the in-office test matched the on-road test in 50% of all cases. A 50% accuracy rate is far from the tenor of the conclusion the author tries to depict ("highly accurate"), and far from a standard that one would consider to be an overall "good" test. In sum, the "savings" provided by the in-office test amount to wasted funds if half of its conclusions about driver ability are wrong. "Just test drivers on the road" should be the conclusion, in my opinion. It is good that the conflict of interest was rep orted, but in this case the conflict appears to have colored the conclusions so much that this article's conclusions are severely flawed and should not have been published as is. This shows that merely reporting a conflict of interest is not enough; a manuscript's interpretations and conclusions need closer scrutiny when there is a conflict. One wonders what the peer reviewers were thinking.
Conflict of Interest:
None declared
Show Less

Competing Interests: None declared.