Archive | March, 2014

Another example of how little we understand numbers

17 Mar


Back in October 2013 I wrote about Kahneman’s Disease X problem, and how it’s a really useful way to explain to students that our brains are really bad a dealing with numbers. In Kahneman’s version people are asked to estimate how reliable a particular medical test is. Most people think the correct answer is 95%, when it is actually 1.96% !

I was reminder of this last earlier in the week when I read the news of a blood test to predict Alzheimer’s disease. The study by Mapstone etal “Plasma phospholipids identify antecedent memory impairment in older adults” was published in Nature Medicine, and reports a small scale, but extremely interesting study that suggests that particular fats in the blood might be predictors of Alzheimer’s Disease. As you might imagine, this story appeared in the popular press under headlines like ‘Blood test that can predict Alzheimer’s’. The first line of the Daily Mail’s version of the story was ‘A simple blood test has been developed that gives healthy elderly people precious early warning they may get Alzheimer’s within the next three years’.

What really interests me about this story is that buried deep in the reporting were two apparently innocuous bits of information, the sensitivity and specificity of the test were 90%. That is the test will be 90% accurate in determining positive cases correctly and 90% accurate in determining negative cases correctly. To the novice thinker this sounds like the test has an overall accuracy of 90%. However, as can be seen from the Disease X problem, there is one piece of information missing from this equation, namely the chances of actually developing Alzheimer’s. One of the difficulties of looking at stats for Alzheimer’s is that the chance of developing the disease increases rapidly with age, e.g. 1 in 1400 people between 40-64 have the disease, but the figure rises to 1 in 6 for the 80+ age group. The best estimate I can find for the lifetime risk of developing Alzheimer’s disease is around 15%.

If you plug all these numbers into the calculation from the Disease X scenario you come up with some interesting information on how useful this would currently be as a predictive tests for Alzheimers:



Using these figures you get a figure of 68% accuracy for the test (rather than the 90% that the stories imply). I should say that I’ve been very conservative with these numbers. The press stories about this imply that relatively young people could test for potential Alzheimer’s, and for them the base rate is not 15% but more like 1 in 1000 !!! Slot 1 in 1000 into the above calculation, and suddenly the figures look very bad.

It should be said that none of this should take away from the quality of this research. It clearly seems to be a big step on the road to a test for Alzheimer’s , it’s just not quite what the press is reporting. All in all this seems like an excellent real-world example to add to any demonstration of how bad we are at numbers

%d bloggers like this: