The perils of scientists who go fishing (Is Oscar the cat making me depressed)

27 Feb

photo

My usual focus here is on materials that are generic, and thus anyone could use them, but I’ve been thinking this week about research methods and thus this idea might really only be applicable to those teaching rational thinking at undergraduate level or above.

If you’ve  read anything else I’ve written here you’ll know that I have a strong focus on students understand the power of the scientific method to find ‘answers’. The current big idea of ‘big data’, and researchers ‘mining’ that data is somewhat alien to me. The  example of ‘big data’ that is currently exercising the British press is the government’s idea to make the data the National Health Service holds available to researchers.

I should say that there are clearly very good reasons to make such data available to researchers. Ben Goldacre has written an excellent article in the Guardian explaining why ‘big data’ is so valuable. My worry about ‘big data’ is that it encourages researchers to go ‘fishing’ through mountains of data without firstly progressing through making an observation, finding a theory and deriving a hypothesis from that theory. I’ve seen a great example of this data ‘fishing’ this week that rather sums up my fears. ‘Describing the Relationship between Cat Bites and Human Depression Using Data from an Electronic Health Record‘ by Hanauer etal was published in the peer-reviewed PLOS ONE journal, and using a very large ‘data mining’ sample produces evidence for exactly what the title describes.

Now, it may well be that this paper has discovered a great unrecognised source of depression (and god knows I’m no stats genius), but I’m really bewildered by the idea of using null-hypothesis testing when you didn’t actually have a hypothesis before you started the study. My worry about this sort of thing was further compounded by reading an excellent article in a recent edition of Nature that gets to the heart of the idea that many scientists really don’t understand the statistics they are using. In particular the article looks at the impact on statistical significance if one factors in the plausibility of the initial hypothesis. Maybe cats really do cause depression (my own cat certainly annoys me when he decides to wake up at 5AM), but just how plausible is this hypothesis, and thus how much can we trust the statistically significant result that has been found.

Here endeth this odd detour into the land of statistics, I shall return to rants about the popular press undermining rational thinking in the near future.

Advertisements

5 Responses to “The perils of scientists who go fishing (Is Oscar the cat making me depressed)”

  1. Paul Braterman February 27, 2014 at 5:36 pm #

    Why only 4 stars? Because you refer to “the scientific method”. To quote Peter Medawar, who thought more deeply about method than most other scientists, quoting Wm Whewell with approval, “There is no such thing as The Scientific Method … ‘An art of discovery is not possible’, wrote a former Master of Trinity, ‘we can give no rules for the pursuit of truth which shall be universally and peremptorily applicable,’ ” Hypothesis and Imagination, in Pluto’s Republic, OUP 1984, p.116

    • Teaching Rational Thinking February 27, 2014 at 11:06 pm #

      I have to admit I need to read more philosophy of science. My knowledge begins and ends with Popper’s Logic of Scientific Discovery.

  2. Paul Braterman February 27, 2014 at 5:37 pm #

    But while we’re on the subject, how soon will quote miners rediscover the correlation between birth rate and stork populations?

    • Teaching Rational Thinking February 28, 2014 at 5:59 pm #

      I suspect that much of my worry about this sort of stuff is borne out of coming from a discipline where inferential statistics are used to determine the validity of hypotheses. I’ve seen far to many psychology papers that seem to be great leaps forward that defy existing theory, but turnout out to be Type I errors.

      Of course it’s impossible to deny the existence of great intuitive leaps from scientific geniuses, it’s just that we still need the ‘poor bloody infantry’ of science to systematically work through all the hypotheses derived from that great leap. I guess what I’m saying is that what makes Darwin great is the combination of his leap and 150 years of lesser mortals testing hypotheses derived from it

Trackbacks/Pingbacks

  1. The glory of people being cleverer than me ! | Teaching Rational Thinking - April 10, 2014

    […] the last couple of days I’ve come across a lovely example of this from my own thinking. In the past few months I’ve written a few times about my worries about the use of ‘big d… I’m no statistics expert, but I’ve always worried about people trawling large datasets […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: