Tag Archives: Scientific Method

The perils of scientists who go fishing (Is Oscar the cat making me depressed)

27 Feb


My usual focus here is on materials that are generic, and thus anyone could use them, but I’ve been thinking this week about research methods and thus this idea might really only be applicable to those teaching rational thinking at undergraduate level or above.

If you’ve  read anything else I’ve written here you’ll know that I have a strong focus on students understand the power of the scientific method to find ‘answers’. The current big idea of ‘big data’, and researchers ‘mining’ that data is somewhat alien to me. The  example of ‘big data’ that is currently exercising the British press is the government’s idea to make the data the National Health Service holds available to researchers.

I should say that there are clearly very good reasons to make such data available to researchers. Ben Goldacre has written an excellent article in the Guardian explaining why ‘big data’ is so valuable. My worry about ‘big data’ is that it encourages researchers to go ‘fishing’ through mountains of data without firstly progressing through making an observation, finding a theory and deriving a hypothesis from that theory. I’ve seen a great example of this data ‘fishing’ this week that rather sums up my fears. ‘Describing the Relationship between Cat Bites and Human Depression Using Data from an Electronic Health Record‘ by Hanauer etal was published in the peer-reviewed PLOS ONE journal, and using a very large ‘data mining’ sample produces evidence for exactly what the title describes.

Now, it may well be that this paper has discovered a great unrecognised source of depression (and god knows I’m no stats genius), but I’m really bewildered by the idea of using null-hypothesis testing when you didn’t actually have a hypothesis before you started the study. My worry about this sort of thing was further compounded by reading an excellent article in a recent edition of Nature that gets to the heart of the idea that many scientists really don’t understand the statistics they are using. In particular the article looks at the impact on statistical significance if one factors in the plausibility of the initial hypothesis. Maybe cats really do cause depression (my own cat certainly annoys me when he decides to wake up at 5AM), but just how plausible is this hypothesis, and thus how much can we trust the statistically significant result that has been found.

Here endeth this odd detour into the land of statistics, I shall return to rants about the popular press undermining rational thinking in the near future.

Learning Styles: A topic in need of some rational thinking

7 Feb


As any group of students if they know anything about ‘Learning Styles’, and you’ll be surprised at how pervasive the concept is. The idea that we might all have our on personal ‘best way’ of learning is very intuitively appealing, and I suspect this is exactly why educational policymakers latched on to ‘Individualised Learning’ so quickly. If you haven’t come across Learning Styles before you can find a decent summary here, but in essence it’s the suggestion that we each have a favoured method of learning : one might be a visual learner (favouring pictures/ diagrams etc or an auditory learner, favouring spoken explanations etc)

This idea gained huge traction in education. I can remember meetings where university management explained that I needed to take into account student learning styles when designing teaching, and if you ask any students with kids you soon find that Learning styles are still around in schools. What’s really intriguing from a rational thinking perspective about Learning Styles is that as soon as you think about the idea it starts to fall apart.

By far the best demolition of learning styles can be found in Daniel Willingham’s work, but you can see what’s wrong with it merely by thinking about how you’d design an experiment to test it. Imagine two groups of participants, one made up of auditory learners and the other made up of visual learners. You give each group a sheet of paper with a list of words to learn, and you read out a second list of words to learn. You’d predict that the visual learners would remember more of the paper list and the auditory learner would learn more of the list read aloud. Of course, if you do that study you don’t find any difference at all.

I like to use this a an example for students who have already understood the importance of the scientific method. They can then easily come up with the experiment suggested above for themselves. Equally, I think it’s a lovely example of yet another idea that once lodged in the public consciousness is very difficult to dislodge.

My rational thinking books of the year (Part 2)

3 Jan

I’ve been thinking for some time about trying to codify my thoughts on teaching rational thinking into a coherent curriculum, and in researching the idea I came across a new book by one of my academic heroes, Jim Flynn.


‘How to improve your mind’ is let down by a cover that makes it look like a dreadful self-help manual, when it actually contains Flynn’s views on the concepts for intelligence growth that first appeared in his 2007 work ‘What is Intelligence ?’. The twenty key concepts that Flynn identifies provide a powerful basis for teaching rational thinking.

For those who might have only read Flynn’s works on Intelligence ‘What is Intelligence’ and ‘Are we getting smarter ?’ might be surprised by the tone of the book. Those two books draw very heavily on the available data with pages of tables and references, whereas ‘How to Improve Your Mind’ reads much more like a conversation with Jim Flynn in that it written in a precise but occasionally acerbic style.

I’d recommend the book for anyone considering teaching rational thinking although I’m not sure about using as a textbook for a rational thinking course. Flynn himself suggests that his key concepts ought to be taught to final year undergraduates, and I can see that the book might appeal to students at that level. However, for the 1st year undergraduates that I teach the book might be pitched slightly to high.

One final word of warning, Jim Flynn has a particular liberal world-view that comes across vigorously in this book. If you object to liberal politics you might want to avoid his work. (That said, if you’ve read this far you probably won’t have any problem with his views !!)

The return of the MMR lecture

23 Jul

For a number of years I used to give a lecture on the MMR vaccine debacle, that illustrated many of the points I wanted students to understand about rational thinking. Following the General Medical Council’s determination of serious professional misconduct against the doctors involved in the original article it seems like the story had run it’s course and so I decided to retire the lecture.

However, with a recent article it looks like the Daily Mail may be trying to kick start the issue. Sue Reid’s article, headlined “MMR: A mother’s victory. The vast majority of doctors say there is no link between the triple jab and autism, but could an Italian court case reignite this controversial debate?” details a recent Italian court verdict linking MMR and autism. I won’t rehash the whole sorry saga here, but you can find an excellent summary of the evidence here. The best that I can say is that any ‘rational’ reading of the evidence  suggests that the Italian court may not have seen all the research when they made their decision !

What I find particularly interesting about this case is that it cuts across many disciplines, and so could be widely used as an example. For ‘science’ disciplines one can discuss the ethics of the original research, the value of peer review, the difference between correlation and causation and a range of other topics. But for disciplines as diverse as Media studies,Political Science and Health Promotion could gain from discuss of the MMR fiasco. Media students might consider how a ‘controversy’ can be maintained in the absence of evidence, and politics students might think about how media coverage of the story peaked with the birth of the then PM Tony Blair’s last child was due to be vaccinated, rather than when the original paper was published. Health studies students might want to think about the impact of such stories on vaccination rates, and how such stories might be combatted.

I shall be spending some time over the summer digging out my old lecture slides and updating them. It seems like this particular lecture isn’t quiet dead yet !

Psychic Powers, Replication and the collapse of psychology ???

27 Apr

As part of my teaching I focus on students gaining an understanding of how knowledge progresses by the employment of the scientific method. One of the tenets of this teaching is the idea that you would never accept a hypothesis after one positive result, but would seek to replicate the finding with different samples and different experimental methods. When students begin to read scholarly journals they may gains rather different impression of how knowledge progresses, as the world appears to consist of positive findings associated with new theories and hypotheses. This absence of replication would seem to be one of those unwritten rules of academia, but recent developments seem to suggest that it might be beginning to unravel.

In 2011 the APA’s Journal of Personality and Social Psychology published a paper by Daryl Bem suggesting the existence of precognitionhttp://www.dbem.ws/FeelingFuture.pdf As you might imagine, a paper in a leading peer-reviewed journal reporting experimental evidence of a psychic phenomena produced a good deal of interest.

Three British psychologists Stuart Ritchie, Richard Wiseman and Chris French set about trying to replicate Bem’s work, and unsurprisingly failed to reproduce his findings. This is where the story gets interesting, as they struggled to get their failed replication published. In particular it was rejected by the Journal of Personality and Social Psychology, who had published Bem’s original work. Ritchie, Wiseman and French have written about the saga of trying to get their replication published in the May 2012 edition of The Psychologist.

All of the above makes for an interesting discussion with students, Psychic powers to engagement initially, the idea that we should be looking for replication and the evidence that we actually aren’t really interested in publishing it. However, another development makes it an even more interesting topic for discussion. A group of researchers started something they call The Reproducibility Project that is aiming to replicate all the work published in three leading journals Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition during 2008. There is an interesting article about this project in The Chronicle of Higher Education.

It’s interesting to speculate about what might happen if they fail to replicate much of the work from three such prestigious journals

%d bloggers like this: