When ‘consensus’ doesn’t mean what you think it means

14 Oct

Tom-And-Jerry-Picture

I’ve just started teaching my new cohort of students, and this week used my favourite example of questionable peer-reviewed research, in which conclusions are drawn from self-report data on penis size ! As ever, even though the student were one-week into a three year degree programme they were well able to see that the paper, although published in a reputable peer-reviewed journal, was clearly nonsense. I was therefore really please to receive another great example this week from our brilliant librarian Ian Clark.

Last week saw a lot of reporting of a paper from ‘Psychology of Popular Media Culture’ suggesting that there was a consensus view that media violence leads to childhood aggression. The general tone of the reporting can be seen in this article from Time magazine. This example is in many ways far better that my favourite ‘penis-size’ paper, in that at first glance it looks entirely sensible and is published in a peer-reviewed journal from the august body that is the American Psychological Association. However, a few interesting points appear when one starts to delve:

  • The paper uses the words ‘broad consensus’ In it’s title, yet it appears that 69% of the participants agreed that media violence led to aggression. I may be a raging pedant, but when I see the phrase ‘broad consensus’ I was expecting something rather higher than 69% !
  • The study is essentially an opinion poll, none of the participants appear to have been asked if they have any evidence to back up their view. Whilst opinion polls are interesting, I’m not sure a peer-reviewed scientific journal is the place for them.
  • Even if one doesn’t think that the above two points are an issue, the fact that 36% of the participants in the survey had no further qualification to comment on the topic than that they were parents is truly worrying. Surely, a peer-reviewed journal ought to be soliciting the views of those who have conducted evidence-based research on the question to hand.

One final point, that I won’t dwell on here, but is very intriguing is the second  footnote that appears on page four of the paper:

 

The version of this manuscript initially submitted and

accepted was based on a different analysis, with communication

scientists and media psychologists combined in one

group as media researchers and identifying consensus as a

significant difference from the midpoint in groups’ average

responses. In reviewing an earlier draft of this manuscript,

the authors of a comment on this article (Ivory et al., in

press) correctly pointed out that these results could not be

interpreted as consensus. The editor gave us permission to

conduct a new set of analyses using a different operational

definition of consensus.

 

All in all this seems like a great way to demonstrate to students the necessity of reading beyond the headlines, even when reading a reputable peer-reviewed journal !

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: