Tag Archives: peer review

When ‘consensus’ doesn’t mean what you think it means

14 Oct


I’ve just started teaching my new cohort of students, and this week used my favourite example of questionable peer-reviewed research, in which conclusions are drawn from self-report data on penis size ! As ever, even though the student were one-week into a three year degree programme they were well able to see that the paper, although published in a reputable peer-reviewed journal, was clearly nonsense. I was therefore really please to receive another great example this week from our brilliant librarian Ian Clark.

Last week saw a lot of reporting of a paper from ‘Psychology of Popular Media Culture’ suggesting that there was a consensus view that media violence leads to childhood aggression. The general tone of the reporting can be seen in this article from Time magazine. This example is in many ways far better that my favourite ‘penis-size’ paper, in that at first glance it looks entirely sensible and is published in a peer-reviewed journal from the august body that is the American Psychological Association. However, a few interesting points appear when one starts to delve:

  • The paper uses the words ‘broad consensus’ In it’s title, yet it appears that 69% of the participants agreed that media violence led to aggression. I may be a raging pedant, but when I see the phrase ‘broad consensus’ I was expecting something rather higher than 69% !
  • The study is essentially an opinion poll, none of the participants appear to have been asked if they have any evidence to back up their view. Whilst opinion polls are interesting, I’m not sure a peer-reviewed scientific journal is the place for them.
  • Even if one doesn’t think that the above two points are an issue, the fact that 36% of the participants in the survey had no further qualification to comment on the topic than that they were parents is truly worrying. Surely, a peer-reviewed journal ought to be soliciting the views of those who have conducted evidence-based research on the question to hand.

One final point, that I won’t dwell on here, but is very intriguing is the second  footnote that appears on page four of the paper:


The version of this manuscript initially submitted and

accepted was based on a different analysis, with communication

scientists and media psychologists combined in one

group as media researchers and identifying consensus as a

significant difference from the midpoint in groups’ average

responses. In reviewing an earlier draft of this manuscript,

the authors of a comment on this article (Ivory et al., in

press) correctly pointed out that these results could not be

interpreted as consensus. The editor gave us permission to

conduct a new set of analyses using a different operational

definition of consensus.


All in all this seems like a great way to demonstrate to students the necessity of reading beyond the headlines, even when reading a reputable peer-reviewed journal !



My penis is bigger than yours ! Quality psychology research ??

7 Oct

As I’ve hopefully demonstrated elsewhere on this blog, newspaper science articles are a great way to get students to hone their rational thinking skills. However the aim of such education is to get those students to employ their newly honed skills when they are reading the academic content of the rest of their undergraduate course. Achieving such ‘transfer of training’ isn’t easy, indeed psychologists have spent the best part of 100 years arguing about the extent to which it happens at all. It is entirely possible that our own teaching methods may actively discourage students from critiquing the content of academic journals. I’m sure many HE teachers spend a long time with new undergraduate students talking about the importance of ‘peer-review’ and how ‘peer-reviewed’ sources are much more reliable than students just ‘googling’ a topic.

One way to try to encourage students to use their rational thinking skills on academic papers is to try to collect ‘peer-reviewed articles’ that contain glaring methodological flaws. In most cases these are flaws that only become obvious years after publication, but htis week I’ve come across current article from a prestigious journal that serves this purpose perfectly.

My interest was piqued by an article in the Daily Mail headlined ‘We knew it all along: British men have bigger penises than the French according to new survey’. These sorts of story are usually obvious nonsense, but I was intrigued by two things, that the story was a report of an article published in the peer-reviewed journal ‘Personality and Individual Differences’, and that even the Daily Mail had highlighted methodological problems with the study

The story is based on the paper ‘An examination of Rushton’s theory of differences in penis length and circumference and r-K life history theory in 113 populations” by Richard Lynn. Ignoring the slightly eccentric theory behind the paper,three very obvious point appear when one reads it. Firstly, the data from the 113 population is presented to a surprising level of accuracy. Mean penis length is quoted for each population, and detailed down to hundredths of centimetres. I was immediately drawn to thinking about what type of measuring equipment would be necessary to achieve this level of accuracy. Secondly, many data points listed from the 113 populations were followed by a small asterisk. Turning to the bottom of the table you find out that the asterisk denotes ‘self-report’. YOu can only speculate how naive you would have to be not to questions the reliability of ‘self-report’ of penis length. Finally, reading through the method section of the paper you find that the data was not collected by the author, but rather culled from a website (http://www.everyoneweb.com/worldpenissize/). A quick skip through the web site leads you to a list of the sources from which they complied their data. WHilst some of them seem to be legitimate sources a number seem quite surprising:

The Elle/MSNBC.com sex and body image survey. Elle, pp. 111–113. Magna-RX. (2005, March). Does size really matter to your lover? More than you can possibly imagine! For Him Magazine, p. 117

The Happy Hook-Up: A Single Girl’s Guide to Casual Sex. Alexa Joy Sherman, Nicole Tocantins. p208. Ten Speed Press, 2004.

Argionic Desire. (2005, March). Argionic desire: Innovative penis enlargement product for men. For Him Magazine

Taken together these three things make me really wonder about the peer-review process. It’s not an exaggeration to say I’d question this sort of stuff it I saw if in a third year student’s thesis and yet it is apparently acceptable is the ‘gold-standard’ for academic writing. Having run through this paper with my 1st year undergraduate class I’d recommend it to anyone trying to encourage students to be active in critiquing ‘peer-reviewed’ material. Not only does the content guarantee you a cheap laugh from the students it gives them an understandingg that they can spot things that journal editors miss (or possible choose to ignore ???)

%d bloggers like this: