PEER REVIEW AND BAD SCIENCEOctober 4, 2016
Mr. S.C., a regular contributor of articles here, found this one that appeared recently in the U.K.s The Guardian newspaper and sent it to me, and I include it here because of its intrinsic interest to what is happening in universities in the West, and the choking effects of "big science" and the emphasis on publication. Indeed, if you'll recall, a few days ago I blogged about the recent "poll" of the top universities in the world, which placed my alma mater, the University of Oxford at the top of the list. That poll, you'll recall, was based on various criteria, among them the amount of citations that appeared in various professional journals from various universities, with Oxford leading the list. As I noted at the time, mere quantity of citations in journals does not necessarily nor logically imply anything about the quality of those submissions.
Well, now the Guardian has weigh in with an interesting article:
Consider these paragraphs:
Paul Smaldino, a cognitive scientist who led the work at the University of California, Merced, said: “As long as the incentives are in place that reward publishing novel, surprising results, often and in high-visibility journals above other, more nuanced aspects of science, shoddy practices that maximise one’s ability to do so will run rampant.”
The paper comes as psychologists and biomedical scientists are grappling with an apparent replication crisis, in which many high profile results have been shown to be unreliable. Observations that striking a power pose will make you feel bolder, smiling makes you feel happy or that placing a pair of “big brother” eyes on the wall will protect against theft have all failed to stand up to replication.Sociology, economics, climate science and ecology are other areas likely to be vulnerable to the propagation of bad practice, according to Smaldino.
“My impression is that, to some extent, the combination of studying very complex systems with a dearth of formal mathematical theory creates good conditions for low reproducibility,” he said. “This doesn’t require anyone to actively game the system or violate any ethical standards. Competition for limited resources – in this case jobs and funding – will do all the work.
Drawing parallels with Darwin’s classic theory of evolution, Smaldino claims that various forms of bad scientific practice flourish in the academic world, much like hardy germs that thwart extermination in real life.
As the article also notes, there is also a lurking problem in publications for the past decades of too small statistical sampling and non-replicability that results from this. The problem is inbuilt into the system, for by emphasizing publication, one floods the market with "product" that emphasizes quantity and not quality:
Vince Walsh, a professor of neuroscience at University College London, said he was not convinced of the existence of a replication crisis, but that the paper raised valid concerns about the culture of science. “I agree that the pressure to publish is corrosive and anti-intellectual. Scientists are just humans, and if organisations are dumb enough to rate them on sales figures, they will do discounts to reach the targets, just like any other sales person,” he said.
There is, however, another factor that I think must be considered in any consideration of academic quality, and by "academic" I mean more than just the hard sciences: money. The problem now is that science has become "big science," dependent on government grants and big corporate money, which more often than not support research into "the narratives of orthodoxy", be those narratives scientific or otherwise. Consider CERN in this philosophical and epistemological context: how does one replicate completely independently the results that CERN is reporting and claiming? Obviously, one cannot built a gigantic particle collider in one's garage. But the point is, to replicate results completely independently of CERN, one has to build a similar and completely separate collider, and run the experiments again. This is perhaps one reason that China wants to build its own (and bigger) version of the large hardon collider in that country. (And, as I have suggested, perhaps another reason is they might suspect that some of the reported results are suspect.)
In short, science has turned into big business, and like all big businesses, it is not immune to the corrupting influences of money. And this bodes ill for the traditional practice of peer review, which as the article also implies without coming right out and stating explicitly, that process itself appears to be being "dumbed down," which prompts the need for for a thorough reconsideration of the process itself. Should there be a "full disclosure" or where the money is coming from for certain studies? I suggest there probably should be. In many cases, such studies are required to disclose the sources of their financing (this study was made possible by a grant from the such-and-such foundation). But perhaps it is time to examine the journals and their moneyed sponsors as well.
But this is a case for "you tell me," for much is at stake here, and the more voices participate, in my opinion, the better. Science, like philosophy and the arts, lie at the core of our civilization, and too much is at stake.
See you on the flip side...