Is the Peer Review Process for Research Broken? A UVA Professor Argues for New Model

What if the peer review process, instead of being a gatekeeper for rewarding scholarly research, was the reward itself?

This is the proposition advanced by Brian Nosek and the framework for the model of research review he promoted during a presentation for the recently launched Distinguished Lecture Series from the University of Virginia’s School of Data Science. 

Nosek, a professor of psychology at UVA, co-founded three nonprofit associations, including the Center for Open Science, which emphasizes openness, integrity, and reproducibility in research.

The old model of review, Nosek said, is based on the notion that peer review is the gatekeeper to the reward that researchers seek, which is publication. 

“I need to publish as frequently as possible in the most prestigious places as possible in order to get my job, advance my career, get tenure, etc.,” he said, in describing the mindset the current system encourages.

However, he argued, the peer review process that leads to publication is fraught with challenges, including that it is opaque, with readers of the published work unaware of what went into the process. Additionally, he noted, publication decisions are final, with corrections rare. 

“They’re almost entirely ineffective at actually getting scholarship corrected or out of the literature because it’s not well linked to the original scholarship,” Nosek said of post-publication revisions.

The reward structure for research leads to dysfunctional behaviors, he said, such as predatory journals that lack peer review and allow for payment for publication.

So, what process could help ensure the trustworthiness of research? Nosek argued for a model that flips many of the underpinnings of the old model on their head.

“If we promote transparency in the peer review process; treat the peer-reviewed process as a life cycle; treat peer evaluation as a marketplace of indicators; and embrace versioning of findings, papers, and otherwise, we can subvert a lot of the dysfunctional practices more effectively,” he said. 

For instance, on the life cycle of the review process, Nosek made the case for reviews during various phases of the research process, rather than just at the end after the work is submitted. 

“Embracing life cycle research as an opportunity to have review occurring along the way means that the work is more visible from the outset when it’s initially planned,” he said, noting that the goals and objectives of proposed research could be part of the review process.

Nosek advocated for what are called registered reports, where researchers submit their proposals when they are being planned, offering the rationale behind their research question. If the research question and methodology pass the review process, a journal commits to publishing regardless of the outcome. The goals, he argued, are to limit the ability of dysfunctional markets to operate and to strengthen the rigor and quality of research. 

Nosek acknowledged that this new model was not a panacea to eliminate dysfunctional practices from the peer review and publication process but that it would make such actions much more cumbersome. 

“Any time that a system can be gamed, people that have an interest in getting the outcome simpler than what is required for it will try to game it,” he said. But, he added, this new model “increases the inconvenience of that.”