← Back to home

Peer Review

Right. You're here for an explanation. Let's get this over with.

If you were looking for the academic journal, you've stumbled into the wrong corner of the internet. See The Independent Review. This is about the process, not the publication.

A reviewer at the American National Institutes of Health, deep in the thrilling act of evaluating a grant proposal. Try to contain your excitement.

Peer review is the process of subjecting work to the scrutiny of one or more people who possess similar competencies to the work's producers. In simpler terms, it's judgment by your supposed equals, or "peers."[1] It operates as a clumsy but necessary form of self-regulation by qualified members of a profession within their relevant field. The entire apparatus is designed to maintain quality standards, theoretically improve performance, and bestow a veneer of credibility upon the final product. In the hallowed halls of academia, the ritual of scholarly peer review is most frequently used to decide if an academic paper is worthy of publication or if it should be quietly buried.[2] This process can be categorized by its type and by the specific field or profession where it occurs, such as the particularly fraught world of medical peer review. It has also been co-opted as a teaching tool, a way to help students improve their writing assignments by forcing them to critique one another's fledgling efforts.[3]

The dubious honor of 'father' of modern scientific peer review is often bestowed upon Henry Oldenburg (1619–1677), a German-born British philosopher who evidently thought getting experts to argue over manuscripts was a productive use of time.[4][5][6] The practice evolved sluggishly over the subsequent centuries, with the journal Nature, for instance, only making it a standard practice in 1973, which tells you something about the pace of innovation in these circles. The term "peer review" itself didn't even enter the lexicon until the early 1970s.[7] In a move that feels both fitting and deeply ironic, a monument to peer review has stood at the Higher School of Economics in Moscow since 2017, immortalizing the process in stone.[8] Make of that what you will.

Professional

Professional peer review directs its gaze upon the performance of professionals, ostensibly with the goal of improving quality, upholding standards, or granting certification. It's the mechanism by which a profession attempts to police itself, with varying degrees of success. Within the academic ecosystem, peer review is a critical, and often brutal, component in decisions concerning faculty advancement and tenure.[9]

A surprisingly early prototype of this process was recommended in the Ethics of the Physician, a text penned by Ishāq ibn ʻAlī al-Ruhāwī (854–931). He mandated that a visiting physician create duplicate notes on a patient's condition during every single visit. After the patient was either cured or had died—the two inevitable outcomes—the physician's notes were scrutinized by a local medical council of their peers. This council would then pass judgment on whether the treatment had met the required standards of medical care, a terrifyingly logical system for its time.[10]

This practice is now deeply embedded in health care, where it's commonly known as clinical peer review.[11] The field is further fragmented by clinical discipline, leading to physician peer review, nursing peer review, dentistry peer review, and so on—a seemingly endless fractal of professional judgment.[12] This isn't unique to medicine. Many other professional fields have instituted some form of peer review: accounting,[13] law,[14][15] and engineering (which includes specialized forms like software peer review and technical peer review). Even aviation and, improbably, forest fire management rely on this system of mutual evaluation.[16]

Peer review has also found a home in education, where it's used to pursue certain learning objectives. It's seen as a tool to push students toward higher-order processes in the affective and cognitive domains, as categorized by Bloom's taxonomy. The implementation can vary wildly, sometimes closely mimicking the formal, often punishing, scholarly peer review processes of science and medicine.[17][18]

Scholarly

This section is an excerpt from Scholarly peer review.[edit]

Scholarly peer review, also known as academic peer review or refereeing, is the core ritual of academic publishing. It involves subjecting a draft version of a researcher's methods and findings to an evaluation, usually conducted anonymously, by experts (the "peers") from the same field. This process is the gatekeeper, widely employed to assist academic publishers—be it an editor-in-chief, an editorial board, or a program committee—in deciding the fate of the work. The verdicts are stark: accept, accept with revisions, or reject for official publication in an academic journal, a monograph, or the proceedings of an academic conference. When the identities of both authors and reviewers are concealed from one another, the procedure is called dual-anonymous peer review, a double-blind dance of criticism.

The entire system hinges on the existence of a community of experts within a specific (and often narrowly defined) academic field. These individuals must be qualified and, ideally, capable of performing a reasonably impartial review. Of course, achieving genuine impartiality, especially for work in less rigidly defined or inter-disciplinary fields, can be a Herculean task. The true significance of an idea, whether brilliant or disastrous, may never be fully appreciated by its contemporaries. Despite its flaws, peer review is generally considered a cornerstone of academic quality and is a non-negotiable step for most major scholarly journals. However, let's be clear: peer review is no guarantee against the publication of invalid research.[19] Furthermore, as experimentally controlled studies of this process are notoriously difficult to conduct, direct evidence proving that peer review definitively improves the quality of published papers is distressingly scarce.[20] One recent analysis of randomized controlled trial abstracts did, however, find that editorial and peer review processes led to substantive improvements between submission and publication, offering a glimmer of hope for the beleaguered system.[21]

Medical

Main article: Clinical peer review

In the medical sphere, peer review can be dissected into four primary classifications, each with its own particular flavor of scrutiny:[22]

  • Clinical peer review: A procedure dedicated to assessing a patient's journey through the healthcare system. It's a component of ongoing professional practice evaluation and focused professional practice assessment—both of which are significant contributors to the credentialing and privileging of providers.[23]
  • Peer evaluation of clinical teaching skills: A process applied to both physicians and nurses to assess their effectiveness as educators in a clinical setting.[24][25]
  • Scientific peer review of journal articles: The standard academic gatekeeping applied to medical research before it sees the light of day in a publication.
  • A secondary round of peer review: This additional layer of scrutiny is applied to articles concurrently published in medical journals to assess their immediate clinical value.[26]

Complicating matters, the term "medical peer review" has been co-opted by the American Medical Association to describe not just the process of enhancing quality and safety within healthcare organizations, but also the rating of clinical behavior or compliance with professional society membership standards.[27][28] The clinical network, with what one might call determined optimism, considers it the best available method for ensuring that published research is dependable and that any advocated clinical treatments are both safe and effective.[29] The result of this terminological sprawl is a landscape of poor standardization and specificity, making "medical peer review" a frustratingly imprecise search term for anyone trying to navigate the literature.\

Technical

Main article: Technical peer review

Within the structured world of engineering, technical peer review stands as a specific type of engineering review. It's presented as a well-defined process, executed by a team of peers with assigned roles, all for the purpose of finding and fixing defects. These reviews are conducted by peers representing the various areas of a project's life cycle that are affected by the material under review, typically limited to a manageable group of six or fewer people. Technical peer reviews are strategically held within development phases, between major milestone reviews, and on completed products or finished portions of those products.[30] It's an attempt to impose systematic order on the inherent chaos of creation.

Government policy

Further information: U.S. Government peer review policies

Even governments have adopted this practice, presumably on the theory that if it's good enough for academics, it's good enough for bureaucrats. The European Union has been utilizing peer review within its "Open Method of Co-ordination" for policies in fields like active labour market policy since 1999.[31] In 2004, a similar program of peer reviews was initiated in the area of social inclusion.[32] Each program sponsors approximately eight peer review meetings annually. In these sessions, a "host country" exposes a particular policy or initiative to the scrutiny of half a dozen other countries and relevant European-level NGOs. These gatherings usually span two days and include site visits to observe the policy in action. The process is preceded by the compilation of an expert report, on which participating "peer countries" submit their comments. The results, for better or worse, are then published on the web.[citation needed]

The United Nations Economic Commission for Europe, through its UNECE Environmental Performance Reviews, employs peer review—rebranded as "peer learning"—to evaluate the progress of its member countries in enhancing their environmental policies.[citation needed]

In the United States, the State of California stands alone in mandating scientific peer review by law. In 1997, the Governor signed Senate Bill 1320, which requires that before any CalEPA Board, Department, or Office can adopt the final version of a rule, the scientific findings, conclusions, and assumptions underpinning that rule must be submitted for independent external scientific peer review. This mandate is codified in the California Health and Safety Code Section 57004.[33]

Pedagogical

Main article: Peer feedback

In the classroom, peer review, or student peer assessment, is the method where writers and their student-editors collaborate, theoretically to help the author establish, flesh out, and develop their writing.[34] It's a widely used tactic in both secondary and post-secondary education, a staple of the writing process. This collaborative learning tool requires groups of students to review one another's work, providing feedback and suggestions for revision.[35] It's often framed not as a simple critique but as a way to build connections between students and help them develop their identities as writers—a noble, if often unrealized, goal.[36] While a mainstay in English and composition classrooms, peer review has also gained traction in other disciplines that demand writing as part of their curriculum, including the social and natural sciences.[37][38]

Proponents argue that peer review in the classroom helps students become more invested in their own work and the classroom environment as a whole.[39] The experience of seeing how their work is interpreted by a diverse readership before a teacher grades it can, in theory, help students clarify their ideas and learn how to persuasively reach different audiences. It also offers a taste of the professional world, where they might one day be asked to review a colleague's work before publication.[40][41] The process can also, surprisingly, bolster confidence. Studies have found that students tend to be more positive than negative when reviewing their classmates' writing,[42] which might help writers feel determined to improve rather than discouraged.[42]

Critics, however, are quick to point out that the process can be deeply ineffective. Students often lack the practice to give constructive criticism or the expertise in the craft of writing to offer meaningful advice.[43] The process can be particularly problematic for developmental writers, especially if students perceive their own writing as inferior and are therefore unwilling to offer suggestions or ask for help.[44] Because students can feel a personal connection to their work, they may be reluctant to either receive or offer criticism, turning the entire exercise into a minefield of bruised egos.[36] When teachers treat peer review as just another assignment, it can lead to rushed, superficial feedback, with students doling out incorrect praise or unhelpful criticism, leaving both writer and editor with little to show for their time.[13] In response to these predictable pitfalls, instructors may provide examples, model the process with the class, or focus feedback on specific areas.[45] They might also experiment with different formats, such as in-class versus homework reviews, or leveraging the technologies of online learning management systems. Older students are generally better equipped to give useful feedback, yet the method is used across all age groups to teach the painful art of revision.[3] As technology continues its relentless march, peer review will undoubtedly evolve with it, with new tools promising to alter the process yet again.[46]

Peer seminar

A peer seminar is a method that involves a speaker presenting ideas to an audience that also serves as a "contest."[47] To be more specific, multiple speakers are called up one by one, each given a set amount of time to present their researched topic. While the topics may vary, each speaker has something to gain or lose, which can cultivate a distinctly competitive atmosphere.[47] This format encourages speakers to adopt a more personal tone as they attempt to appeal to the audience while explaining their subject matter.

Peer seminars bear some resemblance to conference presentations, but with key differences. There is generally more time to present, and speakers can be interrupted by the audience for questions and immediate feedback on both the topic and the quality of the presentation itself.[47] It's a more dynamic, and potentially more brutal, form of intellectual exchange.

Peer review in writing

Professional peer review is about the performance of professionals, aiming to improve quality, maintain standards, or provide certification. Within this broader category, peer review in writing is a crucial component, often driven by educators and involving students, especially in academic contexts. It serves as a fundamental process to ensure the quality, effectiveness, and credibility of scholarly work. And yet, despite its ubiquity, it remains one of the most scattered, inconsistent, and ambiguous practices in writing instruction.[48] Many scholars openly question its effectiveness and the specific methodologies employed. Critics of classroom peer review often cite its ineffectiveness, stemming from students' lack of practice in giving constructive criticism or their limited expertise in the craft of writing.

Critiques of peer review

See also: Scholarly peer review § Criticism, and Predatory publishing

Academic peer review has not escaped unscathed; it has faced considerable and sustained criticism, with numerous studies diligently highlighting the inherent, systemic issues in the process.

A significant concern is "role duality," where individuals simultaneously act as both evaluator and the evaluated.[49] Research has shown that occupying both roles concurrently biases people in their capacity as evaluators, as they engage in strategic actions designed to increase the likelihood of receiving a positive evaluation themselves.[49] It's a closed loop of self-interest.

The editorial peer review process has also been found to be strongly biased against 'negative studies'—that is, studies that report what didn't work. This systematically biases the entire information base of medicine. As Richard Smith noted in the Journal of the Royal Society of Medicine, journals become biased against negative studies when subjective values come into play. "Who wants to read something that doesn't work?" he asks. "That's boring." This bias, compounded by miscommunication inherent in the process, can prevent a writer's original vision from ever being realized.[50] Journals like College Composition and Communication often experience problems during peer review due to the diverse backgrounds of their writers and the varying degrees of bias among reviewers, which inevitably leads to conflict.[51] Teachers, too, have expressed disdain for the process, with many claiming it wastes class time and is pointless if students already have a sense of their grade.[52]

These critiques foster a belief among students that peer review is, fundamentally, pointless. This is especially evident in university classrooms. The primary source of writing feedback for students is often the teacher, whose comments are highly valued due to their position of authority. This can subtly influence students to produce research that aligns with the professor's viewpoints. Benjamin Keating's longitudinal study, "A Good Development Thing," compared writing and non-writing majors and found that non-writing majors tend to undervalue mandatory peer review, while writing majors place more value on their classmates' comments. This suggests that for peer review to be effective, a certain threshold of expertise is required. For non-professional writers, feedback from peers may simply be overlooked, crippling its effectiveness.[53] Further critiques have exposed the vulnerability of editorial structures on public knowledge platforms like Wikipedia. One archived account details how systemic rejections and unverifiable gatekeeping within Wikipedia's own editorial process mirror the same subjectivity and exclusion criticized in traditional academic peer review.[54]

Elizabeth Ellis Miller, Cameron Mozafari, Justin Lohr, and Jessica Enoch state plainly, "While peer review is an integral part of writing classrooms, students often struggle to effectively engage in it." They identify several reasons for this inefficiency based on research in university classrooms:

  • Lack of Training: Students, and even some faculty, often lack sufficient training to provide constructive feedback. Without clear guidance, reviewers struggle to offer meaningful insights.
  • Limited Engagement: Students may treat peer review as a mandatory chore rather than a learning opportunity, resulting in superficial feedback that fails to address core issues.
  • Time Constraints: Instructors frequently allocate insufficient time for peer review, leading to rushed, shallow feedback that lacks the necessary depth for real improvement.

This research underscores that beyond expertise, numerous objective factors contribute to poor student performance in peer review, yielding feedback that is ultimately unhelpful. The study also highlights the role of emotions, suggesting that neither reviewers nor authors can completely detach from their feelings. This can lead to biased feedback, as reviewers and authors approach the text with preconceived positive or negative attitudes, further undermining objectivity.[55]

Pamela Bedore and Brian O'Sullivan also express skepticism. After comparing different forms of peer review following systematic training at two universities, they conclude that "the crux is that peer review is not just about improving writing but about helping authors achieve their writing vision." Feedback from most non-professional writers tends to be superficial—simple grammar corrections and basic questions. This reflects a focus on merely improving the writing, not fulfilling the author's intent. Meaningful peer review, they argue, involves understanding the author's goals and posing valuable questions to guide them toward achieving that vision.[56]

Alternatives

Various alternatives to the traditional peer review model have been proposed, though none have been universally adopted. In the context of science funding, one such alternative is funding-by-lottery, a solution that embraces randomness as a potential antidote to bias.[57]

Comparison and improvement

Magda Tigchelaar's research comparing peer review with self-assessment yielded telling results. An experiment divided students into three groups: self-assessment, peer review, and no review. Over four writing projects, only the self-assessment group showed significant improvement. Tigchelaar's analysis suggests this is because self-assessment allows individuals to clearly understand their revision goals, as the author is most familiar with their own work. Self-checking thus follows a more systematic and planned approach. In contrast, peer review effectiveness is often hampered by unstructured feedback, characterized by scattered, meaningless summaries that fail to meet the author's needs.[58] Some educators suggest that for school assignments, having an instructional assistant conduct reviews instead of students would be more beneficial. Instructional assistants typically have more writing experience and can dedicate more time to discussing ideas, leading to a more valid and less biased review.[59]

Stephanie Conner and Jennifer Gray argue for the value of student feedback but acknowledge its frequent failures. They note that students often lack confidence in their own writing, which makes them hesitant to provide constructive feedback. They propose several improvement strategies, such as segmenting the peer review process into small groups where students present their papers for analysis, before expanding the scope to the entire class. This broadens the sources of review and could enhance its professionalism.[60]

To circumvent the miscommunication common in peer review, a student might ask their reviewer three focused questions about the paper. This directs the feedback, lessens the writer's anxieties, and helps build a sense of trust between the two parties.[48]

With the relentless advance of technology, peer review is also expected to evolve. New tools could transform the process. Mimi Li discusses the effectiveness of an online peer review software used in a freshman writing class. Unlike traditional methods, this software offers editing tools and comprehensive guidance, such as lists of potential questions for reviewers and the ability to add various types of comments. After a semester, students showed varying degrees of improvement and gave high praise to the technology, suggesting a possible, if not perfect, path forward.[61]