Peer review, or independent review by experts, is an important but far from perfect mechanism for quality control within the scientific enterprise. While it has many manifestations, depending upon the type of material in question, peer review generally involves a review of materials by experts who are thought to have adequate knowledge and technical expertise to judge the material’s quality, while being sufficiently impartial and disinterested to provide judgment free of conflict of interest.(1)
Peer review currently plays an important role in the production and shaping of knowledge that is the goal of the larger scientific enterprise. Peer review is invoked in the editorial pre-publication assessment of manuscripts submitted to scientific journals (often called “refereeing”) and in the decision-making processes of agencies and institutions that provide financial support for scientific research.
The widespread acceptance of editorial peer review as an integral part of development of scientific knowledge is a relatively new development, starting in the middle of the 20th century. A review of this history by Burnham found that the institutionalization of peer review practices by editors was not initially the result of a generalized recognition of the intrinsic value of this mechanism; rather the imposition of peer review was a reflection of the needs of editors, scientists or physicians themselves, “either to handle new problems in the numbers of articles submitted or to meet the demands for expert authority and objectivity in an increasingly specialized world.”(2) As a result, its institutionalization occurred in an idiosyncratic and haphazard manner, without the recognition of a single predominate model for editorial peer review.
Peer review is also used widely in evaluating the progress and performance of research programs, particularly those operated or funded by federal agencies. Perhaps the most well-known of the program evaluation activities in which the peer review model is applied is in the allocation of financial support for scientific research. It is uncommon for government agencies not to employ some type of peer review structures to assist in funding decisions.(1) In this way, grant-making peer review is similar to editorial peer review in its resource allocation function. In both cases, scientists are competing for scarce resources -- pages in a scientific journal or funds from an agency.
Peer review in research evaluation shares other characteristics with editorial peer review. For the most part, the identities of researchers are known to the reviewers, while reviewer identities are not (although membership on a federal review panel is public information). In contrast, peer reviews of funding applications rarely have the iterative quality of editorial peer review, in which authors are offered and re-offered the opportunity to refine their manuscript until the a point is reached at which the paper is either accepted for publication, or rejected or withdrawn. In reviewing funding proposals, competing applications are evaluated and scored, with the superior proposals generally being selected for funding by the sponsoring agencies. In contrast to the ultimate power of editors to accept or reject the advice of editorial peer reviewers, agency funders have little discretion to diverge from the rankings of the evaluators.
Interestingly, it appears that the development of editorial peer review and grant or financial support peer review were independent, with little interaction between the two. In the United States, the use of a formal peer review procedure to consider applications for support of research began in the early part of the 20th century. In contrast to editorial peer review, the diffusion and acceptance of peer review for funding reflects a shared recognition of the value of this process within the scientific community.(2)
A relatively new phenomenon is the application of peer review to the science involved in regulatory decision making. As regulatory agencies are increasingly called upon to make decisions and issue rules on the basis of increasingly complex science, they have developed quality assurance programs that share characteristics with the two types of peer review described above.
The federal regulatory agency with the most well-known engagement in the peer review process is the Environmental Protection Agency (EPA), whose regulatory needs require research involving an extraordinarily wide range of science and technology disciplines. The EPA is charged with developing environmental safeguards in areas in which there is much uncertainty and disagreement, often relying on complex mathematical models that may predict risk, exposure, emissions and costs. The inputs, as well as the outputs of these models are often the subject of great debate. Since the regulatory and financial repercussions of EPA’s research are so huge, it is not surprising there have been larger disputes over the nature and quality of EPA science.
Since the first Bush Administration, EPA policy requires that that major scientific and technical work products related to agency decisions should be peer reviewed, with independent or external peer review required for those documents supporting the most important decisions. In 1998, Carol Browner, EPA Administrator in the Clinton Administration issued a detailed handbook to ensure uniform implementation of the peer review policy. Only a relatively small portion of all documents the agency produces are peer reviewed following the procedures outlined in the EPA handbook. This policy, revised and reissued in 2000, has been endorsed by several subsequent reports. According to the EPA policy, scientific and technical work products that are considered candidates for peer review are those that are both used to support a regulatory program or policy position and that meet certain additional criteria, the first of which is that the work product “establishes a significant precedent, model, or methodology.” This policy recognizes that many of the studies and reviews used to support regulatory activity are not novel, but rather summaries and recapitulations of work that had already been subjected to other quality assurance processes and therefore not in need of additional review.
Notes:
(1) US Government Accounting Office. "Federal Research: Peer Review Practices at Federal Science Agencies Vary." GAO/RCED-99-99, March 1999.
(2) Burnham JC. The Evolution of Editorial Peer Review. 263 JAMA. 1990:1323. (abstract)
Read more:
Peer review of regulatory science: Introduction to peer review of regulatory science.
Recent developments in peer review of regulatory science: Information about the stated purpose of the Office of Management and Budget (OMB), Office of Information and Regulatory Affairs (OIRA) guidelines (first proposed in August 2003) and criticisms of the proposal.
Proposal revisions: Revisions to the proposal, published in April 2004, and the final proposal, published in December 2004.
Additional Resources
