The Truth About Peer Review

Bernard Donovan

Secretary-General, Association of Learned and Professional Society Publishers

48 Kelsey Lane, Beckenham, Kent, BR3 3NE

Paper presented to the ICSU Press Workshop, Keble College, Oxford, UK, 31 March to 2 April 1998

Despite some questioning, peer review remains an essential part of the practice of scientific publishing. Indeed, the first Recommendation approved at the ICSU/UNESCO Conference of Experts on Electronic Publishing in Science in 1996 was that "strict peer review should be applied to all scientific material submitted for publication in electronic journals." In the present context, peer review refers to the anonymous peer review of articles carried out on a confidential basis by referees chosen by the editor or editorial board. The identity of the referees is withheld from the authors, unlike the "open" process advocated by some in which the entire exercise is carried out in public view. Anonymous peer review remains the selection process preferred by most journals, despite some acknowledged drawbacks. The authors trust the editor/s to choose expert referees who can be relied upon to produce an objective assessment of the manuscript, and, by and large, the system works. Occasionally a referee may fail to act objectively and betray bias of some kind. Then the editor seeks further advice elsewhere and drops that referee from the list of advisors. That is why it is customary to seek several reviews of each manuscript submitted in order to ensure objectivity and fair treatment. If this end is not achieved, then word gets around and the reputation of the journal suffers, which does no one any good.

While the preceding brief description of refereeing may be regarded as common knowledge and thus unnecessary, it serves to underline the point that the system is labour-intensive and expensive. And if needy learned societies and publishers continue to devote their energies and scarce resources to this activity then they must regard it as worth-while and as money well spent.

Last year, Tenopir & King (1997) produced some average publishing costs for journals produced in the United States. They concluded that the average direct cost to provide the first copy of an article is about $2,000. This figure includes article review, refereeing, subject editing and copy editing, preparation of illustrations and the preparation of master copies. The average direct cost of reproduction (paper, printing and binding) was put at an additional $30 per subscription. If indirect costs, such as lining up authors, subscription maintenance, journal marketing, amortisation of start-up costs and the preparation of book reviews, plus overheads such as labour, accommodation, equipment and so on, are taken into account another $2,000 is added, coming to $4,000 per article. These are assessments calculated from data covering thousands of journals of varying size and complexity in a wide variety of fields and serve as an indicators only; no figures are given for peer review costs. They do, however, serve to highlight the relatively small fraction of overall cost incurred by the costs of reproduction, and the minor savings generated by electronic methods of distribution.

Peer review is a critical component in the competition between rival journals. For good refereeing and editing raises the perceived quality and increases reader appeal. With pressures upon librarians to cancel subscriptions where feasible, this is the only way that journal publishers can reinforce, and hopefully extend, their presence in the library. With increased quality comes increased citation of published articles in other work, and a move towards the summit in citation indexes. Highly-cited journals attract more submissions, so that high quality is inevitably associated with a high rejection rate. It follows that quality journals spend more on the refereeing process, and that much of the investment appears to be wasted on rejected, and hence unremunerative, material. Thus, the costs of peer review loom large in journal economics.

What are the costs of peer review? In order to provide some realistic figures for discussion, I canvassed a number of ALPSP members and present the findings on a non-attributable basis. This is because the sample is small, skewed in the STM direction, and not collected with statistical analysis in mind. Several contributors were reluctant to be identified and it is fairer to treat all in the same manner. As this was an exploratory exercise to assess the kind of information available, no rigid parameters for the responses were applied, so that some of the findings are not directly comparable, one with another. It can also be difficult to dissect relevant figures from society accounts without effort and I did not want to make life too hard for my helpers. Despite these limitations the outcome is informative, and the readiness of publishers to help most pleasing.

One major scientific society employs a staff of about 25 and spends about 1.8 million to process some 9,000 papers a year, which would amount to 200 per paper if all were acceptable. Since the rejection rate is 50%, the cost doubles, to 400 for each publishable manuscript.

Figures from another society indicate that the yearly direct costs of staff solely engaged in managing peer review amounted to 290,000, with some 9,000 papers being handled. Other editorial costs were excluded. On this basis the average cost per submission came to about 33. With a 45% rejection rate the cost for each accepted paper came to 60. This figure appears low and very reasonable (as one would expect from an efficient learned society), but takes no account of editorial board costs, management, or overheads such as accommodation, heating, telephone, stationery and so on. For some small journals published by this society, peer review is administered by external editors and the costs are rather higher, of the order of 150 per paper published.

In a third society, where most of the peer review is administered in-house and there is great reliance upon a considerable input from the members of the editorial board, a 'rough estimate' comes to 60-70 per manuscript, which covers the salary of a staff member directly involved, a small extra allowance for managing and developing the system, plus part of the overheads. My correspondent writes: "I suspect that's quite cheap. I may have underestimated the overhead cost." The figure indeed looks small, but is for each submitted manuscript. With an initial rejection rate of 70%, and a final rejection rate of 50% (for some rejected papers are re-submitted and eventually acceptable after considerable editorial work) the cost per published manuscript doubles to 120-140.

For a fourth society, in the field of biomedicine, peer review is carried out by the academic editor, and the total cost of running the office (including the payment of fees to the editor and editorial board for handling manuscripts and some fees to the authors of reviews) amounts to 29% of turnover. On the basis of a 50% rejection rate, the average cost per accepted manuscript was of the order of 200.

Related figures from another STM journal run as follows: 2,100 papers are submitted yearly, with an outright rejection rate of 52%. Not all of the remaining 48% are published, for some merit major revision and even then may not prove acceptable, and others may not be re-submitted for publication after the first peer review. Dealing with this mass of material incurred editorial board costs (such as honoraria, expenses, committee meetings and an editorial board dinner) of 123,865, to which must be added office costs of 191,670. The latter figure covers staff salaries, training and recruitment, database and network costs, postage, telephone, stationery, premises and depreciation but not central overheads. Accordingly, 315,535 divided by 2,100 (the number of papers submitted) comes to 150 per paper sent in, but 288 for each paper accepted.

Figures are available for a clinical journal receiving about 900 manuscripts a year and with a rejection rate of 65-70%. The editor has a team of five associate editors who meet once a month to decide upon the manuscripts to be published. This procedure incurs travelling expenses, which amount to about 2,000 a year. The editor and associates are paid an honorarium linked to the number of manuscripts submitted, while the referees are not remunerated. However, sound statistical advice is crucial in this area, as elsewhere, and since "the only way to get sound statistical advice is to pay for it", 25 per manuscript is allocated for this purpose. Editorial staff costs amounted to 26,100 in the year examined, while other expenses (office rental, postage, telephone, travel, and other charges) came to 25,500. In summary, the total costs for this journal reached 74,280, so that with an input of 900 manuscripts reviewed the bill for each one was 83. Because more than 65% of the papers handled were rejected, the cost of peer review for each published paper rose to 237.

The trend indicated thus far might appear to be bucked by the information from a smaller journal handling around 500 submissions a year, of which half are rejected. In this case the total costs come to 25,000, or 50 per submission and 100 for each accepted. These observations are matched by another society coping with 650 manuscripts a year intended for a range of journals and spending 30,000 on peer review. The average bill for each manuscript received is thus 46, and, with a rejection rate of 30%, the outlay for each accepted paper is 67. However, if the honoraria for the Editors and the expense of committee meetings (for which the budget for 1998 is 64,700) are added in the figure per article received rises to 146, and that for acceptances becomes 209.

Several conclusions can be drawn from this small collection of material, summarised in Table 1. The first is that peer review is expensive, with the cost for each manuscript submitted ranging between 50-200, and for each paper published, between 100 and 400. The second relates to the remarkably consistent rejection rate, which (apart from one low figure) lies between 45 and 70%, and the third is that the signs of consistency in these figures collected from a range of publishers allows generalization.

However, the figures just cited relate to STM journals and a set of acceptance figures for a range of 10 highly-regarded journals in the social sciences field provokes reflection. In this instance, the acceptance rate ranged between 15 and 98% (15, 26, 29, 30, 31, 41, 47, 48, 74 and 98%). The two divergent journals, with acceptance rates of 74 and 98%, deal with highly specialized and technical topics, where the editors and the editorial requirements are probably familiar to likely contributors; whereas the journal accepting but 15% of submissions covered a much broader and ill-defined subject. Here the pool of possible material is vastly greater, and more would-be authors try their hand.

The results just presented reinforce the view that journals compete on the basis of their reader appeal, and that a quality control mechanism is essential for their well-being. Competition on the basis of subscription price does not work, for in the simpler days of print-only publication those learned societies that responded to pressure from their members to hold prices down in the face of price rises by rival publishers were not rewarded by sustained or increased subscriptions from grateful librarians.

Electronic peer review The application of electronic methods of data transmission should help to trim the costs just outlined, for there are savings to be made in postage, paper and envelopes and filing, as well as the avoidance of mailing delays. Unfortunately, the rewards are not great, as one set of figures kindly supplied to me shows. For this journal, postage costs for rejected papers amounted to 1.44, for papers requiring minimal revision 2.15, and for papers accepted after major revision 2.38. No account has been taken of overseas submission, the need for chase letters to dilatory referees, of the need for stationery, or of the fact that for this journal the Honorary Editors (or their employers) pay their own postage. My informant thus suggests that the figures be rounded up to 2.00 for rejected manuscripts, 3.00 for papers accepted after minor revision and 4.00 for those heavily revised. When set against the overall costs of refereeing the savings are marginal. The use of email, or the Web, can eliminate mailing delays, but the delays are usually small by comparison with the time taken by referees to complete their task. A week or two in the post counts for little if the manuscript has sat on the desk of the reviewer unread for a month or two. Editors make great efforts to reduce reviewing delays, but first class referees quickly learn that fast action is not necessarily good practice, for dispatch of a paper back to an editor is quickly followed by the arrival of another: in other words, there are benefits in a backlog in reducing work load.

Peer review can be managed electronically, but is no panacea in reducing costs, not least because of problems in handling complex formats and the lack of standardization. Although the number of word processing programmes on the market has fallen, at least 18 graphics packages remain in use and editors have to learn how to handle them. In theory, electronic refereeing offers many benefits, and would be readily accepted by the academic community, for 63% of the authors canvassed in a study done by the ESPERE project (Wood,1997) would submit their work electronically. In practice, life is complicated by the use of a variety of word processing programs and files (such as Word, PDF, HTML) and a range of graphics packages, which can be combined in a number of ways. Further, Dee Wood, of ESPERE, has noted the use of 11 different email packages, and these do not necessarily speak to one another in an ungarbled fashion. Thus electronic refereeing methods require the application of skilled staff and the installation of expensive computing equipment and software.

Enthusiasts provided with good computers, printers and software often forget that departments and individuals in other institutions may not be as well equipped as their own. All of the effort devoted to the electronic transmission of an illustrated digitized manuscript from one computer to another is wasted if the recipient lacks a suitable printer to obtain hard copies of the diagrams and pictures for close analysis. It is, as yet, unrealistic to expect a referee to edit a manuscript on screen. Sophisticated methods are required, for editors like to review the changes made by referees and want them clearly indicated, not merged with the rest of the text and requiring detection. Recourse is usually made to printouts of manuscripts for editorial purposes, with the need for subsequent re-coding of any changes made, and few readers are satisfied scanning on screen. They want hard copies.

As one means of reducing publication costs, the Association of Research Libraries is encouraging electronic publishing as a means of reducing dependence upon expensive journals and suggested in a discussion paper issued last October "that scholarly societies would be funded by manuscript submission fees paid in part by the academic employer of the manuscript author. For the fee, the society would conduct the peer-review process and render an evaluation of the article. The 'seal of approval' could then be fixed to any electronic version of the article. Print publication would not be necessary for tenure and promotion review purposes."

While the application of a quality mark to papers has considerable appeal, and is an idea that I have advocated elsewhere, the concept needs to be thought through. For example, on the basis that the cost of refereeing is 100 per paper, an output of three papers a year by each member of a small department of ten would generate costs of 3,000 a year. And that supposes that all are acceptable. If, as would seem reasonable, half were rejected, then it would cost 3,000 to get fifteen papers quality marked, or 200 each. Any presumption that refereeing charges would remain at the 100 level is probably unjustified, for societies would expect to earn revenue from their refereeing efforts and increase the price significantly. The concept also takes no account of the need for editing, and if properly edited papers are required, then the charges may well double.

It also seems that the proposal pays little attention to the trend by learned societies to produce electronic versions of their print journals, which naturally bear the quality mark of the name of the journal. This is of much greater value than a stamp or logo applied to a document lodged on a university server.

Finally, the proposal neglects the sense of community and society loyalties among academics. While referees are happy to act for a society of high reputation and for a valued journal, their enthusiasm will dim when their efforts emerge only as an electronic manuscript posted to a university archive. If their society is seen to act merely as a screening mechanism, and not developing a focus and identity through its journal, they may well lose interest in its activities. The Association of Research Libraries seems to regard the journal as merely a vehicle for publication, and to ignore the reality that societies are known, and shaped, by their journals. That is why societies strive to have their own publication or journal, however small. Disturbance of the close relationship between society members and their journal could have far-reaching, and very undesirable, consequences.

It is a pleasure to thank my colleagues for supplying the information included in this article, although my promise of confidentiality means that they cannot be named.

References

ARL Discussion Paper 1997 Scholarly communication and the need for collective action. URL: http://www.arl.org/sparc/discuss.htm

Tenopir, C. and King, D.W. 1997 Trends in scientific scholarly journal publishing in the United States. Scholarly Publishing, 28, 135-170.

Wood, D. 1997 Project update: Electronic submission and peer review - an update on the ESPERE project. Learned Publishing, 10, 157-159.

 

TABLE 1

The rejection rate and costs of refereeing of papers submitted for publication to eight learned societies

  Submission Rate Rejection Rate Submitted Papers Accepted Papers
  (Papers/year) (Per cent) ( cost per paper) ( cost per paper)
A 9,000 50 200 400
B 9,000 45 331 60
C   50 60-70 120-140
D   50 100 200
E 2,100 52 150 288
F 900 65-70 83 237
G 500 50 50 100
H 650 30 146 209
  1 Other editorial costs and overheads excluded

 


Last updated : 06 July 1998
Copyright 1998 ICSU Press and individual authors. All rights reserved