A new Form for the Evaluation of Scientific Articles under Peer Review*

Omar Sabaj Meruane

Universidad de La Serena

omarsabaj@userena.cl

Carlos González Vergara

Pontificia Universidad Católica de Chile

cgonzalezv@uc.cl

Germán Varas Espinoza

Universidad de La Serena

gvaras@userena.cl

Álvaro Pina-Stranger

CREM, Université de Rennes 1

alvaro.pina-stranger@univ-rennes1.fr

* This research has been supported by the Fondecyt Project 1130290: “The socio-discursive interaction in the collective construction of scientific knowledge: the internal dynamics of the peer review process”.

Abstract: In this article a new form for the evaluation of scientific articles under peer review is presented. This proposal is based on some empirical observations with respect to the process through which scientific knowledge is built. The principles underlying this proposal are simplicity, agility and clarity in the feedback for authors. Some of the weaknesses of peer review could be addressed by applying this new procedure.

Keywords: Peer Review Process, Research Articles, Scientific Evaluation, Evaluation Forms, Scientific Production.

Un nuevo formato para la pauta de evaluación de artículos de investigación en el proceso de revisión por pares

Resumen: En este artículo presentamos un nuevo formato para el formulario de evaluación de artículos de investigación, en el marco del proceso de evaluación por pares. La propuesta está basada en algunas observaciones empíricas respecto de este proceso que determina la producción del conocimiento científico. Los principios que sustentan la propuesta son la simplicidad, la agilidad y la claridad de la retroalimentación que, a través de los informes de evaluación, se les proporciona a los autores. La aplicación de este formulario permitiría mejorar algunas debilidades del proceso de revisión por pares.

Palabras clave: proceso de revisión por pares, artículos de investigación, evaluación científica, formatos de evaluación, producción científica

Un nouveau format pour l’évaluation des articles de recherche dans le processus d’examen par les pairs

Résumé: Dans cet article, nous présentons un nouveau format pour l’évaluation des articles de recherche, dans le cadre du processus d’examen par les pairs. La proposition est fondée sur des observations empiriques sur ce processus qui détermine la production de connaissances scientifiques. Les principes sous-jacents à la proposition sont la simplicité, l’agilité et la clarté de la rétroaction à travers les rapports d’évaluation qui sont remis aux auteurs. L’application de cette forme permettrait d’améliorer certaines faiblesses du processus d’examen par les pairs.

Mots-clés: processus d’examen par les pairs, articles de recherche, évaluation scientifique, rapports d’évaluation, production scientifique

Introduction

One of the most relevant roles of scientists is to legitimize other’s work, e.g. working as a referee or evaluator of other investigators’ knowledge (Campanario, 1986a). This contribution is crystalized in the peer review process (PRP). This article presents a specific proposal with respect to the PRP of research articles for scientific journals (Garfield, 1986a).

From a socio-economic perspective (Squazzoni, 2010) both the generation of scientific knowledge and the process of peer review are highly complex activities as all actors involved (authors, editors and referees) in this process fulfil specific roles depending on the circumstances. For instance, as authors we want our works to be accepted and published as soon as possible. As evaluators, however, we do not have the same promptness to respond to editors’ evaluation requests. Every minute we spend evaluating other’s work is time away from our own work. In the neoliberal logic that leads to competition, evaluating other’s work has become a heavy burden for the already busy academic life rather than a contribution for the scientific community.

In this competitive context, increasing speed rates in the generation and legitimization of knowledge through PRP is an imperative need, especially when this process has been widely criticized for slowing down the scientific progress (Björk & Solomon, 2013; Lyman, 2013).

In this work we propose a new evaluation form that allows improving the speed of PRP. In the first section we identify three specific problems that encourage this new format. After presenting the protocol itself, we conclude by providing some reflexive comments on possible ways of improving PRP.

Background

PRP is a socio-discursive complex process through which scientific knowledge is legitimized (Campanario, 1986b). Since the publication of the first journals, this process evolved from being highly focused on the editor to being first distributed in a closed group (editorial committee), and then in a broad community of peers.

As an epistemic coordination activity among different actors with specific attributes, PRP is considered to have a social character. It is also a discursive process as its different stages are materialized in the form of texts with specific purposes and associated with different actors (author’s manuscript, editor’s evaluation request, and referee’s evaluation report).

PRP has been criticized for multiple reasons (Campanario, 2002). For instance, it is said that it is a biased process (Bornmann & Daniel, 2009), it does not contribute to the improvement of articles (Callaham, Knoop & Gallagher, 2002), it cannot detect fraud or plagiarism, and it is slow and expensive. Depending on the actor’s perspective, the PRP is complex since it can serve different purposes. For example, from the author’s point of view, the PRP is considered a barrier he/she needs to cross in order to get published; for the editor, it is a process that helps him/her both to decide if publishing an article and improve submitted works (this aspect is also of interest for the author). Finally, for referees, the PRP is a way of knowing the last investigations as well as the researchers involved in his/her discipline (Garfield, 1986b). As we will see in the last part of this article, our proposal allows speeding up the process for each aforementioned role.

The evaluator as a scarce resource and the evaluation reports

Although referees are a key element for the scientific production, in the Latin-American context research agencies and journals do not provide evaluators of scientific articles with any monetary or symbolic recompense. Thus, for editors, evaluators are a scarce resource.

Any editor knows that finding competent evaluators who respond within the deadlines is a permanent challenge. The selection of evaluators, especially in the context of specialized knowledge, becomes a common problem for the editors of scientific journals. After being selected, the evaluator has to accept his participation, complete the report and then send it to the editor. Given the lack of suitable referees and the imperative need to make the process more efficient, Haspelmath (2014) suggests rejecting those works for which the editor could not found any evaluator available within a certain period of time.

Evaluation reports (Bolívar, 2008; Myers, 1985) are documents designed by editors in which evaluators provide comments on a manuscript as well as a recommendation for publication. The evaluation protocols are quite heterogeneous and often include elements that can be grouped in a series of criteria or dimensions (Bornmann, Nast & Daniel, 2008), such as design, relevance, formal presentation, data analysis, etc. Editors can also include concepts to be evaluated by means of closed questions (yes/no) and Likert scales. In some cases, it is possible to add both comments linked to those concepts or free comments. Lastly, some protocols offer the possibility to include comments for the author as well as for the editor. Editors often ask referees to evaluate the contribution or originality of an article, a rather difficult task since it is very hard for referees to handle all the necessary knowledge to answer this type of questions.

The complexity of these formats, the scarcity of evaluators and the lack of recognition by the system are all causes that slow down the production of scientific knowledge.

The problem of slow rejections and abuse in PRP

As mentioned, in this highly competitive context, the speed in the generation of knowledge is a crucial factor. Azar (2004) suggests that the destiny of the article relies heavily on a specific stage of PRP, e.g. the response time for the author. This moment is crucial for two reasons: if the article is accepted, the author can start to cite his/her work as “in press”, and, if rejected, he/she can make the modifications to submit it to another journal.

Perception of time in the PRP depends on the decision. Waiting too long for good news is not the same as waiting too long for a bad one. In this sense, the worst case scenario for both authors and editors is a long rejection. Authors will need more time to get their article published while editors will lose not only a publishable work but also their time and the evaluator’s. In this sense, the editorial practices should focus on improving their response time to the author and avoiding long rejections.

Several investigations have shown that a manuscript is never sufficiently bad as not to find a journal where to be published. In this sense, it is known that rejected works end up being published in other journals (Garfield, 1986a and b; Campanario, 1998a and b; Campanario, 2002). A negative consequence of this phenomenon is the perverse game in which authors submit low-quality articles in order to get the evaluator’s feedback and publish their work somewhere else. These authors overload both editor’s and evaluator’s work. In order to avoid this practice, we believe that neither evaluators nor the editor should provide authors with too much feedback when rejecting an article. Listing some general reasons supporting the decision would be fair enough.

Politeness and feedback quality

As its name suggests, the scientific knowledge legitimization process is an interaction among peers. For this reason, there are some often implicit restrictions and recommendations regulating actor’s behavior within PRP. As for language use in evaluation reports, for instance, non-offensive comments are encouraged.

A special feature of the PRP in terms of interaction is that, depending on the case, some of the actors can participate anonymously, as in single blind (referee knows the author’s identity, but not vice versa) or double blind processes (identities remain unknown for participants) (Báez, 2002).

This particular feature of PRP, in terms of an interaction among unknown peers, impacts negatively on the quality of the feedback that is provided to authors. Some investigations have shown that authors receive “mixed messages” (Bakanic, McPhail & Simon, 1989; Fiske & Fogg, 1990) and remain stuck (Gosden, 2203; Fortanet, 2008) after receiving positive comments on the work, typographical error descriptions, and change or inclusion suggestions, etc. Based on the analysis of 300 evaluation reports we have detected that the use of both conditional tense and verbs related to the act of suggesting is quite frequent. This is what we have called “mandatory suggestions”, e.g. statements of mandatory nature that are presented as if they were mere suggestions. In other words, this is a complete contradiction and a confusion source for the author. In this context, it is convenient that evaluators, while still being polite, clearly distinguish a truly mandatory change from a suggestion of change.

A new form for the evaluation of scientific articles under PRP

For the reasons above, a set of guidelines for elaborating a better evaluation form is now presented. The first part of the text will include a series of general aspects, such as article identification, reception date, and evaluator’s name. The evaluation form itself will start with a general question about the recommendation of the evaluator. It only has two options: “accept” or “reject” the article. In case of rejection, the evaluator will be able to check a default list of reasons supporting the rejection of the manuscript. Thus, rejecting the article will be quick since no specific feedback is provided to authors.

The subsequent parts of the form will apply only for accepted articles. No matter how good an article is, all accepted manuscripts need to be edited before being published. Therefore, accepting is always subject to certain modifications. In the second part of the form the referee will consequently have to brief all the changes that the authors will have to make to get their work published. As these changes will be expressed as commands or obligations (have to/must/etc.), the author has to assume that they condition the publication. Registering these changes for every accepted article will be mandatory for referees. In the third part, the evaluator will be able to optionally add suggestions. Consequently, taking these suggestions will also be optional for authors since the publication will not be subject to these modifications. These recommendations should have the formal structure of suggestions, e.g. using certain verbs and conditional tense. Both mandatory modifications (second part) and suggestions may correspond to any criteria the evaluator believes are relevant to improving author’s work and will not be subject to neither default reasons nor types of changes (formal, content, major, minor). The complete proposal is shown in Appendix 1.

Final comments

The format proposed in this work enables to speed up PRP, overcome some problems, i.e. scarcity of evaluators, complexity of formats, ambiguity in feedback, and improve time rates in PRP.

From the evaluator’s point of view, the format is simpler than others. If rejecting an article is needed, a modifiable default list with the reasons can be completed. If the referee decides publishing the manuscript, he/she must complete the second part using assertive and direct statements that the author must follow in order to get the work published. These modifications may correspond to any criteria the evaluator believes are relevant to assess work. As this is the only mandatory section for the evaluator and its nature remains quite clear, general descriptions without useful feedback are prevented.

From the author’s point of view, the use of a simpler evaluation format would allow him/her, in case of rejection, to have a quicker response about the state of his/her work, and, in the case of acceptance, to know what modifications he/she needs to make. In this context, the “mixed messages” are overcome.

With this format the editor would be able to best manage the evaluator’s time and reduce the response time for authors. In the case of accepted articles, the modifications suggested by the referees in the second part of the protocol would allow the editor to follow up the modifications for the definitive acceptation and publication of the manuscript. Obviously, a particular vision of the PRP underlies this proposal, i.e. that the process should be more efficient. The proposed format could be criticized for presenting a restricted vision of the process or being unfair for rejected authors, who will only receive a default evaluation list. However, we think this proposal allows improving some of the aforementioned problems. Due to the complexity of the PRP, further empirical evidence is needed to improve this important process that underlies the generation of scientific knowledge.

References

1. Azar, O. (2004). Rejections and the importance of first response times. International Journal of Social Economics, 31(3), 259-274. DOI: 10.1108/03068290410518247        [ Links ]

2. Báez, B. (2002). Confidentiality and Peer Review: The Paradox of Secrecy in Academe. The Review of Higher Education, 25 (2), 163-183.        [ Links ]

3. Bakanic, V., McPhail, C., & Simon, R. (1989). Mixed Messages: Referees’ Comments on the Manuscripts They Review. The Sociological Quarterly, 30 (4): 639-654.        [ Links ]

4. Björk, B. & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7: 914-923. DOI: 10.1016/j.joi.2013.09.001        [ Links ]

5. Bolívar, A. (2008). El informe de arbitraje como género discursivo en la dinámica de la investigación. Revista Latinoamericana de Estudios del Discurso, 8 (1), 41-64.        [ Links ]

6. Bornmann, L. & Daniel, H.D. (2009). Reviewer and editor biases in journal peer review: an investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18 (4), 262-272.        [ Links ]

7. Bornmann, L., Nast, I. & Daniel, H.D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative con tent analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77 (3), 415-432. DOI 10.1007/s11192-007-1950-2        [ Links ]

8. Callaham, M., Knoop, R. & Gallagher, J. (2002). Effect of Written Feedback by Editors on Quality of Reviews. JAMA, 287 (21), 2781-2783.        [ Links ]

9. Campanario, J.M. (1998a). Peer Review for Journals as It Stands Today –Part 1. Science Communication, 19 (3), 181-211.        [ Links ]

10. Campanario, J.M. (1998b). Peer Review for Journals as It Stands Today –Part 2. Science Communication, 19 (4), 277-306.        [ Links ]

11. Campanario, J.M. (2002). El sistema de revisión por expertos (peer review): muchos problemas y pocas soluciones. Revista Española de documentación científica, 25 (3), 267-285.        [ Links ]

12. Fiske, D. & Fogg, L. (1990). But the Reviewers are Making Different Criticisms of my Paper! Diversity and Uniqueness in Reviewer Comments. American Psychologist, 45 (5), 591-598.        [ Links ]

13. Fortanet, I. (2008). Evaluative language in peer review referee reports. Journal of English for Academic Purposes, 7, 27-37.        [ Links ]

14. Garfield, E. (1986a). Refereeing and Peer Review. Part 1. Opinion and Conjecture on the Effectiveness of Refereeing. Essays of an Information Scientist, 9 (31), 3-11.        [ Links ]

15. Garfield, E. (1986b). Refereeing and Peer Review. Part 2. The Research on Refereeing and Alternatives to the Present System. Essays of an Information Scientist, 9 (32), 3-12.        [ Links ]

16. Gosden, H. (2003). ‘Why not give us the full story?’: functions of referees’ comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2 (2), 87–101.        [ Links ]

17. Haspelmath, M. (2014). How to make scientific publication more efficient: Enforce deadlines without mercy, eliminate revise-and-resubmit. Available at: http://www.frank-m-richter.de/freescienceblog/        [ Links ]

18. Lyman, R. L. (2013). A Three-Decade History of the Duration of Peer Review. Journal of Scholarly Publishing, 44 (3), 211–220. DOI: 10.3138/jsp. 44.3.001        [ Links ]

19. Myers, G. (1985). Texts as Knowledge Claims: The Social Construction of Two Biology Articles. Social Studies of Science, 15(4), 593-630.        [ Links ]

20. Squazzoni, F. (2010). Peering into peer review. Sociologica, 3, 112-115.        [ Links ]

A new format for the evaluation of scientific articles in PRP

Appendix 1: A new format for the evaluation of scientific articles in PRP

Evaluation Form for Scientific Articles

     0. Identification and deadlines

    Code of the article:

        Title:

                   *Author:

                   *Referee:

        Reception date:

        Deadline:

        Submission date:

   * can be omitted according the type of process (single blind, double blind, open)

1. Decision for publication

• We would like to ask you to read the full article and then, as an expert in the field, answer the following question:

• Would you recommend the publication of this work?

If your answer is “yes”, please refer to items 2 and 3. If your answer is “no”, please complete the following table to finish your evaluation:

Reasons for rejection

If you rejected the publication of the article, please use table below. In the first part you will have to choose all the reasons that best justify your rejection. In the second part, you may indicate other reasons justifying your decision.

2. If you recommend accepting the manuscript, please brief the mandatory modifications that the author has to make in order to publish his/her work. Statements should be expressed as commands or obligations (have to/must/etc.).

3. Please, provide suggestions in order to improve the manuscript. Its publication will not be subject to these suggestions. Write statements containing expressions such as “the author should”, “it would be interesting…”, etc. Prefer conditional tense.