Killing me softly with his (referee) report

Dear Colleague,

given that every other day we seem to be telling each other appalling stories of disgraceful article refereeing to which we are subjected, I think we should all try and agree, first of all among ourselves, on what constitutes “bad refereeing”, and pledge to each other not to do it. Ever.
After all, it is as good a starting point as any other. If we are successful at convincing others to agree to our code of conduct, we might see some difference in the future.

Let us be clear on what it is that at which I am aiming, here: I am not so naive to think that there is such a thing as “objective refereeing”. We are all humans, we have limitations, we try to be unbiased and fair-minded but fail, and and on occasion we simply err. There is no “fixing” any of that, and that is not the subject of this post.
However, one thing is not to be completely fair, to allow in part our own pre/misconceptions to influence our judgment, not to recognize in full the value of someone else’s contribution, not to have the time to examine in detail a manuscript submitted for publication, and miss out on some important aspect — we all have been, and will be, guilty of some of that.
Another thing is to submit a report which would embarrass the referee’s own children if they ever saw it, with the main intent of making the author’s blood boil, irreparably precluding the useful and constructive exchange that the refereeing process should achieve.

I know, we would think that the Editor of the journal to which the manuscript was submitted, i.e., the person who picked the referee to begin with, would do her job and edit reports before forwarding them to the authors, redacting sentences that are egregiously out of line — after all, by submitting the manuscript to the journal authors accept to be subjected to peer, not jeer review. Alas, these days editors are busy and pressed for time, and cannot be realistically expected to “sanitize” every single report. Thus, a considerable degree of “self-policing” is required on our part, when we get to play the role of anonymous reviewer, in order to keep the process civil, useful, and ultimately conducive to better science and more fruitful human and professional interactions.
I think that sticking to some basic principles and “refereeing rules” will go a long way toward reducing the time we spend venting and getting aggravated over a referee’s report that really rubbed us the wrong way, focusing instead on the useful part, namely the scientific feedback [0].
I hereby commit to observing the following, simple common sense guidelines:

0. Rule of thumb: Would I say this to the author face to face ?
If the answer is no, I ought not write it on my report. It’s that simple.
Anonymity is granted to the referee for the sole purpose of allowing one to express candidly a professional opinion, without being intimidated by a more prominent scientist and/or fearing possible future repercussions at the hand of a resentful (and childish) colleague.
Anonymity does not grant one the permission to cross the line of civility, go on a rampage, give free range to one’s most primitive impulses (possibly in an attempt to “even the score” with a competitor), lapse into vulgarity. Gratuitously disparaging someone else’s work is unprofessional, and it is low.
Yes, the Editor should not allow that, but because for one reason or another needlessly irritating statements end up being read by the authors, when we review manuscripts we have to make a conscious effort to avoid them at all costs.

1. Why are you working on this subject ?
I have written about this before. This is the one comment that referees make occasionally, that really drives me up the wall. “This subject has been beaten to death”, “This is an old problem”, “The community has moved on” are all examples of comments that do not belong in a referee’s report.
The task of the referee is to assess whether the work expounded in the manuscript is solid, the results reliable, novel and/or represent a useful piece of incremental information, whether the manuscript is written intelligibly, and to ensure that the relevant literature be cited [1].
The referee has not been asked whether the subject of the article is “hot”, “exciting”, “breathtaking”, much less to express an opinion as to whether the authors should be encouraged to keep working on that problem or not [2].

2. Ho-hum
There is nothing wrong, of course, with complimenting the authors if the referee is particularly impressed with the work that she is reviewing, but doing so is optional. By the same token, if the referee does not find the work very interesting, she need not share that with the authors. Sentences like “results are not spectacular”, “nothing to write home about”, “this is boring” and so on (much less any kind of sarcasm or caustic comments) are best avoided — they are uncalled for, and add nothing positive to the scientific discourse. If the referee really feels that way, she can keep that to herself, or maybe she can state her feelings to the authors in a different venue (see bottom of note [2]), not anonymously.
Again, is the work novel and correct — that is what the referee should assess, not whether reviewing the manuscript has been a life changing experience for her.

3. Close, but no cigar
There are times when novelty and correctness are not sufficient to warrant publication. The high profile of some (few) journals require that the referee make an assessment of the absolute importance or timeliness (urgency) of the work.
If the referee does not feel that the work described in the manuscript, albeit correct and worthy of publication in some form, merits the prestige and recognition associated with publication on that journal, she should do that without sounding patronizing and disrespectful [3], and most importantly by being as factual as possible [4].

4. Cite me !
Using the temporary position of “power” associated to anonymous refereeing, to seek personal gain and/or advance one’s own agenda is pathetic. I am sure we have all done it at least once, and we should all feel embarrassed by that.
Obviously, a manuscript should contain references to relevant prior work. A generic recommendation to the authors to make sure that they have carried out an extensive literature search is perfectly appropriate. If we believe that the reference list of a manuscript that we are reviewing is inadequate, and that crucial entries are missing, we should provide precise pointers to work that warrants citation [5].

I say, however, that a referee should never request (much less require) that her own work be cited. I am sorry, maybe it is me, I just find it tacky.
If the authors are neglecting to cite our work, and we feel that that is not fair to us and cannot let go, we should either let other referees point that out to the authors (they will, if it is such an egregious omission), or communicate our displeasure to them in person, non-anonymously, even if it means informing the authors that we are refereeing their manuscript [6]. If we really need to go after one extra citation in this way, then chances are that our work is not being that well cited anyway.

Anything else ?


[0] By the way, we should teach these rules to our graduate students and postdocs, and try to involve them in the refereeing process as early as possible.

[1] Of course, every so often submissions are received that are manifestly odd, outlandish and inconsequential, but those are typically easily spotted by the Editor and need no reviewing by peers.

[2] I cannot exclude that there may be times when “move on, already !” may be the right advice. I suppose it is possible that someone may be stuck on a dead end project, making use of a clearly inadequate methodology or attempting to achieve demonstrably hopeless goals. Frankly, however, that happens to be how a lot of ground breaking scientific discoveries have been, are and will be made, and I cannot help feeling very uncomfortable with the notion of anyone deciding that a specific line of research is “no longer worthwhile”.
As long as there are open scientific questions, as long as there is a community of experimental and theoretical scientists trying to get to the bottom of a particular problem, as long as articles written on that subject are cited and/or investigators funded, I do not believe that it is anyone’s business or prerogative to decide what should or should not be studied.
In any case, if the referee feels so strongly about sharing her personal opinion with the authors, she can easily do that in person, for example at a conference, after the manuscript has been reviewed and an editorial decision on it has been made — it is not part of the process of reviewing the manuscript.

[3] We never should forget that what we are reviewing is often times a graduate student’s first submitted manuscript. In general, language such as “It is self-evident/obvious/well-known that…” should be avoided.

[4] It is not always easy to be factual when making what ultimately amounts to a judgment call, but some effort should be made — to write something that sounds too much like “yeah, good but… eh, not good enough” and leaving it at that, is not fair to the authors. For example, the referee can state that, in her view the work does not constitute significant progress, i.e., it does not alter substantially the state of knowledge in a given field, based on a comparison of the results to existing ones, previously obtained by others (or even by the authors themselves).
Here too, providing references is a must.

[5] Needless to say, providing a reference is a must, if the referee is going to make the claim that authors are trying to publish findings or conclusions previously published by others — either exactly or essentially the same findings, especially if a recommendation of rejection of the manuscript is based on that claim. If that is the case, the referee should go to great length to explain why, in her view, the results contained in the manuscript reviewed constitute duplication of existing work. It is only fair that authors be given the option of rebutting that claim and state their own case.
Simply stating “the findings are not novel” without substantiation is unprofessional and unfair to the authors (Editor, where art thou ?).

[6] This need not be the case. In physics, for instance, manuscript submitted for publication are typically publicly available as preprints (on ArXiv) prior to their publication, allowing anyone to send (non-anonymous) comments to the authors.

Tags: , , ,

17 Responses to “Killing me softly with his (referee) report”

  1. GMP Says:

    In regards to [4]: when it comes to publishing in PRL or a GlamourMag, indeed, as you say, a large portion of the review is a judgement call. The paper may be perfectly solid — novel and correct — but simply does not strike you, the reviewer, as something that will be read widely enough or that is of broad enough interest for PRL.

    I remember a couple of papers I recently rejected for a PRL, where I was able to pinpoint exactly what resulted in rejection:

    1) One was very good and solid, but the phenomenon was noted only for a very specific form of a Hamiltonian within an extremely narrow range of parameters. That was the primary reason I recommended that it be a PRA rapid rather than a PRL.

    2) Another was simply too cramped for PRL — they developed a new method AND proposed to use on a new model system. That’s a nightmare on 4 pages and made for an entirely incomprehensible paper. I recommended that they write a longer paper in which the method is properly validated; you don’t embark on a new and as yet uncharacterized model systems with a new and uncharacterized method.

    Often, these calls are hard to justify in their entirety and I understand it sometimes boils down to the gut feeling. We recently had a split vote on a PRL where a total of 4 referees were involved in round 2 of review (3 in first round, 2 new added in the second round but 1 from the first was MIA): two were saying “it’s all nice and swell but it still strikes me as not PRL”, two were saying that it is PRL material. The pro-PRL two were more specific in why exactly they thought it’s PRL material, and I think it did eventually influence the editor to accept after we resubmitted.

  2. Massimo Says:

    The paper may be perfectly solid — novel and correct — but simply does not strike you, the reviewer, as something that will be read widely enough or that is of broad enough interest for PRL.

    Yeah but, I find it unfair to put authors in the condition where they have to accept a “no” without anything substantive, to which they could respond in their appeal and have the Divisional Associate Editor compare two view points.
    Sure, in the end it is always going to be a judgment call, but I still think that a good referee will try and motivate these feelings in some “semi-quantitative” fashion, in order not to come across as entirely whimsical. The examples you gave are good.

  3. James Says:

    This post seemed particularly relevant to me.

    In a recent and private letter to Editor of Big Important Journal, I wrote of Referee A: “I don’t understand the apparent lack of tact in the presentation of their criticisms. I have trouble imagining that this reviewer would speak to me with the same manner in which they have written.”

    This was in response to Referee A’s inital report, a sampling of which included: “The manuscript suggests that these researchers are new to Research Field X, and their efforts and choice of topic are welcome. Unfortunately, the manuscript is rife with rookie mistakes, and there are indications that the underlying study is fatally flawed.”

    Another hair-tearing example was: “The protocol for Procedure C is not reported, but this reviewer suspects that an opportunistic sample at Author’s Institution was used and that Objectives Of Test Z were determined after Development of Test Z – a recipe for confirmation bias.” Thanks for the vote of confidence, Referee A, and also that’s not what confirmation bias is.

    I worked hard to make all the changes Referee A suggested. In their second report, they wrote: “While the authors’ efforts are appreciated, the study is deeply flawed and unworthy of publication. Their Experimental Process is not something Researchers in Field X should emulate, nor is the End Result useful to Others.”

    Of course, Referees B (and now C and D as well, because of my appeal) are all relatively happy with the work and suggest only minor changes.

    • Anonymous Says:

      This comment appears to have been written by a blog novice. It is fatally flawed, and as such of no use to any of the blog’s followers. The author should be discouraged from future blog commenting.

    • Massimo Says:

      Sorry, I could not resist 🙂

      • James Says:

        Ha ha, very funny.

        The good news, I should’ve added, is that Referees B, C, and D are all now extremely happy with the state of the manuscript. Furthermore, Editor has since agreed that A was out-of-line (and is now also out-of-this-refereeing-process); I anticipate that this manuscript will get published soon.

  4. transientreporter Says:

    Anonymity makes for cowards. Just like blogging… =)

    Incidentally, speaking of anonymity, why isn’t peer-review double-blind? Aren’t reviewers swayed by big names and dazzling pedigrees?

    • Massimo Says:

      Good question, I have never understood myself why peer review is not double-blind. Maybe at the end of the day it would not make any difference, in that figuring out who the author is could be pretty straightforward in most cases (e.g., from the reference list).

    • Schlupp Says:

      I am fairly sure that I am swayed. Even if it would probably not be hard to figure out the authors (e.g. from the arXiv), I’d prefer to get the papers without author names (and affiliations), so that at least my first impression has some chance of being less affected.

      IOP has a cheesy guide in an annoying format here:
      They also emphasize that referees should give some basis for their recommendation.

    • transientreporter Says:

      I’m with Schlupp. Yes, you can (probably) figure out the author from the citations, but that’s no reason not to do double-blind submissions. It’s still not foolproof, obviously. For instance, the recent kerfuffle over the EO Wilson paper in Nature (Ooooooooooooh, EO Wilson…; Oooooooooooh, Harvard…) is probably as much to do with editors strong-arming reviewers as anything else.

      A friend of mine suggested TRIPLE-blind submissions, where the authors submit to an impartial third party first, who then pass it onto the journal editors – hence, even the editors don’t know who the authors are…

      • Massimo Says:

        I would go one extra step and actually have referees see not the actual names– replace those with names of Harvard people chosen at random, how brilliant is that ?

      • GMP Says:

        where the authors submit to an impartial third party first, who then pass it onto the journal editors

        This sounds exactly like “manuscript review insurance companies” — I wouldn’t be surprised if the impartial party started making money on it.

      • transientreporter Says:

        Looks like we’ve got a business plan. Massimo, call up some venture capitalists and hit ’em up for some cash.

    • prodigal academic Says:

      I totally agree–I have often wondered why peer review isn’t double blinded. When one of the ecology journals switched to double blind review, the % of female first authors increased, while in a comparable non-double blinded journal there was no change (ref). This is surely evidence that unconscious bias from seeing the authors names is alive and kicking. Who cares if it is easy to figure out the authors. I think lots of people won’t bother, and some who will at least have an initial impression without author name bias.

  5. Schlupp Says:

    Today at lunch, I learned a great strategy to save time an effort: Just don’t bother replying to reviewing requests from journals, after a while, they will stop coming. How come *I* never thought of that one??????

    • Massimo Says:

      How come *I* never thought of that one??????

      Because you made the mistake of thinking “Oh, wait… how do I feel when I check the APS manuscript status web site to see what’s going on with my paper, and after 14 weeks it says ‘(Yet another) reminder to referee’ ?”

  6. Schlupp Says:

    Well, that does answer my question, but also raises another one: How to I avoid this mistake in the future? Thinking is a stupid mistake to make!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: