Killing me softly with his (referee) report

Dear Colleague,

given that every other day we seem to be telling each other appalling stories of disgraceful article refereeing to which we are subjected, I think we should all try and agree, first of all among ourselves, on what constitutes “bad refereeing”, and pledge to each other not to do it. Ever.
After all, it is as good a starting point as any other. If we are successful at convincing others to agree to our code of conduct, we might see some difference in the future.

Let us be clear on what it is that at which I am aiming, here: I am not so naive to think that there is such a thing as “objective refereeing”. We are all humans, we have limitations, we try to be unbiased and fair-minded but fail, and and on occasion we simply err. There is no “fixing” any of that, and that is not the subject of this post.
However, one thing is not to be completely fair, to allow in part our own pre/misconceptions to influence our judgment, not to recognize in full the value of someone else’s contribution, not to have the time to examine in detail a manuscript submitted for publication, and miss out on some important aspect — we all have been, and will be, guilty of some of that.
Another thing is to submit a report which would embarrass the referee’s own children if they ever saw it, with the main intent of making the author’s blood boil, irreparably precluding the useful and constructive exchange that the refereeing process should achieve.

I know, we would think that the Editor of the journal to which the manuscript was submitted, i.e., the person who picked the referee to begin with, would do her job and edit reports before forwarding them to the authors, redacting sentences that are egregiously out of line — after all, by submitting the manuscript to the journal authors accept to be subjected to peer, not jeer review. Alas, these days editors are busy and pressed for time, and cannot be realistically expected to “sanitize” every single report. Thus, a considerable degree of “self-policing” is required on our part, when we get to play the role of anonymous reviewer, in order to keep the process civil, useful, and ultimately conducive to better science and more fruitful human and professional interactions.
I think that sticking to some basic principles and “refereeing rules” will go a long way toward reducing the time we spend venting and getting aggravated over a referee’s report that really rubbed us the wrong way, focusing instead on the useful part, namely the scientific feedback [0].
I hereby commit to observing the following, simple common sense guidelines:

0. Rule of thumb: Would I say this to the author face to face ?
If the answer is no, I ought not write it on my report. It’s that simple.
Anonymity is granted to the referee for the sole purpose of allowing one to express candidly a professional opinion, without being intimidated by a more prominent scientist and/or fearing possible future repercussions at the hand of a resentful (and childish) colleague.
Anonymity does not grant one the permission to cross the line of civility, go on a rampage, give free range to one’s most primitive impulses (possibly in an attempt to “even the score” with a competitor), lapse into vulgarity. Gratuitously disparaging someone else’s work is unprofessional, and it is low.
Yes, the Editor should not allow that, but because for one reason or another needlessly irritating statements end up being read by the authors, when we review manuscripts we have to make a conscious effort to avoid them at all costs.

1. Why are you working on this subject ?
I have written about this before. This is the one comment that referees make occasionally, that really drives me up the wall. “This subject has been beaten to death”, “This is an old problem”, “The community has moved on” are all examples of comments that do not belong in a referee’s report.
The task of the referee is to assess whether the work expounded in the manuscript is solid, the results reliable, novel and/or represent a useful piece of incremental information, whether the manuscript is written intelligibly, and to ensure that the relevant literature be cited [1].
The referee has not been asked whether the subject of the article is “hot”, “exciting”, “breathtaking”, much less to express an opinion as to whether the authors should be encouraged to keep working on that problem or not [2].

2. Ho-hum
There is nothing wrong, of course, with complimenting the authors if the referee is particularly impressed with the work that she is reviewing, but doing so is optional. By the same token, if the referee does not find the work very interesting, she need not share that with the authors. Sentences like “results are not spectacular”, “nothing to write home about”, “this is boring” and so on (much less any kind of sarcasm or caustic comments) are best avoided — they are uncalled for, and add nothing positive to the scientific discourse. If the referee really feels that way, she can keep that to herself, or maybe she can state her feelings to the authors in a different venue (see bottom of note [2]), not anonymously.
Again, is the work novel and correct — that is what the referee should assess, not whether reviewing the manuscript has been a life changing experience for her.

3. Close, but no cigar
There are times when novelty and correctness are not sufficient to warrant publication. The high profile of some (few) journals require that the referee make an assessment of the absolute importance or timeliness (urgency) of the work.
If the referee does not feel that the work described in the manuscript, albeit correct and worthy of publication in some form, merits the prestige and recognition associated with publication on that journal, she should do that without sounding patronizing and disrespectful [3], and most importantly by being as factual as possible [4].

4. Cite me !
Using the temporary position of “power” associated to anonymous refereeing, to seek personal gain and/or advance one’s own agenda is pathetic. I am sure we have all done it at least once, and we should all feel embarrassed by that.
Obviously, a manuscript should contain references to relevant prior work. A generic recommendation to the authors to make sure that they have carried out an extensive literature search is perfectly appropriate. If we believe that the reference list of a manuscript that we are reviewing is inadequate, and that crucial entries are missing, we should provide precise pointers to work that warrants citation [5].

I say, however, that a referee should never request (much less require) that her own work be cited. I am sorry, maybe it is me, I just find it tacky.
If the authors are neglecting to cite our work, and we feel that that is not fair to us and cannot let go, we should either let other referees point that out to the authors (they will, if it is such an egregious omission), or communicate our displeasure to them in person, non-anonymously, even if it means informing the authors that we are refereeing their manuscript [6]. If we really need to go after one extra citation in this way, then chances are that our work is not being that well cited anyway.

Anything else ?

Notes

[0] By the way, we should teach these rules to our graduate students and postdocs, and try to involve them in the refereeing process as early as possible.

[1] Of course, every so often submissions are received that are manifestly odd, outlandish and inconsequential, but those are typically easily spotted by the Editor and need no reviewing by peers.

[2] I cannot exclude that there may be times when “move on, already !” may be the right advice. I suppose it is possible that someone may be stuck on a dead end project, making use of a clearly inadequate methodology or attempting to achieve demonstrably hopeless goals. Frankly, however, that happens to be how a lot of ground breaking scientific discoveries have been, are and will be made, and I cannot help feeling very uncomfortable with the notion of anyone deciding that a specific line of research is “no longer worthwhile”.
As long as there are open scientific questions, as long as there is a community of experimental and theoretical scientists trying to get to the bottom of a particular problem, as long as articles written on that subject are cited and/or investigators funded, I do not believe that it is anyone’s business or prerogative to decide what should or should not be studied.
In any case, if the referee feels so strongly about sharing her personal opinion with the authors, she can easily do that in person, for example at a conference, after the manuscript has been reviewed and an editorial decision on it has been made — it is not part of the process of reviewing the manuscript.

[3] We never should forget that what we are reviewing is often times a graduate student’s first submitted manuscript. In general, language such as “It is self-evident/obvious/well-known that…” should be avoided.

[4] It is not always easy to be factual when making what ultimately amounts to a judgment call, but some effort should be made — to write something that sounds too much like “yeah, good but… eh, not good enough” and leaving it at that, is not fair to the authors. For example, the referee can state that, in her view the work does not constitute significant progress, i.e., it does not alter substantially the state of knowledge in a given field, based on a comparison of the results to existing ones, previously obtained by others (or even by the authors themselves).
Here too, providing references is a must.

[5] Needless to say, providing a reference is a must, if the referee is going to make the claim that authors are trying to publish findings or conclusions previously published by others — either exactly or essentially the same findings, especially if a recommendation of rejection of the manuscript is based on that claim. If that is the case, the referee should go to great length to explain why, in her view, the results contained in the manuscript reviewed constitute duplication of existing work. It is only fair that authors be given the option of rebutting that claim and state their own case.
Simply stating “the findings are not novel” without substantiation is unprofessional and unfair to the authors (Editor, where art thou ?).

[6] This need not be the case. In physics, for instance, manuscript submitted for publication are typically publicly available as preprints (on ArXiv) prior to their publication, allowing anyone to send (non-anonymous) comments to the authors.

Tags: Physics, Research, Science, Theoretical Physics