The future of scientific publishing

“The public is the only critic whose opinion is worth anything at all”
(Mark Twain)

An interesting debate has been taking place over the past few weeks inside blogosphere. The matter of contention is whether science-inspired (possibly but not necessarily anonymous) blogs, should be regarded as an appropriate venue to discuss, and provide criticism of, peer-reviewed scientific articles published on academic journals (a summary of the discussion, together with an eminently reasonable opinion on this subject, can be found in this post). Aside from the specific article and blog post that have originated the debate, the relationship between “official” scientific journals and relatively new, interactive media such as weblogs, is undoubtedly an interesting topic and a timely issue.

It has been long the position of the academic establishment, that peer review provides the necessary, official “vetting” of a manuscript, i.e., of the scientific quality of the work expounded therein, and that the orthodox way for a scientist to criticize, or attempt to debunk its results or conclusions, is to submit a competing manuscript for publication, thereby undergoing the same type of scrutiny as the original article.
Indeed, I remember reading this very point some fourteen years ago, made by physicist David Pines, in a letter to the editor of Physics Today, written in response to strong scientific criticism of his work levied by Nobel laureate Philip W. Anderson, in another letter to the same issue of that journal [0]. In general, I can see the merits of such a position, but I cannot help regarding it as becoming rapidly obsolete.
Obviously, much like scientific journals, not all scientific blogs can be good. However, I think that we all had better get used to the idea of defending our work on the web, including on blogs, and to the direct, interactive discussion of scholarly content that takes place on this type of medium. It is not just a fad — science blogs are here to stay, for a number of reasons.

First off, there is the issue of peer review, and its often overstated value. As eloquently stated by P. Z. Myers,
“Passing peer review and getting published does not mean that your work is right. Some incredibly awful papers get through the review process, somehow. Getting published only means that now your paper is going to be opened up to wider criticism.”
I have to agree.
An article that is accepted for publication, even in a high profile journal, has been assessed by a number of scientists that is seldom grater than five or six (and typically just one or two). With everyone being pressed for time these days, the high degree of specialization of every scientific field and the fierce competition existing among scientists, the goal of a fair, thorough, timely and unbiased review of a manuscript submitted for publication is often little more than a pipe dream.
While peer review probably remains an effective device to weed out the (relatively small) fraction of submitted manuscripts that ostensibly fail to meet accepted standards of accuracy and scientific rigor, the fate of the majority of articles submitted for publication, chiefly to the most prestigious journals, is all too often determined by less than scientific, highly subjective (sometimes borderline whimsical) and certainly fallible criteria.
And there are fundamental reasons to believe that anonymous peer review, once regarded as a cornerstone of the scientific enterprise, may become ill-suited in the years to come, especially vis-a-vis the predicted growth in size of the scientific community [1].
Indeed, it has been long recognized that, due to the unreliability of peer review, the actual impact of a scientific paper ought to be ultimately assessed by the number of citations that the paper receives in the course of the years, (as opposed to, e.g., its publication venue), hence the introduction of indices such as Hirsch’s.

Then there is the issue of timeliness. These days, research advances are made at a breakneck pace and scientific communication takes place at a speed inconceivable only two decades ago.
Online article repositories such as are rapidly supplanting journals as the method of choice of scientists to keep abreast of developments in their fields. Manuscripts are uploaded every day by the hundreds, freely accessed and downloaded by scientists all over the world, discussed not just in private, but at conferences, as well as on blogs and web-based journal clubs, and extensively cited in the “regular” literature even before their actual journal publication (in fact, often times long before that) [2].
I think it is fair to say that, for the most part, by the time a paper is published (in the traditional sense), it is not “novel” anymore, and in some cases peer review and eventual “regular” publication are an almost frivolous exercise, formally sanctioning a judgment that the community has already collectively expressed about a particular piece of work. I can easily see such a formal appraisal becoming irrelevant, in the future. Interesting pieces of work will be cited and elicit discussion, even without being published in the traditional sense; uninteresting ones will be forgotten, regardless of whether they eventually get published somewhere “formal”.
It is true that a paper appearing on a repository has not been officially “vetted” by peer review. But, as mentioned above, peer review is no surefire guarantee of quality. Moreover, much like the number of citations offers a more robust indication (admittedly one not free of bias) of the impact of a journal article, measures rendered available by online media (such as the number of downloads and hits) appear capable of offering at least comparably reliable an assessment.
Meanwhile, scientific journals are struggling, especially second-tier ones, frantically exploring novel business models (none of which seems to work), overwhelmed by exploding costs, and inexorably unable to offer a satisfactory product to authors eager to claim priority, and unwilling to let their manuscripts languish for weeks inside the electronic in-box of lazy or malicious referees (read: competitors).

Finally, there is the issue of practicality. It is impossible to prevent anyone from writing anything about anything. While this obviously gives no green light to anyone wanting to indulge in slander, disrespect, threat or any other behavior that is either illegal or otherwise not tolerated by society (on the internet or elsewhere), any discussion or attempt to regulate what, by its own nature, cannot be truly controlled, seems a waste of time. Much better is to think of a way to coexist peacefully.
Anonymous peer review with reviewers selected (more or less carefully) by an editor, dates back to a time when the internet was not even a word, and the scientific world far more elitist a place than it is nowadays. Blogs allow virtually anyone to be a reviewer, and while this requires that good judgment be exercised by everyone, I tend to regard this as a good thing.
I am afraid that those who wish to preserve the traditional modus operandi and its solemnity, with evaluation of scholarly work confined to the editorial offices of established journals and conforming to tried-and-true practices, rituals and hierarchies, and dread the thought of scientific articles not been reviewed in this way, discussed in the informal setting provided by a blog, are in for a rough ride.

[0] Physics Today 47 (2), 1994. Although both letters discussed in technical terms a well-defined scientific topic (high-temperature superconductivity), neither could be regarded as a regular article, of the type that are normally cited in the scientific literature, nor had either been peer reviewed before being formally accepted for publication.
[1] In his 1994 article The Big Crunch, physicist David Goodstein pointed to the inherent conflict of interest of an anonymous reviewer, as an increasing problem and potential Achilles’ heel of this practice, especially in a time of diminished funding opportunities and increased competition among scientists.
[2] Some web sites such as Naboj are introducing interesting new concepts such as open review of manuscripts uploaded on public repositories such as arXiv. While this experiment is clearly at a very early stage and it may be a while before it becomes common practice, it does not seem too far-fetched to see this as a plausible direction in the evolution of scientific evaluation.

Tags: , , ,

5 Responses to “The future of scientific publishing”

  1. Anonymous Says:

    Hey Massimo

    I think we can all agree that:
    1) the current system cannot continue to exist in its present form.
    2) the internet is going to play a role in whatever system that replaces it.

    So, here’s a question for you. If the current tiered publication system + anonymous reviewers etc did not exist, and you had to design a review mechanism from scratch, how would you do it?

    Who gets to act as reviewers? Anybody? Including the general public? Also, why not dispense with “papers” i.e. a set of experiments that tell a coherent story – and just publish individual experiments. Yikes! But why not?

    This is going to be fun, isn’t it…

    • Massimo (formerly known as Okham) Says:

      Actually, it is not a “hypothetical” scenario, as much of it is taking place already. One could think of having all papers uploaded on arXiv. Now, currently you have to register with the system but I think anyone can do it, which is probably not a good idea. Presumably a system would have to be set in place to examine and approve individual registrations.
      Now, a way to provide official evaluation of papers uploaded on arXiv exists already (it is called “endorsement”). It is not widely adopted yet, but I can certainly see it becoming more popular in the future. One could think of additional ways of providing evaluations, but ultimately I would just count things like number of hits, downloads, and citations.

  2. Anonymous Says:

    Something like pre-publication public-vetting of manuscripts on, for example, a science blog, could work – or it could be a nightmare for the institution of science. Post-publication, people should debate to their heart’s content, write poetry about chemical reactions, I don’t care.

    Anyway, I would start very slowly, taking more of an evolutionary approach to changing the peer-review system. One scheme would be to initially retain formal referees as the final arbiters, but invite semi-public review of all submitted manuscripts. The manuscript would be posted in a place accessible only to those published in the given field, who could post comments visible only to the formal referees (i.e. not to the author, nor to eachother). Obviously more lively review could occur in a less official capacity elsewhere, as it already does anyway thanks to things like

    We have the unfortunate emotional tendency to amplify the weight of an argument simply because it is voiced by more people (“Me too, I agree, great point, cosigned…”) – not on its own scientific merits. We don’t decide the value of an integral by majority vote (and it bothers me tremendously to see people – educated people who should know better – taking the mean value of quantities in blatant mutual disagreement and calling the result a “best estimate”). I would resist incorporating direct communication between the informal reviewers for publication purposes simply to minimize (but obviously not eliminate) this effect of follow-the-leader syndrome (herd-like mentality, bandwagon effect, whatever you want to call it), as well as grandstanding and editorializing.

    I submit that if one’s problem with the status quo is that too much authority is placed in the hands of too few (i.e. the 2-5 formal referees), then this does nothing to alleviate this concern. Addressing this problem necessarily involves some unfortunate trade-offs, and I would need to invest much more thought to endorse any wholesale revision along such lines. Maybe some group could simply start a real open-review journal (in the same spirit as Naboj) and see how it goes?

    Debate from the lay community will hopefully never have much weight in the review process. This may not be a big deal with theoretical physics (where the greatest threat would be annoyance from sci-fi enthusiasts), but other disciplines (those with a more direct connection to public policy especially) would suffer under the weight of science-bereft criticisms (social implications, questions about interpretation by policy analysts, machinations of various activist/corporate groups). Those afflicted with fanatical (i.e. ignorant) convictions would lobby (yes, lobby – moreso than they already do) for bad science to bring about “good” policy, and my doubts are beyond serious that the learned would have enough integrity and courage to speak truth to the mob in such a direct venue (especially when doing so would paint them as villains, lackeys, etc.). Let’s not suppress science with political correctness – we can let the mushy-brains debate poetry and burn effigies after publication.

    • Massimo (formerly known as Okham) Says:

      Anyway, I would start very slowly, taking more of an evolutionary approach to changing the peer-review system. One scheme would be to initially retain formal referees as the final arbiters, but invite semi-public review of all submitted manuscripts.

      You know, I have the feeling that, while many of us will be spending a significant amount of time thinking of, and debating how best to re-organize scientific publishing in light of the emergence of new media, meanwhile things will … just happen.
      Fundamentally I think that the problem will always be that of disentangling the short-term, high-frequency component of the reaction to an article, dominated by things like political connections, fads and other non-scientific elements, to the long-term one, which tends to be determined by the actual value and impact of the work.

  3. Anonymous Says:

    Blogs are also a great place for games… like TAG!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: