Read vs cited

Which scientific article has a greater impact, the one that is more frequently cited, or the one that is more frequently read ? The answer is far from trivial, but it seems reasonable that both measures should give an indication of the influence of a published piece of scholarly work. At the present time, however, that importance is quantitatively assessed exclusively through the number of citations that the article garners over time.
This is because a) determining how many people actually read an article was essentially impossible a proposition until not so many years ago, whereas the number of citations is (relatively) easily countable b) the underlying assumption is that a scientist citing a published article has presumably read it, an assumption often questionable, as argued in a recent study.

Do the two things, namely reading and citing, necessarily go together ? Maybe, or maybe not. There is no question, however, that measuring the readership of an article seems much more feasible nowadays, as essentially all scientific journals, as well as ArXiv, maintain online repositories of articles and keep track of things such as article hits and downloads. While these two numbers still do not necessarily measure directly readership [0], they should reasonably be expected to correlate fairly closely with it. In other words, it is another measure which, while imperfect, ought to have some usefulness.

This immediately raises the question of correlation between readership, as assessed through the procedure suggested above, and number of citations. If, hypothetically, the two things did not go together, i.e., many articles that are widely cited are not equally extensively read, and vice versa articles that are downloaded and read by many end up cited only seldom for a variety of possible (legitimate and illegitimate) reasons, then which papers should be deemed to have had the greater impact ? [1]
And, along the same lines, is the journal that is most prestigious the one that is most widely read, or rather that with the most cited articles, should the two things not necessarily go together ?

[0] I for one frequently download the same article several times, just because I have the tendency of purging files ruthlessly, thereby deleting material which I should keep instead.

[1] For example, let us consider the case of a novel idea that first appears in a manuscript that is extensively cited, but is illustrated more accessibly and clearly on a subsequent manuscript (perhaps by other authors), not equally cited but from which most scientist end up learning about that idea.

7 Responses to “Read vs cited”

  1. Mad Hatter Says:

    Aside from the issue of how to reliably measure readership (I also often download a paper multiple times, and probably even more often download papers I don’t end up reading at all), my opinion is papers that are more frequently cited should be considered to have higher impact.

    Due to space/length restrictions, I can usually only cite a subset of the papers I actually read on the topic at hand. This forces me to pick the ones that contain the seminal findings and that most clearly demonstrate the finding I am discussing. These are usually the papers that represent the greatest advancements in that particular subfield (although not always the ones that ended up in high-profile journals). In cases such as what you describe in Note [1], I usually cite both articles.

    Determining impact by citation also gets us around the problem of papers which sound like they are reporting a really cool finding (based on title and abstract), but have shaky supporting data, missing controls, flaws in logic, etc. People would read such a paper not knowing it was a crappy paper, but would decide not to cite it because the results are less than believable. Using readership would give undeserved “points” to the papers I read that make me think, “Who were the idiot reviewers who let that get by?”

    • Cath@VWXYNot? Says:

      I was just about to post the exact same comment as outlined in your last paragraph. If a paper is frequently read but cited only very rarely, there is probably something wrong with it…

  2. Schlupp Says:

    As far as it concerns ‘conventional’ papers, I’m with Mad Hatter. Any snide piece on the archive, preferably containing a few accusations or at least insinuations, can be sure of an eager readership, but there’s good reason it’s not going to be cited. Admittedly, there is also a good chance that it’s never going to be published in a refereed journal, but Mad Hatters point remains valid if we restrict the discussion to journal papers.

    I do think readership means something when it comes to stuff that is not usually captured in publication metrics, like lecture notes that someone might put on the web.

    Oh, and lest I forget: Since this comment contains the word “refereed”, there is a chance that you will refuse to accept it claiming it does not fulfill your blog’s stringent standards. To help acceptance, I would like to point out that my comment does not contain any “hard facts, precisely stated and substantiated.” Additionally, let me see…. need a few four letter words… “free” electron theory, “wave” function, “work” and “heat”, “body”- as well as “face”-centered. That should do it, I hope.

  3. Massimo (formerly known as Okham) Says:

    MH — In general I agree with what you say. In my field, however, citations often occur very much like argued in the reference that I have in my post — largely by some non-linear, often accidental mechanism whereby the first to get cited is also the one that that continues to get cited, pretty much without being read by anyone.

    Schlupp — I doubt if in the example that you mention the number of download would be so large. I suspect that it would be a short-term effect only, and then people would get bored and move on. But there are some papers that we, our postdocs, our students read, and their future students and postdocs will likely also read. In some cases these papers are not cited as much as the original reference, but I think the authors do deserve some recognition.

    • Mad Hatter Says:

      Massimo–I don’t understand how one can cite papers one hasn’t read. How would you know the paper actually says what you’re citing it for? Your comment implies that once Scientist 1 cites Paper A for finding X, then everyone else follows suit regardless of whether Paper A is actually the appropriate one to cite for finding X. If everyone is simply going to take Scientist 1’s word for it that Paper A reports X, doesn’t that sort of defeat the purpose of citing references to begin with? And if there is a Paper B on finding X that everyone reads and finds informative, why would they not cite Paper B simply because some random Scientist 1 chose to cite Paper A instead? Seems to me the problem then lies more with the citation habits of scientists rather than with the credit attribution system.

      • Massimo (formerly known as Okham) Says:

        I don’t understand how one can cite papers one hasn’t read.

        MH — Ideally you would read all the papers that you cite, but in practice this proves a difficult proposition, mostly due to lack of time. Thus, often times one simply cites the paper(s) that everyone else cites, without really doing any research to establish if those are indeed the papers that should be cited. That is precisely how the non-linear phenomenon discussed in the paper referenced to in my post occurs.

        For example, you are writing the introduction, and you want to cite a few seminal papers. You take a published paper on the same subject and you cite the same articles that they cite, regardless of whether their citations are fair or not.
        Or, you want to cite a known experimental or theoretical result — again, you look at a published paper written by others quoting the same result and you use the same citation, trusting that is it accurate and fair. Is this the right thing to do ? Of course not, in principle I should check every reference for priority. Do I have the time to do that ? No, I am sorry.

      • Cath@VWXYNot? Says:

        I think this happens more often than anyone would like to admit.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: