IF versus EF

I have only recently become aware of the existence of the Eigenfactor (EF). It is a proposed measure of the overall influence, impact, prestige of a scholarly journal in its own discipline, or field. The one and only measure with which I was familiar is the well-known Impact Factor (IF), which is actually fairly straightforward to understand. By contrast, the eigenfactor is determined through a rather complex procedure (I am not going to discuss its computation in this post — for details, see here).

IF and EF aim at measuring the same thing. Quoting from the relevant Wikipedia entries:
The impact factor … is frequently used as a proxy for the relative importance of a journal within its field…
The Eigenfactor … is a rating of the total importance of a scientific journal…
So, it seems fair to say that if there is a difference in scope, we are mostly talking nuances.
Since they both purport to measure generically “importance”, the obvious question is whether these two indices agree with one another. In other words, given any two science publications in the same field, are the EF and IF generally consistent, when it comes to assessing the relative importance of one versus the other ? Exploring this aspect is purpose of this post — not that of offering an opinion on which one of the two is better (of course there may be many more such measures around of which I am unaware) [0].
Identifying quantitative, as objective as possible measures of performance in any field of human activity, is a problem of general interest to me. As a reasonably active scientist, and as a member myself of editorial boards of two journals, the issue of which yardstick best gauges the impact of a journal (something ultimately so relevant to the careers of many of us [1]) acquires substantial, more immediate practical significance. Should we go by IF or EF when evaluating a candidate’s CV for purposes such as a faculty appointment, promotion, prize ?

In Table I, IF and EF are compared for a set of 40 physics journals. Values of the EF can be obtained online directly from Eigenfactor.org, those of the IF (2010) from the home pages of the journals and/or from Wikipedia. Also included for comparison are the corresponding values for a few prestigious, interdisciplinary science publications. The sample seems sufficiently large to justify some statistical analysis.
The Pearson’s correlation coefficient for EF and IF based on this set of data is 0.1, which in practice means that IFs and EFs do not go together — for instance, a high IF does not correspond to a high EF any more often than it does to a low EF. So, at least one of the two does not really measure what it claims.
Table II shows the top six physics journals, as ranked by either EF or IF. Only two journals, Physical Review Letters and Astrophysical Journal, appear at the top of both rankings; all other journals are only regarded as “influential” on going by one of the two scores, not the other.

Which is the right one ? Are they both wrong ?
As I wrote above, I am not going to express an opinion because I do not know enough about the EF.
Aside from the fact that some intelligence has gone into it, one of its alleged properties is that, unlike the IF, which merely counts citations, the EF weighs them differently, depending on where they are made — citations that appear on more influential publications are given a greater weight. Some of the most striking aspects of the EF are the following:

  • The EF does not seem to like review articles, hence the relatively low score of publications such as the Review of Modern Physics.
  • The EF does not like for some reason Nature Physics, which is the physics journal of highest IF (except for Reviews of Moden Physics and Advances in Physics, which publish review articles). If the community went by EF, Nature Physics would be regarded as a fairly low key publication. I would be very interested in knowing what it is about Nature Physics that the EF penalizes, or does not reward. Too few articles published, maybe ?
  • While the IF tends to amplify differences at the very top, the EF does the opposite. For instance, Nature is regarded as roughly five times more influential than Physical Review Letters by IF, but it is only so by roughy 40 percent, based on EF. Here too, if EF were to become the law of the land, I can imagine a huge drop in the number of physics submissions to journals such as Nature, Science, PNAS, given that the payoff in terms of prestige, compared to Physical Review Letters, would no longer be so impressive as it is now, with people mostly looking at IF.

    If we call “influential” the journal that we read often, in which we are most likely to find articles that have direct relevance to our own research activity, then the ranking yielded by EF makes much more sense to me than that based on IF. EF seems to reflect reality as perceived by the vast majority of us who do physics as a profession.
    It may well be that the difference between EF and IF is that one of the two does not really measure “influence” as much as prestige. A physics article published in Nature [2], for example, may bring its authors recognition and accolade from their university administration and a restricted circle of scientists working in the same area, but perhaps will not set a new trend, spark a collective investigative effort within the community, significantly more than one published in, say, Physical Review Letters.

    One thing that seems indisputable, though, is that only one of the two measures will stand the test of time, and emerge as the standard one.

    Notes

    [0] For a discussion, see, for instance, this reference.

    [1] As opposed to leaving it to the idiosyncratic judgment of a small committee, or even of a single individual.

    [2] The comparison between a journal like Nature, which is interdisciplinary, and one like Physical Review Letters, remains ultimately meaningless, at least insofar as one does not know what the contribution of physics articles is to the high IF of Nature.

    Table I

    Journal EF IF (2010)
    1 Advances in Physics 0.012 21.214
    2 Astrophysical Journal 0.487 6.063
    3 Astrophysical Journal Letters 0.017 5.158
    4 Classical and Quantum Gravity 0.044 3.099
    5 Central European Journal of Physics 0.002 0.696
    6 European Journal of Physics 0.003 0.757
    7 European Physical Journal A 0.021 2.592
    8 European Physical Journal B 0.027 1.575
    9 European Physical Journal C 0.035 3.248
    10 European Physical Journal D 0.018 1.513
    11 European Physical Journal E 0.014 2.096
    12 Europhysics Letters 0.089 2.753
    13 International Journal of Modern Physics A 0.014 1.000
    14 International Journal of Modern Physics B 0.009 0.402
    15 International Journal of Modern Physics C 0.004 1.022
    16 International Journal of Modern Physics D 0.010 1.046
    17 International Journal of Modern Physics E 0.004 0.643
    18 Journal of Chemical Physics 0.255 2.920
    19 Journal of Computational Physics 0.057 2.345
    20 Journal of Experimental and Theoretical Physics 0.008 0.892
    21 Journal of Physics A 0.069 1.641
    22 Journal of Physics B 0.033 1.902
    23 Journal of Physics CM 0.138 2.332
    24 Journal of Physics D 0.093 2.109
    25 Journal of Physics G 0.024 1.770
    26 Journal of Low Temperature Physics 0.009 1.403
    27 Journal of Statistical Physics 0.018 1.447
    28 Modern Physics Letters A 0.012 1.075
    29 Modern Physics Letters B 0.004 0.512
    30 Nature Physics 0.152 18.423
    31 New Journal of Physics 0.098 3.849
    32 Physical Review Letters 1.233 7.621
    33 Physical Review A 0.231 2.861
    34 Physical Review B 0.778 3.772
    35 Physical Review C 0.083 3.416
    36 Physical Review D 0.306 4.964
    37 Physical Review E 0.241 2.352
    38 Physics Letters A 0.070 1.963
    39 Physics Letters B 0.140 5.255
    40 Reviews of Modern Physics 0.107 51.695
    Nature 1.735 36.101
    PNAS 1.659 9.771
    Science 1.455 31.364

    Comparison of Eigenfactors and Impact factors of 40 physics journals. Impact factor values are from 2010. Also shown for comparison are the corresponding data for three prestigious, interdisciplinary science journals (entries in red). Back

    Table II

    Rank EF IF (2010)
    1 Physical Review Letters (1.233) Reviews of Modern Physics (51.695)
    2 Physical Review B (0.778) Advances in Physics (21.214)
    3 Astrophysical Journal (0.487) Nature Physics (18.423)
    4 Physical Review D (0.306) Physical Review Letters (7.621)
    5 Journal of Chemical Physics (0.255) Astrophysical Journal (6.063)
    6 Physical Review E (0.241) Astrophysical Journal Letters (5.158)

    Top physics journals by eigenfactor and impact factor. Back

  • Tags: , , ,

    22 Responses to “IF versus EF”

    1. Carl Bergstrom Says:

      Massimo,

      Thank you for the blog post. The reason you observe such dramatic differences between Eigenfactor scores and Impact factor scores is that Eigenfactor and Impact factor DO NOT measure the same thing. Once one recognizes this, your observations become less puzzling.

      We developed the Eigenfactor score as a metric of the total importance of a journal, analogous to total count of citations to that journal over a given period. This metric scales with the size of a journal. All else equal, if a journal doubles the number of articles published annually, its Eigenfactor score should double. Such a metric can be useful for a number of purposes, such as making collections decisions about which journals to purchase with a limited library budget.

      However, the Eigenfactor score is not the proper metric for deciding which journal is the most prestigious venue for publication. Individual authors or researchers interested in deciding where to send their work will be interested in a journal’s influence per published article. This is what the Impact Factor score measures. All else equal, if a journal doubles the number of articles published annually, its impact factor should remain unchanged.

      Another of the Eigenfactor metrics, the Article Influence score, measures per-article influence and thus is more appropriate for comparison with the Impact Factor score. The correlation coefficients between Article Influence and Impact Factor scores are typically in the 0.5-0.99 range, as we describe in Figure 1 of this paper: http://octavia.zoology.washington.edu/people/jevin/Documents/West2010JASIST.pdf

      Given that Eigenfactor scores and Impact Factors scores are different tools that do different jobs, the major differences you observe are not as problematic as it might seem at first glance.

      Best regards,
      Carl Bergstrom
      eigenfactor.org

    2. Massimo Says:

      Hi Carl,

      thank you for the clarification. The reason I became interested in this subject is that it is felt by many of us that the widespread adoption of the impact factor as a measure of the importance of the publication venue, may have a number of unfortunate consequences, perhaps the most serious being that of putting a number of respectable specialized publications at a disadvantage — in some cases to the point where their own very existence is jeopardized, given the pressure on researchers to go for high IF journals. Granted, much of that stems from the fact that we all have to deal with university administrations all too keen on reducing evaluation to a single number, or restricted set thereof.
      So, the notion of possibly replacing the IF with another measure, one giving a more realistic, balanced and fair assessment of the true impact of a publication in a specific area of inquiry, is one that many of us would happily entertain.

    3. Transient Reporter Says:

      Man, how would Italy have fared in the last World Cup with Cassano and Balotelli?
      Balotelli does seem like a real head-case though.

      Onto the next round!

    4. Transient Reporter Says:

      Did you ever see this? As the Italians would say, “vergognoso…”

      • Massimo Says:

        SuperMario is not something I am willing to discuss, to me it’s like religion, it requires a private act of faith…

      • transientreporter Says:

        For a moment there, I thought Pirlo would do a Baggio…
        God, England are awful…

        • Massimo Says:

          Of course… it could not possibly be that Italy are good…

        • transientreporter Says:

          Italy were definitely the better team, no doubt about that. I’ve really come to appreciate Andrea Pirlo who really does anchor that team. But Balotelli missed way too many opportunities and Cassano was invisible. Better wake up for the Germans…

        • Massimo Says:

          OK, listen: true, Mario did miss one great opportunity, though you have to give some credit to both Hart and Terry on that occasion. However, the man constantly had two-three English players marking him, he kept creating favorable situations for us, in which the English defence would be in numerical inferiority, of which Italy did not take advantage, mostly because Cassano was non-existent. I think he played a good game, even with the missing opportunities.

        • transientreporter Says:

          Spain and Italy in the final – Final score 2-1 to the Azurri, Balotelli gets the winner…

        • Massimo Says:

          Nah, that would make Germans really mad, and we can’t afford to be thrown out of the Euro zone…

        • transientreporter Says:

          Euro Zone? What Euro Zone?

        • Massimo Says:

          S-U-P-E-R-M-A-R-I-O !
          F-A-N-T-A-N-T-O-N-I-O !
          That’s all I have to say…

        • transientreporter Says:

          My prediction for the final – De Rossi scores early in the first half from outside the box, ball taking a deflection. Spaniards equalize late in the second on a Pedro header at the near post. Balotelli scores in overtime on a breakaway.

          After the game, Spain defaults, there is a bank run in Italy, the European Central Bank and Angela Merkel says, “Screw you all.” The Euro disintegrates, the EU falls apart, chaos ensues, Balotelli decides to play for the Los Angeles Galaxy.

        • Massimo Says:

          Sorry, what is “bold” about this prediction ? You are just reading the newspapers, man…

        • transientreporter Says:

          Here’s a bold prediction then: Fernando Torres will do something interesting with the ball.

        • Massimo Says:

          I see a bright future for you as a financial commentator.

        • transientreporter Says:

          Well, that was a bit anti-climactic…

        • Massimo Says:

          Well, the score seems harsh, I thought a two goal difference was an accurate and fair reflection of a difference which is there and is undeniable. Luck wasn’t on our side either as we suffered two major injuries, but the substance is that Spain is the number one now…

        • transientreporter Says:

          I thought the first goal was fabulous, but you wonder if Chiellini would’ve cut off the run that led to the second goal. I would’ve liked to have seen more of Thiago Motta who is an exciting player.

          I’m sure this will get examined to death by the Italian press, but should you be making three substitutions that early in the game? Shouldn’t you at least hold one back for a little while just to see how the second half is developing? Cassano’s injury certainly had important consequences in that respect, because it ties up a substitution right away. But better one half of Cassano than none, I guess…

        • Massimo Says:

          Look, the Italian squad was outplayed in just about every respect. Yes, we were also unlucky and yes, maybe Prandelli could have made different choices, so that the score could have been kept down to a more reasonable level. Does it make all that much difference ? To me, no. 4-0 is bad, but it isn’t Italy’s fault if they were the team that made it to the final…

        • transientreporter Says:

          My final thoughts…
          1) It’s amazing that Spain could be that good without two of their best players (Puyol and David Villa). That’s a deep team…
          2) Trust Torres to start playing once the game is over – typical.
          3) Italy have to feel pretty good about their chances in the World Cup – with Balotelli, a healthy Cassano, Motta, and a final campaign for Pirlo and Buffon. That’s a pretty damn good team right there.
          4) Balotelli needs to get over himself. Coming out after everyone else has lined up for the national anthems, not shaking hands after the game… Dude needs to grow up. Sometimes I wonder if he plays for the nation of Italy or for the Republic of Mario Balotelli.

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s


    Follow

    Get every new post delivered to your Inbox.

    Join 28 other followers

    %d bloggers like this: