Yield sequences as journal attractivity indicators: “payback times” for Science and Nature
The Authors
Liming Liang, University of Antwerp (UA), IBW, Wilrijk, Belgium and KHBO (Association K.U. Leuven, IWT, Oostende, Belgium
Ronald Rousseau, University of Antwerp (UA), IBW, Wilrijk, Belgium and KHBO (Association K.U. Leuven, IWT, Oostende, Belgium
Acknowledgements
The authors would very much like to show their appreciation to Zhong Zhen, Xu Chaofeng and other persons for collecting and checking the accuracy of the SCI data. The authors' colleague Shi Fei from Thomson Scientific China (Beijing) is especially thanked for joining the discussion on the data collection. Finally, the authors thank Gwendolyn Rogge (KHBO, IWT) for carefully reading the manuscript, and Q. Burrell for some useful suggestions. The work presented in this paper was supported by the National Natural Science Foundation of China by grant no. 70673019.
Abstract
Purpose – The yield period of a journal is defined as the time needed to accumulate the same number of citations as the number of references included during the period of study. Yield sequences are proposed as journal attractivity indicators describing dynamic characteristics of a journal. This paper aims to investigate their use.
Design/methodology/approach – As a case study the yield sequences of the journals Nature and Science from 1955 onward are determined. Similarities and dissimilarities between these sequences are discussed and factors affecting yield periods are determined.
Findings – The study finds that yield sequences make dynamic aspects of a journal visible, as reflected through citations. Exceptional circumstances (here the publication of Laemmli's paper in 1970 in the journal Nature) become clearly visible. The average number of references per article, the citation distribution and the size of the database used to collect citations are factors influencing yield sequences.
Originality/value – A new dynamic indicator for the study of journals is introduced.
Article Type:
Research paper
Keyword(s):
Reference services; Electronic journals; User studies; Case studies.
Journal:
Journal of Documentation
Volume:
64
Number:
2
Year:
2008
pp:
229-245
Copyright ©
Emerald Group Publishing Limited
ISSN:
0022-0418
Introduction: the publication-citation system as an input-output system
Research can be considered as an input-output system. In this article we will concentrate on the act of writing and publishing a research article. As stated by Tijssen et al. (2001) citations from one research paper to another are more indicative of a paper's direct scientific impact, i.e. its actual influence on research activities, than of its scientific importance or quality. It is this aspect of direct impact that will play an essential role in our approach.
Let us consider the items included in the reference list of article A as inputs to the scientific process leading to the generation of this article. The contents of these cited articles can be regarded as an intellectual gift from contemporaries or earlier generations of scientists. One could even say that this content constitutes an “intelligent loan”, because science, as a system, expects rewards for or interests on this loan. Uses of the new article A, codified as citations, or more specifically, the ideas and research results published in it, can then be considered as outputs. Summarising, the following process occurs: research results, codified as articles and acknowledged as references, are inputs for a new article. Outputs consist of the citations that this new article receives over time. The whole idea is illustrated in Figure 1. Note that inputs are fixed, but that outputs grow dynamically over time. It is this dynamic aspect that will lead to the new indicator. Instead of one article it is also possible to consider a larger item set, consisting of a related group of articles. This is why we write item set, and not article, in Figure 1. Examples of such item sets are for instance all articles published in one particular journal or serial, by one institute, or even one country, during a given time period. Of course considering an item set consisting of a single article is not excluded.
The middle part of Figure 1 is considered as a black box. Besides tangible inputs such as articles published in the open literature, this black box also receives input from researchers' accumulated knowledge, skills and creative ideas. These aspects are not taken into account in our approach. Note that the input-output idea is essential in research performance studies applying Data Envelopment Analysis (DEA) (Johnes and Johnes, 1992; Rousseau and Rousseau, 1997, 1998; Arcelus and Coleman, 1997). These ideas can be placed in a framework focussing on similarities between informetrics and econometrics (Liang, 1995; Rousseau, 1994).
The basic idea of the new indicator was put forward in the second part of a presentation made at the 10th International Conference of the International Society for Scientometrics and Informetrics, Stockholm, Sweden (Liang et al., 2005).
A new indicator
In earlier publications we introduced a set of indicators aimed at reflecting the internal rhythm of actors in scientific production processes (Liang, 2005; Liang et al., 2005, 2006). These indicators were referred to as R-indicators, from which we constructed several clusters of related forms. In this study we will introduce another indicator describing dynamic aspects of a serial's time series. It is created based on the input-output analogy explained above.
The time needed to accumulate the same number of citations as the number of references in the item set is defined as a yield period. Articles published in different years may influence fellow scientists in different ways. Therefore yield periods for a journal, institute, or country will fluctuate. Our new indicator (defined more precisely in the next section) can be considered as a kind of time-dependent indicator of the speed with which rewards are reaped from the social-cognitive input of the scientific system into the item set. As such it also reflects the attractivity of this item set. From an economic point of view the reference set may be considered as an investment, and the citations as a form of return on investment. In this metaphor, our new indicator corresponds to the payback time, namely the time needed to repay the system's investment. The highly visible journals Nature and Science were selected for a case study. As these journals are highly visible, they are also highly cited, making it possible to determine more than one yield period.
Methodology
The calculation of yield indicators is easy to explain. Consider a journal J as an example. Denote the total number of references cited in its articles in year i as L i . By A ih we denote the cumulative number of citations received by the items published in year i from the publication year i to (and including) year h: A ih =∑ j=i h C ij , where C ij denotes the number of citations received in year j by the items published in year i. Comparing L i with all the A ih, we may find a time t, so that, A i,i+t ≤L i <A i,i+t+1 . Then, for publication year i, the time needed to accumulate a number of citations equal to L i is denoted as T 1i , and is called the first yield period of journal J for the publication year i. The first yield period, T 1i , is determined as: T 1i =t+ (L i −A i,i+t )/(A i,i+t+1−A i,i+t ). This formula uses linear interpolation, assuming – as an approximation – a uniform distribution of citations over one year. Assuming that we study journal J over a time period P of length n (years), all the T 1i (i=1 to n) form a time series T 1= (T 1i ) i=1,…n of yield periods for this journal. Data needed for the calculation of the yield indicator are shown in Table I.
Similarly, we may define a sequence (S k ) k where each S k is a time series. The first sequence S 1 is equal to T 1, and S 2 , S 3, … are the sequences where the yield (number of citations received) is twice, thrice, … that of the number of references. We refer to S 1 , S 2 , S 3, … as the first, second, third, etc. cumulative yield sequences. For every k and i, S ki , is defined through the following requirement: Equation 1 It is also of interest to study the sequences (T k ) k where T 1 =S 1 and the components T ki of sequence T k are equal to T ki =S ki −S k−1,i .
Let us recall that index i refers to a publication year. The symbol T ki denotes the time (expressed in years) required for documents published in year i, to grow from a total number of citations equal to (k−1)L i to kL i . The sequences T k are called yield sequences.
If the total number of citations received by journal J since year i is equal to TOT, then two other concepts can be defined. The first one is the number of yield sequences that can be determined. If this number is denoted as k max(i) then k max (i)=TOT/⌊L i ⌋, where ⌊x⌋ is the floor function of x: this is the largest integer smaller than or equal to x. Some examples: ⌊2.718⌋=2;⌊3⌋=3. Similarly, one may simply define N(i)=TOT/L i , which is usually not an integer.
A simple theoretical result
What can yield sequences look like? Let us assume that the distribution of received citations can be described by a Weibull function (Burrell, 2002; Börner et al., 2004). Note that the Weibull distribution is a continuous distribution used here as an approximation for the real distribution, which is, of course, a discrete one. Then the cumulative number of citations is given as F(t)=TOT.(1−e −(t/) b ), where TOT is the total number of citations received since publication, (> 0) is the scale parameter and b (>0) the shape parameter. If we further assume that the number of references is L, then S k is determined through the relation: F(S k )=kL. From this equation we find TOT(1−e −(S k /) b )=kL,as long as kL≤TOT. A simple calculation then leads to: Equation 2
In order to study how S k depends on k we will study the function y b =(−ln(1−x))1/b , for 0≤x<1 (see Figure 2). For all values of b, y b passes through the origin and through the point with coordinates(1−e −1,1). Clearly y is always increasing. For 0<b≤1 the curve is convex. This means that the (T k ) k are increasing: it is taking more and more time to reach the next L citations. Note that b=1 is the case of an exponentially decreasing citation curve. If b>1, then the curves y b are concave on the interval [1,1−e 1−b/b ], and convex on the interval [1−e 1−b/b ,1]. In particular, for x>1−e −1, the function is always convex. This means that in case b>1 the T k are first decreasing and later increasing. Initially it takes less and less time to reach the next L citations, later it takes more and more time. This corresponds intuitively to a citation distribution curve that first increases and then decreases, a quite natural situation. Note though that this analysis is performed using continuous variables. In reality, depending on the exact values of L, TOT and b, it might be that the T k are always increasing (if the initial, increasing part of the citation curve is too short).
Yield sequences of Science and Nature: similarities and dissimilarities
All publication and citation data used to study the yield sequences of Science and Nature are retrieved from ISI's Web of Science. We explored the period from 1955 to 2003, a total of 49 years. In this period Science and Nature published many kinds of documents: articles, letters, book reviews, editorials, etc. For Science the total number of published documents over this period of 49 years amounts to 95,733. For Nature, the corresponding data are 157,616. We observe that the mix of different document types changes annually, and in particular, that the proportion of “normal” articles changes from year to year. As “normal” articles, i.e. publications classified as article by ISI, usually receive much more citations than letters or other documents, we are using only “normal” research articles in our investigation. Thus, our sample set of Science consists of 45,415 articles. These articles contain a total of 1,048,423 references, or an average of 23.1 per article. After their publication in Science they attracted a total of 4,128,681 citations over the period 1955-2003. Let us recall that the 1,048,423 references are considered to be an intellectual input of scientists in the journal Science. Note further that these 1,048,423 references are of course not all different articles: many of them were used several times. The sample set of Nature consists of 65,687 articles, containing a total of 1,058,419 references, or an average of 16.1 per article. Over the period 1955-2003 these 65,687 articles received a total of 4,951,022 citations. Clearly, Nature's average number of references per article is much smaller than that of Science. This is one of the reasons (to be discussed later) causing dissimilarities between the yield sequences of the two journals. Finally, we have calculated all the P i , L i and C ij (i=1, 2, … , 49; j=i, i+1, … , 49) for Nature as well as for Science. These numbers are the basic numerical data for the derivation of their yield sequences. Lengths of all the yield periods T i for Nature and Science are listed in Tables II and III. The average number of references per article (AR for short) and N(i) for each of the 49 years are also displayed in Tables II and III.
The T-sequences of Tables II and III show how Nature and Science draw knowledge from the scientific community on the one hand, and contribute knowledge to society on the other hand, by being used (through citations they receive) by the scientific community.
We will now discuss the main characteristics of the data in Tables II and III. There clearly is one outlier in the profiles of yield periods, namely Nature 1970 which is the only year having ten yield periods. The reason for this is Laemmli's (1970) paper published that year, which has been cited nearly 200,000 times since its publication, and is the second most-cited article ever in the Web of Science (the most-cited one being the famous Lowry et al. (1951) article).
The profile of yield periods further shows that the time span from 1987 to 1995 is the period with the largest values of k max. This observation holds for Nature as well as for Science. In this time span Nature has eight years with seven yield periods, while for Science eight of the nine years have five yield periods. Generally yield periods, T j with j fixed, get shorter and shorter over time. Figures 3 and 4 show this decreasing trend, though there are, sometimes heavy, fluctuations in the earlier years.
In general, for Nature as well as for Science, the second yield period T 2 is the shortest one among all yield periods. Figures 5 and 6 compare the lengths of T 1, T 2 and T 3 of Nature and Science since 1985. We see that for Science, the yield periods T 1, T 2 and T 3 are larger than one in all years, while for Nature, in some years, T 2 and T 3 are even smaller than one. We further observe that since 1985 the difference between the T 1 and T 2 sequences (Nature and Science), has been close to one. Before 1978 Nature and Science's N(i)s behave irregularly. Since 1978, Nature's N(i)s have been larger than Science's each year, forming two clearly distinguishable levels, see Figure 7.
Three factors affecting yield periods
In our exploration we see three main factors affecting the lengths of yield periods and the number of existing yield periods of a given publication year. These three factors are the average number of references per article, the distribution of citations received by articles published in a certain year, and the size of the database from which we collect the citation data.
Factor 1: the average number of references per article (AR)
The average number of references per article, i.e. AR in a certain publication year, clearly is a factor affecting a journal's yield periods. AR can be considered as the average intelligent input for producing the articles published in that year. The study of yield periods and yield series analyses how long a journal needs to pay back the intelligent loan in the form of citations received by articles after their publication, and how many times this “intelligent input” has been paid back.
Generally speaking, when citation distributions of two journals are similar in a certain year (see Figure 8), the one with the larger AR is expected to have the larger yield periods and the shorter N(i)s. We see there that the citation curves of Nature and Science for the publication year 1990 are practically the same. From Tables II and III, however, we learn that Nature's T i s are always shorter than the corresponding T i s for Science. Finally Science's N(1990) is 5.32, while Nature's is 7.76, 46 per cent larger than Science's. The reason for this difference is that in 1990, Nature's AR is 25.05, while Science's AR is 35.85, which is 43 per cent larger than Nature's.
From Tables II and III we learn that in all 49 years except for 1965, 1968, 1969 and 2003 Science's AR are larger than Nature's AR. Figure 7 shows that since 1978, Nature' and Science's N(i)s have formed two distinct levels. These two observations lead us to the following experiment. What would happen if we gave Nature Science's AR, and Science Nature's AR? Table IV, and Figures 9 and 10 show the result. Instead of having two distinct levels as in Figure 7, Figures 9 and 10 show that the predominance of Nature has disappeared, as has the inferior position of Science. In some years Nature's N(i) is larger than Science's, in other years the opposite is true. A very interesting phenomenon happens when choosing the same AR, Science's or Nature's AR. Which one does not matter. Then we see that the sum of Science's N(i)s is almost equal to Nature's (see bottom row of Table IV).
Factor 2: distribution of citations received by articles published in different years
When two journals have the same average number of references (AR) in a publication year, this means that they receive a similar amount of intelligent input. If for that publication year they also have similar citation distributions, as is the case for the citation curves shown in Figure 8, the two journals must have similar T i (for each i) and N(i)s. If in one year two journals have a similar AR, but different citation curves, their T i and N(i)s will probably not be similar. Here is such an example. In 1964 Nature's AR was 13.88, whereas Science's AR was 14.01 – just 1 percent larger than that of Nature. In Figure 11 Science's citation curve is situated above Nature's. Consequently it is no surprise to see that for 1964, Science's T i are shorter than Nature's and Science's N(i) is larger than Nature's, see also Tables II and III.
Factor 3: the size of the database used to retrieve the citation data
We have mentioned that in this study all publication and citation data used to explore the yield sequence of Nature and Science are retrieved from ISI's Web of Science, including SCI, SSCI and A&HCI. The time span we explored is 1955-2003, covering 49 years. During this long period the sizes of these databases have been increasing. It is easy to understand that the larger the database size, the larger the potential for citations. Therefore, database size influences the number of citations received by articles, and hence also the length of yield periods. One could say that this is “unfair” to the earlier publication years. A smaller database automatically makes the yield period longer. This phenomenon occurred in Figures 3 and 4.
How can we eliminate the influence of database size in a yield sequence study? A normalisation accounting for database sizes in different years is a valid method. Since Nature and Science basically are natural science journals the majority of their citations comes from source journals of the SCI. Consequently we only use the SCI as an example to show this normalisation procedure.
In the methodology section we stated that C ij denotes the number of citations received in year j by the items published in year i(i=1, 2, … , n; j=1, 2, … , n). All the C ij form a citation matrix. Let us define the size of the SCI as the number of documents indexed by SCI and let us denote SCI's size in year j by Z j . Then let us divide each C ij by Z j and denote the sum of all C ij by S, the sum of C ij /Z j by S n, and calculate S/S n . Then, (C ij /Z j )*(S/S n ) are the elements of the normalised citation matrix, denoted as M D . The T-sequence calculated based on M D is called the T D -sequence.
Usually, articles receive citations mainly from articles. In the study of Nature's and Science's T-sequences, we may also take another normalised measure. Let us define the size of the SCI now as the number of all “normal” articles indexed by SCI. Then one may also normalise with respect to the number of articles in the database. This new normalised citation matrix, is denoted as M A . The T-sequence calculated based on M A is called the T A -sequence.
We know that the size of the SCI increases approximately linearly (Liang et al. 2006). Therefore, theoretically, when a normalised measure is used, one expects smaller yield periods and longer N(i)s for the earlier years. The actual calculations of the T D - and T A -sequences, and N(i)s, confirm this. Figures 12 and 13 show the T D1 and T D2 curves of Nature and Science. Recall that the corresponding curves in Figures 3 and 4 have a totally different pattern.
Denoting the N(i)s calculated based on M D as N D (i), and those based on M A as N A (i), we found no obvious difference between Nature's N D (i) and N A (i). Only in the earlier time the N D (i) are a little shorter than the N A (i)s. The reason is that in the earlier time the proportion of articles compared with all documents covered by SCI is lower than that at a later time. The same situation occurs in Science: there is no obvious difference between N D (i) and N A (i), and the earlier N D (i)s are a little shorter than the N A (i). A comparison between Nature's and Science's N D (i) shows however that there still exist two levels: since 1975 Nature's N D (i)s have always been longer than Science's (see Figure 14).
Summary, conclusion and discussion
The T indicator studied in this article is a simple indicator, trying to reflect part of the “rhythm of science”. T describes an input-output relationship for knowledge production and as such it reflects the attractivity of the item set under study. The R-sequence, studied in earlier articles, and the T-sequences, studied here, can be used to describe the evolutionary rhythm of science, or elements in a scientific production process, from two different points of view. We believe that if it were feasible to obtain the required data, it would become possible to demonstrate how science evolves in a field, a country and even in the world as whole. We are aware though of the limitations of our methodology for measuring the global rhythm of science. The main limitation of our approach lies in the data collection. Even a database such as the Web of Science can never cover all publications and citations. Therefore, we can never obtain all citation data for an article, or an item set in general. Of course, this caveat also applies to most other publication and citations studies based on the Web of Science. Maybe however, in the near future, the evolution of the Internet and its search engines will reduce this limitation. (Note though that we do not write “the open, freely available Internet”, nor do we claim that this limitation will disappear, just “be reduced”.)
We can conclude from our investigations, though, that yield sequences make dynamic aspects of a journal visible. The term dynamic aspects refers here to the relation of the length of reference lists, and the amount of citations received. Exceptional circumstances (in our case: the publication of Laemmli's paper in 1970 in the journal Nature) become clearly visible.
We have further studied factors affecting the yield indicators: the average number of references per article, the citation distribution and the database used to collect citations. We could ask, however, whether these three factors are independent. Does the AR influences citation curves? Or, is it true that the bigger AR, the more the journal is cited? The normalisation for database size just considered the quantity of source journals in the database. One may wonder though if enlarging the database does not influence the overall “quality” of source journals, and hence the shape of the citation situation. That would mean that the factors “database size” and “citation distribution” are not independent either.
Finally, we would like to mention the following question. We have observed that the yield period of Science is getting shorter and shorter. Is there a minimum value for a yield period? If there is, what is it? To answer this question, the citing behaviour of authors, the publication period of the citing and cited journals, and the emergence of more and more electronic journals should be taken into account. In the end the question becomes: is it possible to receive L citations in one day, one hour, one minute? The answer is “yes' if L=1, but what about a more realistic situation?
Another question is whether this indicator is field-dependent? Or are fields in which journals have on average a long reference list also those fields where articles are cited more rapidly? The short table published by Moed and Garfield (2003) seems to indicate this, as articles in high-impact fields such as Molecular Biology and Biochemistry have on average a much longer reference list than articles in the field of Engineering. The answer to these questions is left for future research.
Equation 1
Equation 2
Figure 1Dynamic input-output process leading to the yield indicator
Figure 2The function y
b
=(−ln(1−x))1/b, for 0≤x<1
Figure 3Comparison of Nature's and Science's T
1 sequences
Figure 4Comparison of Nature's and Science's T
2 sequences
Figure 5
Nature's recent yield periods: T-sequences since 1985
Figure 6
Science's recent yield periods: T-sequences since 1985
Figure 7Comparison of N(i)s between Nature and Science
Figure 8
Nature and Science: curves of average citations per article published in 1990
Figure 9
Nature and Science: N(i) calculated based on Nature's AR (1978-2003)
Figure 10
Nature and Science: N(i) calculated based on Science's AR (1978-2003)
Figure 11
Nature and Science: curves of average citations per article published in 1964
Figure 12Comparison of T
D1 between Nature and Science
Figure 13Comparison of T
D2 between Nature and Science
Figure 14Comparison of N
D
(i)s between Nature and Science
Table IData needed for the calculation of journal J's yield indicator for the year i
Table IIYield sequences of Nature
Table IIIYield sequences of Science
Table IV
Nature and Science: N(i) recalculated based on different AR
References
Arcelus, F.J., Coleman, D.F. (1997), "An efficiency review of university departments", International Journal of Systems Science, Vol. 28 No.7, pp.721-9.
Börner, K., Maru, J.T., Goldstone, R.L. (2004), "The simultaneous evolution of author and paper networks", Proceedings of the National Academy of Sciences, Vol. 101 No.suppl. 1, pp.5266-73.
Burrell, Q.L. (2002), "Modelling citation age data: simple graphical methods from reliability theory", Scientometrics, Vol. 55 No.2, pp.273-85.
Johnes, G., Johnes, J. (1992), "Apples and oranges: the aggregation problem in publication analysis", Scientometrics, Vol. 25 No.2, pp.353-65.
Laemmli, U.K. (1970), "Cleavage of structural proteins during the assembly of the head of bacteriophage T4", Nature, Vol. 227 No.5259, pp.680-5.
Liang, L. (1995), Scientometrics: Indicators, Models, Applications, Science Press, Beijing, (in Chinese), .
Liang, L. (2005), "The R-sequence: a relative indicator for the rhythm of science", Journal of the American Society for Information Science and Technology, Vol. 56 pp.1045-9.
Liang, L., Rousseau, R., Shi, F. (2005), "The rhythm of science, the rhythm of Science", in Ingwersen, P., Larsen, B. (Eds),Proceedings of ISSI 2005, Karolinska University Press, Stockholm, pp.398-405.
Liang, L., Rousseau, R., Shi, F. (2006), "A rhythm indicator for science and the rhythm of science", Scientometrics, Vol. 68 No.3, pp.535-44.
Lowry, O.H., Rosebrough, N.J., Farr, A.L., Randall, R. (1951), "Protein measurement with the folin phenol reagent", Journal of Biological Chemistry, Vol. 193 pp.265-75.
Moed, H.F., Garfield, E. (2003), "Basic scientists cite proportionally fewer authoritative references as their bibliographies become shorter", in Jiang, G.H., Rousseau, R., Wu, Y.S. (Eds),Proceedings of the 9th International Conference on Scientometrics and Informetrics, Dalian University of Technology Press, Dalian, pp.190-6.
Rousseau, R. (1994), "Similarities between informetrics and econometrics", Scientometrics, Vol. 30 No.2-3, pp.385-7.
Rousseau, S., Rousseau, R. (1997), "Data envelopment analysis as a tool for constructing scientometric indicators", Scientometrics, Vol. 40 No.1, pp.45-56.
Rousseau, S., Rousseau, R. (1998), "The scientific wealth of European nations: taking effectiveness into account", Scientometrics, Vol. 42 No.1, pp.75-87.
Tijssen, R.J.W., Visser, M.S., van Leeuwen, T.N. (2001), "Searching for scientific excellence: scientometric measurements and citation analyses of national research systems", in Davis, M., Wilson, C.S. (Eds),Proceedings of the 8th International Conference on Scientometrics & Informetrics, BIRG, Sydney, pp.675-89.
Corresponding author
Ronald Rousseau can be contacted at: ronald.rousseau@khbo.be