Usage Impact Factor: the effects of sample characteristics on usage

Usage Impact Factor: the effects of sample characteristics on usage

arXiv:cs/0610154v1 [cs.DL] 26 Oct 2006 Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics. Johan Bollen⋆† and H...

2MB Sizes 0 Downloads 6 Views

Recommend Documents

Adolescents
Today with more than 1.5 billion Internet users around the world (Jaffe. A. & Chen. A., 2010), the Internet has become a

Preferred Stock: Usage and Characteristics - Clute Institute
Preferred Stock: Usage and Characteristics. Dr. Bill D. Cox, Accounting, University of Northern Colorado. Abstract. This

The Final Countdown: On the Historiography of the Usage of
A Historiographical Analysis on Language in the Year 1000 A.D. ... Richard Landes presents in Giants With Feet of Clay:

Characteristics of Foam Concrete with usage of Foam Agent which
Abstract— Technology and innovation making foam concrete started to widely used on industrial building construction, e

Pronoun Usage
Although MAN in its original sense carried the dual meaning of adult human and adult male, its meaning has come to be so

Trademark Usage
marks. This will reduce the chance of the mark being seen as a generic term. Correct use:After you install the Win- dows

MCWT Usage of lynda.com
MCWT Usage of lynda.com. MCWT lynda.com Scheduling Information.doc. Page 1 of 3. Revised: February 28, 2012. General Inf

The Impact of Landline and Cell Phone Usage - RTI International
May 16, 2008 - phones and are difficult to reach by landline phones. ... this paper examines patterns of landline and ce

Usage Table
I shall see him, but not now: I shall behold him, but not nigh: there shall come a Star out of Jacob, and a Sceptre shal

Evaluating the Impact of ICT Usage on the Performance of Jamaican
Research has demonstrated that ICT adoption is a key influencing factor in enhancing hotel performance at both the strat

arXiv:cs/0610154v1 [cs.DL] 26 Oct 2006

Usage Impact Factor: the effects of sample characteristics on usage-based impact metrics. Johan Bollen⋆† and Herbert Van de Sompel† †

Digital Library Research & Prototyping Team Los Alamos National Laboratory { jbollen, herbertv}@lanl.gov LA-UR-06-7626

There exist ample demonstrations that indicators of scholarly impact analogous to the citation-based ISI Impact Factor can be derived from usage data. However, contrary to the ISI IF which is based on citation data generated by the global community of scholarly authors, so far usage can only be practically recorded at a local level leading to community-specific assessments of scholarly impact that are difficult to generalize to the global scholarly community. We define a journal Usage Impact Factor which mimics the definition of the Thomson Scientific’s ISI Impact Factor. Usage Impact Factor rankings are calculated on the basis of a large-scale usage data set recorded for the California State University system from 2003 to 2005. The resulting journal rankings are then compared to Thomson Scientific’s ISI Impact Factor which is used as a baseline indicator of general impact. Our results indicate that impact as derived from California State University usage reflects the particular scientific and demographic characteristics of its communities.

according to their impact derived from a library’s access statistics. Bollen and Luce (2002) and Bollen, Sompel, Smith, and Luce (2005) propose the use of social network metrics calculated for journal networks derived from usage sequences in a library’s access log. Kurtz et al. (2004b, 2004a) discuss the potential of usage data for impact ranking. Brody, Harnad, and Carr (2006) later explore how early article usage statistics can predict citation rates. In addition to these research developments, practical standards for publisher reported usage statistics (COUNTER project1 ) and their aggregation (SUSHI project2 ) have been developed. Thomson Scientific has recently included usage statistics in its ISI Web of Knowledge product3. Since usage data is recorded by particular information systems, the acquired data naturally pertains to the user community of those systems. For example, when Bollen and Luce (2002) rank journals according to their usage this is done on the basis of usage data recorded by the Los Alamos National Laboratory Research Library servers and therefore reflects the preferences of the LANL community. In a similar manner, the results reported by Brody et al. (2006) apply to the user community of the UK arXiv mirror4. A similar argument can be made for the "citation-download correlation tool" of the University of Southampton’s CiteBase system5 which uses download information from the UK arXiv mirror.

1 Introduction Usage of scholarly resources as recorded by digital information systems has been gaining acceptance as a tool to study the scholarly community. Usage data has been used to study trends in science (Bollen, Luce, Vemulapalli, & Xu, 2003) as well as to visually map the interests of certain subsets of the scholarly community (Bollen & Van de Sompel, 2006). In addition, usage data has been shown to be a promising alternative to citation data in the assessment of scholarly impact. As early as 2001 (Darmoni, Roussel, Benichou, Thirion, & Pinhas, 2002) propose a reading factor to rank journals

In all cases the community for which usage was 1 http://www.projectcounter.org/ 2 http://www.niso.org/committees/SUSHI/SUSHI_comm.html 3 ISI

Web of Knowledge Usage Reporting System (WURS)

4 http://uk.arxiv.org/ 5 http://www.citebase.org/

1

recorded is delimited by the boundaries of a particular service or information system. The resulting sample of the scholarly community that generated the usage data through its interaction with these systems is unknown both in terms of its diversity and span. The CiteBase user community could in fact be a diverse mix of undergraduate students, professors, university staff, laypersons, and scholars. Its span may or may not be limited to the United Kingdom. The resulting usage data and its subsequent analysis could therefore be shaped by a set of sample characteristics that are not well-understood. In fact, when considering usage statistics as a population statistic, the question then emerges for which sample of the scholarly community usage has been recorded, and how the characteristics of that particular sample will influence the outcomes of a subsequent assessment of scholarly impact based on these statistics.

the scholarly community, i.e. increase its span. In fact, Bollen and Sompel (2006) propose an architecture for the large-scale aggregation of usage data which could be employed to achieve such global samples. This architecture however only addresses the technical issues involved in aggregating such samples; it does not address the issue of what constitutes a representative global sample, nor which services usage should be aggregated for. The second dimension, i.e. diversity, entails efforts to better understand and control how community characteristics, i.e. sample diversity, affect usage-based impact assessments, regardless of whether the sampled community is representative of the global scholarly community. Whereas (Bollen & Sompel, 2006) is focused on aspects of the first dimension, i.e. sample span, this article addresses the second dimension, i.e. sample diversity: studying the effects of sample characteristics on usagebased assessments of impact. Usage of scholarly resources for all 23 California State University (CSU) campuses, comprising about 405,000 students and 44,000 faculty and staff, was recorded throughout the entire October 2003 to August 2005 period by the CSU linking servers (Sompel & Beit-Arie, 2001), thereby generating an extensive, high-granularity usage data set covering one of the world’s largest and most diverse scholarly communities. A simple Usage Impact Factor (UIF) was defined to mimic the definition of the ISI IF and was then used to determine journal rankings on the basis of the recorded CSU usage data. Correlations between the resulting CSU UIF and ISI IF rankings are determined for a set of scholarly disciplines, demarcated by ISI journal classification codes. These correlations are then matched to the demographic features of the CSU community to yield insights into how they affect usage-based assessment of impact.

The issue of sampling permeates the field of scholarly impact assessments, even where citation data is used. Thomson Scientific’s ISI Impact Factor (ISI IF) is calculated from citation rates recorded for a set of ISI-selected journals. The corresponding sample of the scholarly community consequently has the following characteristics: 1. Span: extends to the global set of scholarly authors. 2. Diversity: limited to scholarly authors, and articles published in the set of ISI-selected journals. In spite of the latter limitation, the ISI IF is perceived to be based on a representative and respected sample which supports its general acceptance as an indicator of scholarly impact. In comparison to the ISI IF, usage-based assessments of scholarly impact are generally based on samples of the scholarly community with the following characteristics:

2

1. Span: delimited by the local boundaries of a particular information service.

2.1

Background Citation Impact Factor

2. Diversity: extends to all user types who can request services for any type of scholarly communication The IF of a particular journal in a particular year as defined by Garfield (1979) is determined by counting unit. the number of citations that occur in a given year to In order to realize impact measures derived from usage articles published in the journal during the two previous data that could achieve the same level of acceptance as years and dividing that number by the total number of the ISI IF, explorations along both the above dimensions published items in that two year period. As such, the IF need to take place. The first dimension, i.e. span, entails corresponds to the probability that the articles published the aggregation of usage data across a wide range of in a particular journal over a 2 year period are effectively services to create a more global, representative sample of cited in a given year. 2

two proceeding years y − 1 and y − 2 More formally, the IF can be defined as follows. We denote the set A of (citable) articles published in journal j in year y as Ayj so that Ayj = {a1 , a2 , · · · , an }, where ai ∈ Ayj represents an article published in journal j in year y. We introduce the citation function C y that maps a set of citable articles to the number of times these articles were cited by articles published in year y, i.e. C y (A) → N. It follows that C y (Akj ) returns the number of citations recorded in year y that point to the set of articles published in journal j in year k.

and |Ajy−1 ∪ Ajy−2 | represents the number of articles published by journal j in the two proceeding years y − 1 and y − 2.

The UIF expresses the probability that an article published in a journal within a 2 year period is used in a particular year, much like the IF expresses the probability that an article published in a journal within a 2 year The IF of a journal j in year y, denoted IFyj , is defined period is cited in a particular year. The similarities between the IF and the UIF are clarified in Fig. 1. as the ratio of two quantities: C y (Ajy−1 ∪ Ajy−2 )

To ensure that the IF and UIF for a particular journal (1) are determined on the basis of similar samples, the UIF ∪ denominator |Ajy−1 ∪ Ajy−2 | is chosen to be that of the IF, where namely the number of citable items published by journal j in years y − 1 and y − 2. In other words, the number of C y (Ajy−1 ∪ Ajy−2 ) represents the number of citations citable or "usable" articles in a journal are considered the in year y to all citable articles published in journal j in same quantity for a particular year. the two proceeding years y − 1 and y − 2, IFyj =

|Ajy−1

Ajy−2 |

articles in journal j

and all articles

|Ajy−1 ∪Ajy−2 | represents the number of citable articles published by journal j in the two proceeding years y − 1 and y − 2.

2003

2004

# citations

articles in journal j all articles

2003

2004 2002

# usage

2002

2.2 Usage Impact Factor 2004 IFj

A similar reasoning can be applied to the definition of a Usage Impact Factor (UIF) which can be framed in terms of the probability that an article published in a particular journal over a 2 year period is used, rather than cited. Analogous to the IF, we define the Usage Impact Factor of journal j in year y, denoted UIFyj , as follows. We replace the citation function C y (Akj ) with the usage function Ry (Akj ) → N which returns the number of times the articles in Akj are used in year y. The UIF can then be defined as the ratio between two quantities:

# publications

2004 UIFj

# publications

Figure 1: Usage Impact Factor (UIF) defined in analogy to the ISI Impact Factor (ISI IF).

In this work, we use the full-text downloads of an article as an approximation of article usage. A similar problem of approximation exists in citation analysis where author motivations to cite a particular article can vary strongly (MacRoberts & MacRoberts, 1989) and a citation can express any modality of agreement or interest. y−1 y−2 y Contrary to citation data which lacks any formal indicaR (A ∪ A ) j j UIFyj = (2) tion of author motivation, usage logs typically do specify y−1 y−2 |Aj ∪ Aj | the user request type thereby allowing a careful selection where of which to consider for a particular analysis. Although yet finer distinctions can be made between different types Ry (Ajy−1 ∪ Ajy−2 ) represents the number of uses of usage, e.g. surveys to determine actual reading rates recorded in year y of articles published in journal j in the (King, Tenopir, & Clarke, 2006), such an investigation 3

2.3.1 Sample Considerations

UIF

Global

SAMPLE

2.3 Data Acquisition

IF

ISI IF

GUIF

Local

METRIC

was beyond the scope of this study; full-text downloads were considered to be the most reliable, if somewhat partial, indicator of usage.

LIF

CSU UIF

The significance of sample span and diversity was outlined in the introduction. Therefore, when discussing usage- or citation-based metrics of impact, two orthog- Figure 2: Two orthogonal factors: formal metric definition and the sample to which it has been applied. onal factors need to be taken into account : 1. The characteristics of the sample that the specific metric has been calculated for, i.e. sample span and global samples of the scholarly community. The resulting diversity, UIF rankings would then reflect a more global rather than 2. The formal definition of a metric as an indicator of a local, institutional sample of the scholarly community. scholarly impact. Such metrics are labeled Global Usage Impact Factor This perspective is represented in Fig. 2. The IF, as (GUIF) in Fig. 2. defined in Eq. 1, can be calculated for any set of journal This paper outlines a comparison of the globally oricitation data. However, the most common instantiation of the IF is the one published by Thomson Scientific’s ented ISI IF which is used as a baseline indicator of genISI. This ISI IF is calculated on the basis of citation data eral impact versus the CSU UIF which represents a local, for a core set of about 8000 ISI-selected journals. With CSU-specific facet of scholarly impact. It is however conregards to the span of its sample, the ISI IF places no ceivable that once aggregated usage data becomes availrestrictions on the origin or affiliation of authors and able a comparison between CSU UIF and the GUIF, the therefore represents a global sample of the scholarly latter used as a global baseline, could be equally informacommunity, albeit one whose diversity is limited by the tive. focus on authors who published journal articles in the set of ISI-selected journals. 2.3.2 ISI IF Citation Data ISI IF values were extracted from the 2004 Journal Citation Reports (JCR) that are published on a yearly basis by Thomson Scientific’s ISI. Combined, the Science and Social Science edition of the 2004 JCR contained impact factors for 7,356 scholarly journals.

The IF can be calculated for local citation samples. For example, McDonald (2006) extracts citation data pertaining only to California Institute of Technology authors to determine local citation impact. This approach results in a Local Impact Factor (LIF) as indicated in Fig. 2.

2.3.3 CSU UIF Usage Data

The UIF as defined in Eq. 2 can in principle be calculated for any usage data set, but the nature of usage data is such that it is generally recorded for the local user communities of a specific service. This paper reports on UIF values calculated on the basis of usage data set for the California State University system which corresponds to a local, CSU-specific sample of the scholarly community. We therefore label the consequent UIF values "CSU UIF" to indicate the fact that they apply to local CSU usage.

A large-scale usage log was created by aggregating usage data recorded by the linking servers (Sompel, 1999a, 1999b; Sompel & Beit-Arie, 2001) of the entire California State University system in the period October 2003 to August 2005. Recording started November 11th, 2003 (10:44 AM) and continued uninterrupted until August 8th, 2005 (11:43PM). Linking server logs aggregate usage across all OpenURL-enabled information services, and thereby contain records of all user requests, The aggregation of usage data sets across different ser- including abstract requests and full-text downloads. They vices and institutions may in the future yield increasingly may additionally provide extensive usage, document 4

and user metadata which allows e.g. requester type, 3 Results request types and publication dates to be taken into account when considering usage-based indicators of 3.1 CSU UIF journal rankings scholarly impact. As linking servers become increasingly prevalent, they achieve a growing importance among the Table 1 lists the 10 journals with highest 2004 CSU UIF tools by which enabled library services can record us- as well as their 2004 ISI IF values. The list of 10 journals age (Gallagher, Bauer, & Dollar, 2005; McDonald, 2006). with highest 2004 CSU UIF values reveals a strong social science focus in the CSU community. The journals TopUsage for nine major institutions, i.e. Chancellor, ics in Early Childhood Special Education (TOP EARLY California Polytechnic State University, CSU Los Ange- CHILD SPEC), Hispanic Journal of Behavior Sciences les, CSU Northridge, CSU Sacramento, San Jose State (HISPANIC J BEHAV SCI), Intervention in School and University, CSU San Marcos, San Diego State University, Clinic (INTERV SCH CLIN) and Monographs of the and finally San Francisco State University was retained Society for Research in Child Development (MONOGR since they had recorded usage data most consistently and SOC RES CHILD) are found at the top of the list. The reliably, and represented the majority of CSU linking low 2004 ISI IF values of these journals indicates a strong server data. A total of 3,679,325 unique usage events discrepancy between the degree by which journals are was thus recorded in the resulting master log for a total used by the CSU community and their overall scholarly of 176,575 users (identified by their IP addresses6 ), impact as indicated by the 2004 ISI IF. requesting services for 1,657,312 unique documents. A majority of the requests, i.e. 73%, pertained to journal articles. A range of service request types was recorded, including but not limited to full-text downloads, requests for holding information, requests for journal citation data and abstract requests.

The 10 journals with highest 2004 ISI IF values are listed on the right-hand side of Table 1 along with their CSU UIF values. This ISI IF ranked list contains journals with high impact factor rankings such as Nature, Science, New England Journal of Medicine (NEW ENGL J MED), Cell and the Journal of the American Medical AssociaThe resulting master log was then filtered to only in- tion (JAMA). The corresponding 2004 CSU UIF values are relatively low for these journals in spite of their high clude events conforming to the following: 2004 ISI IF rankings. 1. Article full-text downloads.

3.2

2. Year of download was 2004.

Correlating CSU UIF and the ISI IF

The Spearman rank order correlation coefficient between 3. Download concerned articles published in 2002 and 2004 CSU UIF and 2004 ISI IF values was found to be 2003. -0.207 (p-value < 0.001, N=3,164) indicating a modest negative correlation between usage and the ISI IF for the A total of 140,675 usage requests remained after this California State University community. This negative filtering. These events pertained to articles published in relationship is confirmed by the log-log scaled scatterplot 6,423 unique journals. The number of full-text article in Fig. 3. Some of the journals on the extremities of downloads was tallied for each of these journals. The re- the scatterplot are labeled. It is notable that the journals sulting download frequency table was then merged with with a high ISI IF value (top of plot), regardless of their the 2004 ISI IF data resulting in a list of 3,146 journals 2004 CSU UIF values, mostly correspond to medicine. for which download data as well as non-zero ISI IF were In addition, a significant number of prominent physics available. Following Eq. 2, the journal download fre- journals (Physical Review B and Physical Review quency values were then divided by the number of citable Letters) are located in the quadrant of the plot which articles as was used to calculate the 2004 ISI IF, result- corresponds to high ISI IF and low CSU UIF values. ing in 2004 UIF values in conjunction with a 2004 ISI IF In other words, they are considered high impact in the value for each journal. general scholarly community but their articles are used relatively infrequently in the CSU community. 6 It is acknowledged that IP addresses do not uniquely identify individual users. However the presented analysis relies on overall article download frequencies and does not require unique user identification.

This comparison of 2004 CSU UIF and 2004 ISI IF 5

Ordered by 2004 CSU UIF Title UIF04 TOP EARLY CHILD SPEC 6.759 HISPANIC J BEHAV SCI 6.720 INTERV SCH CLIN 6.017 MONOGR SOC RES CHILD 5.571 J SCHOOL PSYCHOL 5.000 J FAM VIOLENCE 4.964 SEX ROLES 4.804 J YOUTH ADOLESCENCE 4.723 EDUC URBAN SOC 4.653 J AUTISM DEV DISORD 4.513

Rank 1 2 3 4 5 6 7 8 9 10

IF04 0.862 0.500 0.172 7.286 1.750 0.491 0.639 0.855 0.224 2.128

Ordered by 2004 ISI IF Title UIF04 ANNU REV IMMUNOL 0.059 CA-CANCER J CLIN 0.667 NEW ENGL J MED 0.262 PHYSIOL REV 0.164 NATURE 0.277 SCIENCE 0.288 ANNU REV BIOCHEM 0.077 CELL 0.002 JAMA-J AM MED ASSOC 1.196 ANNU REV NEUROSCI 0.048

IF04 52.431 44.515 38.570 33.918 32.182 31.853 31.538 28.389 24.831 23.143

5e+01

Table 1: Journals ranked by 2004 CSU UIF and 2004 ISI IF values.

The disciplines used by CSU to tally enrollment and faculty numbers in its Statistical Abstracts (Analytic Studies Division, 2004) are the starting point of the discipline-specific comparisons of 2004 CSU UIF and 2004 ISI IF values in this paper. These disciplines are listed in Table 2 (reproduced from Analytic Studies Division (2004), page 125, table 81).

ANNU REV IMMUNOL CA−CANCER J CLIN NEW ENGL J MED PHYSIOL REV ANNU REV BIOCHEM

CELL CHEM REV J EXP MED

PHARMACOL REV ANNU REV ASTRON ASTR

JAMA−J AM MED ASSOC LANCET ENDOCR REV

CURR OPIN CELL BIOL

TRENDS NEUROSCI ANNU REV PSYCHOL

PLANT CELL CURR OPIN IMMUNOL

ANNU REV NUTR

BLOOD AM J PSYCHIAT

5e+00

PHYS REV LETT

MONOGR SOC RES CHILD AM PSYCHOL

PHYS REV D J PHYS CHEM B

DEV PSYCHOL

J AUTISM DEV DISORD J SCHOOL PSYCHOL

J APPL POLYM SCI

IF04

5e−01

TOP EARLY CHILD SPEC

HISPANIC J BEHAV SCI

CHINESE CHEM LETT EDUC URBAN SOC INTERV SCH CLIN J BLACK STUD

ASSEMBLY AUTOM CHEM TECH FUELS OIL+

CONTEMP FAM THER CLIN SOC WORK J

POLITY

5e−02

CIVIL ENG STUD E EUR THOUGHT ELECTRON COMM JPN 3

Disciplines Agriculture and Natural Resources, Architecture and Environmental Design, Area Studies, Biological Sciences, Business and Management, Communications, Computer and Information Sciences, Education, Engineering, Fine and Applied Arts, Foreign Languages, Health Professions, Home Economics, Interdisciplinary Studies, Letters, Library Science, Mathematics, Physical Sciences, Psychology, Public Affairs, Social Sciences

PROF ENG FRONTIERS

R&D MAG

AIRCR ENG AEROSP TEC

POWER ENG−US

FEMINIST STUD

COMMUN NEWS IND ENG

rho = −0.207*** N=3146

CONTROL SOLUT RUSS EDUC SOC

5e−03

EE−EVAL ENG

1e−03

1e−02

1e−01

1e+00

1e+01

UIF04

Figure 3: CSU Usage Impact Factor and ISI Impact Factor values for 3,146 journals.

Table 2: California State University disciplines used to tally enrollment and faculty numbers. values fails to take into account variations among the different disciplines in the CSU system. A set of disciplineTo separate the group of examined journals in specific comparisons of the correlation between the 2004 discipline-related sets, we manually matched each of CSU UIF and 2004 ISI IF is therefore provided in the fol- the listed CSU disciplines with a set of ISI journal lowing sections. classification codes7 . These classification codes were then used to demarcate discipline-related sets of journals within which a comparison of CSU UIF and ISI IF could 3.3 Discipline-specific comparisons be conducted. The ISI journal classification codes for The scatterplot in Fig. 3 suggests that the relationship the CSU disciplines listed in Table 2 are provided in between the 2004 CSU UIF and 2004 ISI IF values Table 7 (appendix). The 2004 CSU UIF and 2004 ISI IF differ for particular disciplines, e.g. among the set of correlations calculated for each of the thus demarcated journals with high ISI IF values and low CSU UIF CSU disciplines are listed in Table 3. Statistically values we find a preponderance of physics journals. It significant correlations, marked in bold font, were found is therefore warranted to assess the CSU UIF and ISI for only 3 of the 17 disciplines, namely Interdisciplinary IF correlations within, rather than between, individual 7 This is a subjective matter. However, specific care was taken to scholarly disciplines. match ISI Journal Classification Codes as literally as possible to the specific CSU disciplines.

6

Studies (ρ = −0.470, N = 89, p < 0.001), Education (ρ = 0.228, N = 127, p = 0.010) and Engineering (ρ = −0.147, N = 259, p = 0.018). Physical Sciences was found to have a marginally significant, negative correlation (ρ = −0.225, N = 56, p = 0.096). Log-log scaled scatterplots of the 2004 CSU UIF vs. 2004 ISI IF values for the mentioned four disciplines are shown in Fig. 4 and confirm the reported correlations.

Education 0.23 *** EDUC PSYCHOL

Engineering −0.15 ** ANNU REV BIOMED ENG

CHILD DEV

5e+00

2.00

J LEARN SCI

LEARN INDIVID DIFFER J SCHOOL PSYCHOL

ENVIRON SCI TECHNOL IEEE COMMUN MAG

TOP EARLY CHILD SPEC

5e−01 INTERV SCH CLIN EDUC STUD

IEEE TECHNOL SOC MAG

IF04

0.20

IF04

0.50

WIREL NETW

J URBAN PLAN D−ASCE

5e−02

0.05

J PROF ISS ENG ED PR

J EDUC GIFTED

AEROSPACE AM PLAST ENG

CIVIL ENG

5e−03

0.01

IND ENG

0.01

0.05

0.20

0.50

2.00

5.00

0.001

0.005

0.020

UIF04

PHYS REV LETT

6

5.0

BIOINFORMATICS

IEEE T MED IMAGING

PHYS TODAY

J AM MED INFORM ASSN

4

IEEE J QUANTUM ELECT

SOC SCI RES

YOUTH SOC

CONTEMP PHYS

IF04

1.0

2

2.0

FUTURE CHILD

0.5

IF04

0.200

Physical Sciences −0.22 * 8

Interdisciplinary studies −0.47 ***

2004 CSU UIF vs. 2004 ISI IF Discipline rho N p-value Interdisciplinary Studies −0.470 89 >0.001 Education +0.228 127 0.010 Engineering −0.147 259 0.018 Physical Sciences −0.225 56 0.096 Agriculture and Natural Resources +0.238 40 0.138 Business and Management +0.132 115 0.160 Computer and Information Sciences +0.077 155 0.338 Area Studies +0.169 27 0.397 Public Affairs −0.073 106 0.455 Library +0.126 25 0.546 Psychology +0.033 316 0.556 Architect. and Environ. Design +0.041 188 0.572 Mathematics +0.077 44 0.617 Biological Sciences −0.024 331 0.669 Communications +0.049 58 0.712 Social Sciences +0.026 59 0.843 Health Professions −0.012 126 0.890

0.050

UIF04

PHYS WORLD AM J PHYS

J HOMOSEXUAL

IEEE T SEMICONDUCT M

0.2

FOUND PHYS

J MATH SOCIOL

INT SOC SCI J

SOLID STATE TECHNOL

J BLACK STUD

0.1

HYPERFINE INTERACT

0.002

0.010

0.050 UIF04

0.200

1.000

5e−04

2e−03

5e−03

2e−02

5e−02

2e−01

UIF04

Figure 4: CSU UIF and ISI IF comparisons for 4 disciplines with highest and lowest correlations.

UIF and ISI IF values was not affected by the total number of students enrolled in a particular discipline. No staTable 3: Discipline-specific 2004 CSU UIF and 2004 ISI tistically significant correlation was found between total IF Spearman rank-order correlations. student enrollment numbers and the correlation between CSU UIF and ISI IF correlations (ρ = −0.262, N = 17, p = 0.308). It is of particular interest that three out of the four mentioned disciplines exhibit a negative correlation 3.4 Community demographics between 2004 CSU UIF and 2004 ISI IF values. Whereas a zero correlation would have indicated the absence of On the basis of the hypothesis that the observed correa relationship, in this case the two metrics are inversely lations between CSU UIF and ISI IF values for these correlated indicating that members of the communities disciplines may be related to the academic demographics interested in the particular discipline specifically do of the CSU communities corresponding to the invesnot frequently use articles published in high-impact tigated disciplines, 2004 undergraduate and graduate journals and vice versa. However, for Education a enrollment and faculty numbers were matched to the significant positive correlation was found between the observed correlations. Faculty numbers are estimated 2004 CSU UIF and 2004 ISI IF, indicating that for this in terms of Full Time Equivalent Faculty (FTEF), i.e. particular CSU discipline journal usage is moderately the total number of hours taught in a particular division related to scholarly impact as indicated by the 2004 ISI IF. divided by the assumed 15 hours required for full-time faculty status. The particular number of FTEF and The size of a discipline in terms of the number of students respectively teaching or enrolled at the underjournals that it comprises may affect ISI IF values. A graduate or graduate level are listed in Table 4. Note that marginally significant correlation was found between the undergraduate FTEF numbers are split into low and high CSU UIF and ISI IF correlation vs. the number of jour- divisions which need to be summed to determine total nals in that particular discipline (ρ = −0.459, N = undergraduate FTEFs. 17, p = 0.065). However, the correlation between CSU 7

Discipline Agri. & Nat. Res. Arch. & Env. Des. Area Studies Biol. Sci. Bus. & Mngmt. Communications Comp. & Inf. Sci. Education Engineering Fine & Appl. Arts Foreign Lang. Health Prof. Home Econ. Interdisc. Stud. Letters Library Mathematics Phys. Sci. Psychology Public Affairs Social Sciences

Students U.Grad Grad. 5,381 302 2,902 358 319 148 13,642 1,052 60,069 5,242 14,252 674 16,415 2,322 16,084 15,452 22,877 4,146 19,418 1,321 2,252 486 13,386 3,984 3,261 738 29,780 948 13,594 3,413 561 3,325 816 3,310 741 16,944 1,380 14,250 4,643 24,597 2,956

Low 62.7 33.9 12.9 243.3 143.3 139.5 119.7 49.6 191.8 425.3 226.2 31.2 29.4 146.6 729.6 6.6 488.6 425.6 84.6 47.4 570.4

FTEF High 127.5 72.1 25.1 264.7 914.4 299.5 223.8 750.7 483.6 712.1 138.5 142.9 93.0 225.5 691.6 2.0 189.8 320.2 332.9 287.0 1,081.9

to graduate faculty. A highly significant negative CSU UIF vs. ISI IF correlation was observed for this discipline.

Grad. 21.0 19.2 4.3 89.1 161.3 31.4 68.3 836.6 123.9 102.0 21.2 143.1 16.4 24.8 170.9 17.3 48.5 75.3 108.9 216.8 162.8

Conversely, Education is characterized by a ±1 to 1 ratio of undergraduate students and faculty to graduate students and faculty. A significant positive correlation was observed between journal CSU UIF vs. ISI IF values within this discipline. This pattern is further confirmed by the undergraduate vs. graduate ratios for Engineering and Physical Sciences which has a moderate ±5 to 1 and ±10 to 1 undergraduate vs. graduate enrollment rate. Moderate negative CSU UIF vs. ISI IF correlation were observed. A linear regression model was generated for the relation between the ratio of graduate to undergraduate numbers versus the observed 2004 UIF and 2004 ISI IF correlation on the basis of the 4 data points listed in Table 5. Since similar results were obtained for all three demographic ratios ("All", "Student and "Faculty"), only the linear regression model for the combined student and faculty ratios ("All") is discussed. Fig. 5 shows a scatterplot of the mentioned values and the corresponding linear regression model. The linear regression model was found to have an intercept of -0.3873 and a slope of 0.7183 (r2 = 0.9029).

Table 4: California State University student enrollment and Full Time Equivalent Faculty (FTEF) numbers (undergraduate and graduate) for 2004. Three ratios of the undergraduate versus the graduate community were defined as follows:

1. All: the ratio of total graduate student enrollment plus graduate FTEF numbers over the total number of undergraduate student enrollment plus undergraduate (high and low division combined) FTEF numFrom this it could be predicted that CSU UIF vs. ISI bers. IF correlations become positive as soon as the graduate community becomes twice as large as the undergraduate 2. Student: the ratio of graduate over undergraduate community in a particular discipline. It can be noted that student enrollment. the overall ratio of graduate vs. undergraduate enrollment 3. Faculty: the ratio of graduate FTEF numbers over for the entire CSU system is 51,694 / 326,483 = 0.158, which together with the observed UIF vs. ISI IF correlaundergraduate FTEF numbers. tion of ρ = −0.207 (p-value < 0.001, N=3,164) support The thus defined ratios were then compared to the the above mentioned pattern. observed CSU UIF vs. ISI IF correlations in Table 3. It must be stressed this comparison was restricted to 3.5 Baseline assessment the mentioned four disciplines for which significant or marginally significant CSU UIF vs. ISI IF correlations The 2004 ISI IF is used as a baseline assessment of were observed. The results are listed in Table 5 and scholarly impact against which 2004 CSU UIF values suggest the possibility of a relationship between the can be compared. Although CSU UIF and ISI IF are ratio of the graduate to undergraduate community within deliberately compared for the same years in which usage, a discipline and the observed CSU UIF vs. ISI IF citation and publication samples were recorded, questions correlations. arise with regards to the sensitivity of this comparison to longitudinal changes in the ISI IF over time. In particular, the discipline of Interdisciplinary Studies is characterized by a ±15 to 1 ratio of undergraduate to For this reason we investigated the degree of correlagraduate students, and a ± 30 to 1 ratio of undergraduate tion between the 2004 CSU UIF vs. past ISI IF values, 8

Discipline Interdisciplinary Studies Physical Sciences Engineering Education

N 89 56 259 127

ρ(UIF,IF) −0.470 −0.225 −0.147 +0.228

Grad. vs. Undergrad. ratio Student Faculty All 0.067 0.032 0.032 0.101 0.224 0.202 0.183 0.180 0.180 1.045 0.881 0.888

p-value 0.000 0.096 0.018 0.010

Table 5: 2004 CSU UIF and ISI IF correlations compared to ratios of faculty and student numbers.

0.0

0.2

0.2 0.1 −0.1

rho(UIF,IF)

−0.2

Physical Sciences

−0.4

−0.3 Interdisciplinary Studies

Engineering

−0.3

−0.2

Engineering

Physical Sciences

−0.4

−0.4

−0.3

Physical Sciences

0.0

0.1 −0.1

rho(UIF,IF)

0.0

0.1 0.0 −0.1

Engineering

−0.2

rho(UIF,IF)

Education

0.2

Education

0.2

Education

Interdisciplinary Studies

0.4

0.6

Graduate/undergraduate ratio (2004)

0.8

0.2

Interdisciplinary Studies

0.4

0.6

0.8

1.0

0.0

0.2

Graduate/undergraduate faculty ratio (2004)

0.4

0.6

0.8

Graduate/undergraduate student ratio (2004)

Figure 5: Comparisons of Fall 2004 student and faculty populations vs. 2004 CSU UIF vs. ISI IF correlation.

rho(UIF04,IF99)=−0.17, p<0.001

1e−02

1e−01

1e+00

1e+01

1e−02

1e−01

1e+00

1e+01

IF00

5e−02

5e−01

5e+00

5e+01

5e+01 5e+00 IF99

5e−03

5e−03 1e−03

1e−03

1e−02

1e−01

1e+00

1e+01

1e−03

1e−02

1e−01

1e+00

1e+01

UIF04

UIF04

rho(UIF04,IF03)=−0.204, p<0.001

rho(UIF04,IF04)=−0.207, p<0.001

1e−03

1e−02

1e−01 UIF04

1e+00

1e+01

5e+01 5e+00 IF04

5e−02 5e−03

5e−02 5e−03 1e−03

1e−02

1e−01 UIF04

1e+00

1e+01

5e−01

5e+00 IF03b

5e−01

IF02

5e−03

5e−03

5e−02

5e−01

5e−01

5e+00

5e+00

5e+01

5e+01

UIF04

rho(UIF04,IF02)=−0.203, p<0.001

5e+01

UIF04

rho(UIF04,IF01)=−0.197, p<0.001

5e−02

IF01

rho(UIF04,IF00)=−0.171, p<0.001

5e−02

IF98

5e−03 1e−03

3.6 Results Summary

5e−01 5e−02

0.50 0.02

0.10

IF97

2.00

5e+00

10.00

5e+01

50.00

rho(UIF04,IF98)=−0.159, p<0.001

5e−01

rho(UIF04,IF97)=−0.186, p<0.001

i.e. ISI IF values that were published in 1997 through 20048. The results are listed Table 6. These correlations indicate a stable, negative correlation between 2004 CSU UIF values and past ISI IF values over the mentioned period of 8 years. The absence of a particular trend in CSU UIF vs. ISI IF correlations is supported by the plot in Table 6. The scatterplots of CSU UIF vs. ISI IF values for each specific year are shown in Fig. 6.

1e−03

1e−02

1e−01 UIF04

1e+00

1e+01

1e−03

1e−02

1e−01

1e+00

1e+01

UIF04

Figure 6: CSU UIF vs. ISI IF comparisons for 1997-2004 period.

The picture that emerges from these results can be summarized as follows: 1. A moderate negative correlation between 2004 CSU UIF and 2004 ISI IF values was found without taking into account CSU disciplines.

3. Some CSU disciplines exhibit negative correlations between CSU UIF and ISI IF values whereas others exhibit positive correlations. Most disciplines however exhibit zero or insignificant correlations.

2. This negative correlation persists over a period of 8 years counting back ISI IF values from the year in which usage was recorded (2004).

4. CSU UIF vs. ISI IF correlations seemed to be related to the ratio between the sizes of the undergraduate and graduate community in a discipline.

8 At

the time this analysis was conducted, 2005 ISI IF values were not yet available.

9

1.0 0.5 0.0

rho(UIF04,IFyear)

−0.159

−0.17

−0.171

1999

2000

−0.197

−0.203

−0.204

−0.207

2001

2002

2003

2004

−1.0

−0.5

−0.186

1997

1998

year

ISI IF year 2004 CSU UIF N p-value

ISI IF 1997 −0.186 2636 <0.001

ISI IF 1998 −0.159 2750 <0.001

ISI IF 1999 −0.170 2819 <0.001

ISI IF 2000 −0.171 2892 <0.001

ISI IF 2001 −0.197 2960 <0.001

ISI IF 2002 −0.203 3050 <0.001

ISI IF 2003 −0.204 3096 <0.001

ISI IF 2004 −0.207 3146 <0.001

Table 6: Spearman rank-order correlation values between 2004 Usage Impact Factor and 1997-2004 ISI ISI Impact Factors.

4 Conclusion Usage-based metrics of scholarly impact are gradually gaining acceptance in the domain of bibliometrics. However, little attention has been paid to how usage-based impact assessments are influenced by the demographic and scholarly characteristics of particular communities. The discussed analysis of CSU usage data indicates significant, community-based deviations between local usage impact and global citation impact as indicated by the generated CSU UIF and ISI IF rankings respectively. In particular, we found a general negative correlation between the CSU IF and the ISI IF, which indicates usage over the entire CSU community is inversely related to general citation impact. The observed negative correlations between the CSU UIF and ISI IF run counter to previous findings. In fact, Brody et al. (2006) and Bollen et al. (2005) report positive correlations between usage and citation rates. However, the services that recorded this usage, namely the UK arXiv mirror and the LANL Research Library systems, mostly accommodate a community of scholars in computer science and physics. The CSU community for wich

usage was recorded is composed of a mix of students, faculty, staff and others, focused on a variety of science and social science domains. It can be speculated that both the nature of the CSU library collection as well as the CSU community that uses it jointly contributed to the negative correlations between CSU UIF and ISI IF values. However, positive as well as negative CSU UIF vs. ISI IF correlations were observed for specific scholarly disciplines. In addition, a comparison of the relative sizes of the undergraduate and graduate communities at CSU to the correlations of CSU UIF vs. ISI IF values within specific disciplines, suggested that the size of the graduate community (students and faculty) relative to that of the undergraduate community within a discipline could be related to the magnitude of the observed CSU UIF vs. ISI IF correlations. The tentative linear relationship that was observed between the ratio of graduate to undergraduate enrollment and CSU UIF vs. ISI IF correlations raises the possibility that applications of usage data can take into account demographic data to extract different facets of impact. We must however caution that the latter observations are based on only those 4 disciplines for which significant or marginally significant CSU UIF vs.

10

ISI IF correlations were observed. Future research could impact as they exist in the scholarly community. Indeed, focus on validating these tentative results for a larger one could argue that an article that is often read by a number of disciplines. majority of students, yet is seldom cited by scholars in this field, nevertheless has considerable scholarly impact. In Section 2.3.1 we distinguished two factors that In fact, on the basis of sufficiently detailed usage data, shape metric-based assessments of scholarly impact, impact could be separately assessed for any subset of namely the formal definition of a metric and the sample the scholarly community including undergraduate and that it has been applied to. Although the UIF has been graduate students, research faculty, lecturers and the defined to mimic the IF, the CSU UIF and ISI IF rankings public at large. in this manuscript have been generated for very different samples of the scholarly community. The ISI IF rests on Finally, where only local usage data is collected, there citation data collected for a set of ISI-selected journals; is still particular value in being able to determine local its rankings therefore express the global community of all impact rankings which correspond to the preferences and scholarly authors publishing in those journals. The CSU characteristics of specific communities such as CSU. The usage data on the other hand reflects the characteristics of CSU UIF generated in this article may not be globally the local CSU academic community that comprises a mix applicable, but offers CSU administrators an interesting of students and faculty among others. It can therefore perspective on what is valued in their community. Our be considered at the same time more diverse than the analysis demonstrates that considerable, yet locally ISI-defined sample in terms of its composition, yet more meaningful deviations can occur between impact as it is limited in terms of its span since it applies to CSU users perceived by particular scholarly disciplines and the ISI only. IF. Such deviations are not problematic, but offer considerable possibilities to optimize local information services We envision three future paths along which usage- and adopt policies to accommodate the preferences of based metrics such as the UIF can be developed. These local communities. paths are not mutually exclusive and are related to the issues mentioned in the introduction. Many issues remain to be addressed in future research on this topic. The Andrew W. Mellon foundation has The first path is one in which attempts are undertaken awarded a grant to our team to investigate a range of issues to mimic the properties of the ISI IF on the basis of usage related to the definition of usage-based metrics of scholdata. This requires the aggregation of a meaningful, rep- arly impact. The funded project, named MESUR9 , aims to resentative sample of the scholarly community, similar construct a large-scale model of the scholarly community in span to the ISI IF sample, and efforts to compensate which merges usage and bibliographic data to support the for the increased diversity of the usage data sample, definition and validation of a range of usage-based metrics e.g. excluding all agents that are not scholarly authors of scholarly status. This paper describes our first exploand taking into account particular discipline-specific rations in this research area. demographics and preferences. This article has provided an initial exploration of the second issue, whereas the architecture described in (Bollen & Sompel, 2006) References may offer at least a technical solution to the first issue. Questions remain as to how one can create a truly repre- Analytic Studies Division. (2004). Statistical abstract 2004-2005 (Tech. Rep.). California State Universentative usage sample of the global scholarly community. sity. The second path along which usage-based metrics of scholarly status can be developed is focused on leveraging Bollen, J., & Luce, R. (2002). Evaluation of digital library impact and user communities by analysis of usage the greater diversity (in terms of agents and community patterns. D-Lib Magazine, 8(6). characteristics) that usage data generally engenders. This path may still require the aggregation of a meaningful, Bollen, J., Luce, R., Vemulapalli, S., & Xu, W. (2003). representative sample of the scholarly community, but Detecting research trends in digital library readerits assessment of scholarly impact specifically leverages 9 http://www.mesur.org/ sample diversity to assess the many different facets of 11

ship. In Proceedings of the seventh European Con- MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal ference on Digital Libraries (LNCS 2769) (pp. 24– of the American Society for Information Science, 28). Trondheim, Norway: Springer-Verlag. 40(5), 342–349. Bollen, J., & Sompel, H. Van de. (2006). An architecture for the aggregation and analysis of scholarly us- McDonald, J. D. (2006). Understanding online journal usage: A statistical analysis of citation and use. Jourage data. In Joint Conference on Digital Libraries nal of the American Society for Information Science (JCDL2006) (pp. 298–307). Chapel Hill, NC. and Technology, 57(13). Bollen, J., Sompel, H. V. de, Smith, J., & Luce, R. (2005). Toward alternative metrics of journal impact: a Sompel, H. V. de. (1999a). Reference linking in a hybrid library environment (i). D-Lib Magazine, 5(4). comparison of download and citation data. Information Processing and Management, 41(6), 1419– Sompel, H. V. de. (1999b). Reference linking in a hybrid 1440. library environment (ii). D-Lib Magazine, 5(4). Bollen, J., & Van de Sompel, H. (2006). Mapping the Sompel, H. V. de, & Beit-Arie, O. (2001). Open linking structure of science through usage. Scientometrics, in the scholarly information environment using the 69(2). OpenURL framework. D-Lib Magazine, 7(3). Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Informa- Acknowledgements tion Science and Technology, 57(8), 1060 – 1072. We thank the Andrew W. Mellon Foundation for supportDarmoni, S. J., Roussel, F., Benichou, J., Thirion, B., & ing this research. We also thank Marko A. Rodriguez for Pinhas, N. (2002). Reading factor: a new biblio- proofreading the earlier versions of this manuscript and metric criterion for managing digital libraries. Jour- Joan Smith at the Department of Computer Science at Old nal of the Medical Library Association, 90(3), 323– Dominion University for producing the raw citation data on which parts of this analysis are based. 327. Gallagher, J., Bauer, K., & Dollar, D. M. (2005). Evidence-based librarianship: Utilizing data from all available sources to make judicious print cancellation decisions. Library Collections, Acquisitions and Technical Services, 29. Garfield, E. (1979). Citation indexing: Its theory and application in science, technology, and humanities. New York: John Wiley and Sons. King, D. W., Tenopir, C., & Clarke, M. (2006). Measuring total reading of journal articles. D-Lib Magazine, 12(10). Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M., & Murray, S. S. (2004a). The bibliometric properties of article readership information. JASIST, 56(2), 111–128. Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M., & Murray, S. S. (2004b). Worldwide use and impact of the NASA Astrophysics Data System digital library. JASIST, 56(1), 36–45. 12

Appendix Agriculture and Natural Resources: AD (AGRICULTURE, DAIRY & ANIMAL SCIENCE), AE (AGRICULTURAL ENGINEERING, AF (AGRICULTURAL ECONOMICS & POLICY), AH (AGRICULTURE, MULTIDISCIPLINARY), XE (AGRICULTURE, SOIL SCIENCE) Architecture and Environmental Design: IH (ENGINEERING, ENVIRONMENTAL), JA (ENVIRONMENTAL SCIENCES, NE (PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH), JB (ENVIRONMENTAL STUDIES) Area Studies: BM (AREA STUDIES) Biological Sciences: CQ (BIOCHEMISTRY & MOLECULAR BIOLOGY, CU (BIOLOGY), DB (BIOTECHNOLOGY & APPLIED MICROBIOLOGY), DR (CELL BIOLOGY), HT (EVOLUTIONARY BIOLOGY), HY (DEVELOPMENTAL BIOLOGY), PI (MARINE & FRESHWATER BIOLOGY), QU (MICROBIOLOGY), WF (REPRODUCTIVE BIOLOGY), BV (PSYCHOLOGY, BIOLOGICAL) Business and Management: DI (BUSINESS), DK (BUSINESS, FINANCE), PE (OPERATIONS RESEARCH & MANAGEMENT SCIENCE), PC (MANAGEMENT) Communications: YE (TELECOMMUNICATIONS, EU (COMMUNICATION) Computer and Information Sciences: EP (COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE), ER (COMPUTER SCIENCE, CYBERNETICS), ES (COMPUTER SCIENCE, HARDWARE & ARCHITECTURE, ET (COMPUTER SCIENCE, INFORMATION SYSTEMS), EV (COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS), EW (COMPUTER SCIENCE, SOFTWARE ENGINEERING), EX (COMPUTER SCIENCE, THEORY & METHODS), ET (COMPUTER SCIENCE, INFORMATION SYSTEMS), PT (MEDICAL INFORMATICS), NU (INFORMATION SCIENCE & LIBRARY SCIENCE) Education: HB (EDUCATION, SCIENTIFIC DISCIPLINES), HA (EDUCATION & EDUCATIONAL RESEARCH), HE (EDUCATION, SPECIAL), HI (PSYCHOLOGY, EDUCATIONAL) Engineering: AE (AGRICULTURAL ENGINEERING), AI (ENGINEERING, AEROSPACE), EW (COMPUTER SCIENCE, SOFTWARE ENGINEERING), IF (ENGINEERING, MULTIDISCIPLINARY), IG (ENGINEERING, BIOMEDICAL), IH (ENGINEERING, ENVIRONMENTAL), II (ENGINEERING, CHEMICAL), IJ (ENGINEERING, INDUSTRIAL), IK (ENGINEERING, MANUFACTURING, IL (ENGINEERING, MARINE), IM (ENGINEERING, CIVIL), IO (ENGINEERING, OCEAN), IP (ENGINEERING, PETROLEUM), IQ (ENGINEERING, ELECTRICAL & ELECTRONIC), IU (ENGINEERING, MECHANICAL), IX (ENGINEERING, GEOLOGICAL), PZ (METALLURGY & METALLURGICAL ENGINEERING) Fine and Applied Arts: No results Foreign Languages: No results Health Professions: HL (HEALTH CARE SCIENCES & SERVICES), NE (PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH), LQ (HEALTH POLICY AND SERVICES) Home Economics: No results Interdisciplinary Studies: EV (COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS, PO (MATHEMATICS, INTERDISCIPLINARY APPLICATIONS), WU (SOCIAL SCIENCES, INTERDISCIPLINARY) Letters: No results Library: NU (INFORMATION SCIENCE & LIBRARY SCIENCE) Mathematics: PN (MATHEMATICS, APPLIED), PO (MATHEMATICS, INTERDISCIPLINARY APPLICATIONS), PQ (MATHEMATICS) Physical Sciences: UB (PHYSICS, APPLIED), UF (PHYSICS, FLUIDS & PLASMAS), UH (PHYSICS, ATOMIC, MOLECULAR & CHEMICAL), UI (PHYSICS, MULTIDISCIPLINARY), UK (PHYSICS, CONDENSED MATTER) Psychology: VI (PSYCHOLOGY), BV (PSYCHOLOGY, BIOLOGICAL), EQ (PSYCHOLOGY, CLINICAL), HI (PSYCHOLOGY, EDUCATIONAL), , MY (PSYCHOLOGY, DEVELOPMENTAL), NQ (PSYCHOLOGY, APPLIED), VJ (PSYCHOLOGY, MULTIDISCIPLINARY), VP (PSYCHOLOGY, PSYCHOANALYSIS), VS (PSYCHOLOGY, MATHEMATICAL), VX (PSYCHOLOGY, EXPERIMENTAL), WQ (PSYCHOLOGY, SOCIAL) Public Affairs: NE (PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH), VM (PUBLIC ADMINISTRATION)| Social Sciences: PS (SOCIAL SCIENCES, MATHEMATICAL METHODS), WU (SOCIAL SCIENCES, INTERDISCIPLINARY), WV (SOCIAL SCIENCES, BIOMEDICAL)

Table 7: ISI journal classification codes for CSU disciplines listed in Table 2.

13