Bibliometrics

Bibliometric Analysis for the U.S. Environmental Protection Agency/Office of Research and Developments Water Quality Research Program
September 2008

This is a bibliometric analysis of the papers prepared by researchers of the U.S. Environmental Protection Agency (EPA) for the Water Quality Research Program. For this analysis, a total of 867 journal and 109 non-journal publications published from 1998 to 2008 were reviewed. The 867 journal publications were cited 8,697 times and the 109 non-journal publications were cited 279 times in the journals covered by Thomsons Web of Science¹ and Scopus² . Of the 867 journal publications, 738 (85.1%) have been cited at least once in a journal. Of the 109 non-journal publications, 50 (45.9%) have been cited at least once in a journal.

Searches of Thomson Scientifics Web of Science and Scopus were conducted to obtain times cited data for the journal publications. The analysis was completed using Thomsons Essential Science Indicators (ESI) and Journal Citation Reports (JCR) as benchmarks. ESI provides access to a unique and comprehensive compilation of essential science performance statistics and science trends data derived from Thomsons databases. For this analysis, the ESI highly cited papers thresholds as well as the hot papers thresholds were used to assess the influence and impact of the Water Quality publications. JCR is a recognized authority for evaluating journals. It presents quantifiable statistical data that provide a systematic, objective way to evaluate the worlds leading journals and their impact and influence in the global research community. The two key measures used in this analysis to assess the journals in which the EPA Water Quality papers were published are the Impact Factor and Immediacy Index. The Impact Factor is a measure of the frequency with which the average article in a journal has been cited in a particular year. The Impact Factor helps evaluate a journals relative importance, especially when compared to other journals in the same field. The Immediacy Index is a measure of how quickly the average article in a journal is cited. This index indicates how often articles published in a journal are cited within the same year and it is useful in comparing how quickly journals are cited.

The report includes a summary of the results of the bibliometric analysis, an assessment of the 867 Water Quality journal articles analyzed by ESI field (e.g., Chemistry, Environment/Ecology, Microbiology), an analysis of the journals in which the Water Quality papers were published, a table of the highly cited researchers in the Water Quality Research Program, information on the patents/patent applications that have resulted from the program, and an assessment of the 109 non-journal publications.

Summary of Results

1. More than one-seventh of the 867 Water Quality journal publications are highly cited papers. 132 (15.2%) of the 867 Water Quality journal publications qualify as highly cited when using the ESI criteria for the top 10% of highly cited publications. This is 1.5 times the number expected. 9 (1.0%) of the Water Quality journal papers qualify as highly cited when using the ESI criteria for the top 1%, which is the number expected. 1 (0.1%) of the Water Quality publications qualifies as very highly cited when using the ESI criteria for the top 0.1% of highly cited publications, which is the number expected. 1 (0.1%) of the Water Quality journal papers qualifies as extremely highly cited when using the criteria for the top 0.01% threshold for the most highly cited papers, which is 10 times the number expected.
2. The Water Quality journal publications are more highly cited than the average paper. Using the ESI average citation rates for papers published by field as the benchmark, in 9 of the 15 fields in which the 867 Water Quality journal papers were published, the ratio of actual to expected cites is greater than 1, indicating that the Water Quality journal publications are more highly cited than the average papers in those fields. For all 15 fields combined, the ratio of total number of cites to the total number of expected cites (8,697 to 6,826.3) is 1.3, indicating that the Water Quality journal papers are more highly cited than the average paper.
3. Nearly one-seventh of the Water Quality journal papers are published in high impact journals. 120 of the 867 journal papers were published in the top 10% of journals ranked by JCR Impact Factor, representing 13.8% of the Water Quality journal publications. This number is 1.4 times higher than expected. 103 of the 867 papers appear in the top 10% of journals ranked by JCR Immediacy Index, representing 11.9% of EPAs Water Quality journal publications. This number is 1.2 times higher than expected.
4. There were 4 hot papers among the 867 Water Quality publications. Using the hot paper thresholds established by ESI as a benchmark, 4 (0.5%) hot papers were identified in the analysis. This number is five times the number expected. Hot papers are publications that are highly cited shortly after they are published.
5. The authors of the Water Quality journal publications cite themselves much less than the average author. 482 of the 8,697 total cites are author self-cites. This 5.5% author self-citation rate is well below the accepted range of 10-30% author self-citation rate.
6. 25 of the 1,716 authors of the Water Quality journal publications are included in ISIHighlyCited.com,which is a database of the worlds most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999.
7. 1 patent was issued to investigators from 1998 to 2008 for research that was conducted under EPAs Water Quality Research Program. The patent was not cited by any other patents.
8. The 109 non-journal publications were cited 279 times in journals and there were no author self cites. When applying the ESI benchmark for journal publications to these 109 non-journal publications, 2 (1.8%) of the publications are highly cited when using the ESI criteria for the top 10% of papers. None of the non-journal publications meet the criteria for highly cited when using the ESI thresholds for the top 1%, 0.1%, or 0.01%.

Highly Cited Water Quality Publications

All of the journals covered by ESI are assigned a field, and to compensate for varying citation rates across scientific fields, different thresholds are applied to each field. Thresholds are set to select highly cited papers to be listed in ESI. Different thresholds are set for both field and year of publication. Setting different thresholds for each year allows comparable representation for older and younger papers for each field.

The 867 Water Quality journal publications reviewed for this analysis were published in journals that were assigned to 15 of the 22 ESI fields. The distribution of the papers among these 15 fields and the number of citations by field are presented in Table 1.

Table 1. Water Quality Journal Publications by ESI Fields

ESI Field	No. of Citations	No. of EPA WQ Papers	Average Cites/Paper
Agricultural Sciences	25	6	4.2
Biology & Biochemistry	137	16	8.6
Chemistry	265	29	9.1
Clinical Medicine	211	13	16.2
Computer Science	1	3	0.3
Engineering	98	21	4.7
Environment/Ecology	4,492	439	10.2
Geosciences	231	18	12.8
Microbiology	467	50	9.3
Molecular Biology & Genetics	0	1	0
Multidisciplinary	147	4	36.8
Pharmacology & Toxicology	162	7	23.1
Physics	12	1	12.0
Plant & Animal Science	2,432	254	9.6
Social Sciences	17	5	3.4
Total = 15	Total = 8,697	Total = 867	10.0

There are 132 (15.2% of the 867 journal papers analyzed) highly cited Water Quality journal publications in 11 of the 15 fieldsAgricultural Sciences, Chemistry, Clinical Medicine, Engineering, Environment/Ecology, Geosciences, Microbiology, Multidisciplinary, Pharmacology & Toxicology, Plant & Animal Science, and Social Scienceswhen using the ESI criteria for the top 10% of papers. Table 2 shows the number of Water Quality journal publications in those 11 fields that meet the top 10% threshold in ESI.

Table 2. Number of Highly Cited Water Quality Journal Publications by Field (top 10%)

ESI Field	No. of Citations	No. of Papers	Average Cites/Paper	% of WQ Papers in Field
Agricultural Sciences	19	1	19.0	16.7%
Chemistry	92	3	30.7	10.3%
Clinical Medicine	98	1	98.0	7.7%
Engineering	45	5	9.0	23.8%
Environment/Ecology	2,082	51	40.8	11.6%
Geosciences	76	2	38.0	11.1%
Microbiology	253	7	36.1	14.3%
Multidisciplinary	147	4	36.8	100.0%
Pharmacology& Toxicology	134	2	67.0	28.6%
Plant & Animal Science	1,429	55	26.0	21.6%
Social Sciences	4	1	4.0	20.0%
TOTALS	Total = 4,379	Total = 132	33.2	15.3%

Nine (1.0%) of the journal publications analyzed qualify as highly cited when using the ESI criteria for the top 1% of papers. This is the number of papers expected to meet this threshold. These nine publications are in two of the ESI fieldsEnvironment/Ecology and Plant & Animal Science (see Table 3). The citations for these papers are provided in Tables 4 and 5. One (0.1%) of the Water Quality journal publications meets the top 0.1% ESI threshold for highly cited papers, which is the number expected to meet this threshold. This publication is listed in Table 6. One (0.1%) of the Water Quality journal publications actually meets the top 0.01% threshold in ESI, which is 10 times the expected number of publications to meet this threshold for this program.

Table 3. Number of Highly Cited Water Quality Journal Publications by Field (top 1%)

Table 4. Highly Cited Water Quality Journal Publications in the Field of Environment/Ecology (top 1%)

Table 5. Highly Cited Water Quality Journal Publications in the Field of Plant & Animal Science (top 1%)

Table 6. Very Highly Cited Water Quality Journal Publications (top 0.1%)

Table 7. Extremely Highly Cited Water Quality Journal Publications (top 0.01%)

Ratio of Actual Cites to Expected Citation Rates

The expected citation rate is the average number of cites that a paper published in the same journal in the same year and of the same document type (article, review, editorial, etc.) has received from the year of publication to the present. Using the ESI average citation rates for papers published by field as the benchmark, in 9 of the 15 fields in which the EPA Water Quality journal papers were published, the ratio of actual to expected cites is greater than 1, indicating that the Water Quality journal publications are more highly cited than the average papers in those fields (see Table 8). For all 15 fields combined, the ratio of total number of cites to the total number of expected cites (8,697 to 6,826.3) is 1.3, indicating that the Water Quality journal publications are more highly cited than the average paper.

Table 8. Ratio of Actual Cites to Expected Cites for Water Quality Journal Publications by Field

JCR Benchmarks

Impact Factor. The JCR Impact Factor is a well known metric in citation analysis. It is a measure of the frequency with which the average article in a journal has been cited in a particular year. The Impact Factor helps evaluate a journals relative importance, especially when compared to others in the same field. The Impact Factor is calculated by dividing the number of citations in the current year to articles published in the 2 previous years by the total number of articles published in the 2 previous years.

Table 9 indicates the number of Water Quality journal publications published in the top 10% of journals, based on the JCR Impact Factor. One hundred twenty (120) of 867 journal papers were published in the top 10% of journals, representing 13.8% of EPAs Water Quality journal publications. This indicates that about one-seventh of the Water Quality journal publications are published in the highest quality journals as determined by the JCR Impact Factor, which is 1.4 times higher than the expected percentage.

Table 9. Water Quality Journal Publications in Top 10% of Journals by JCR Impact Factor

Immediacy Index. The JCR Immediacy Index is a measure of how quickly the average article in a journal is cited. It indicates how often articles published in a journal are cited within the year they are published. The Immediacy Index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year.

Table 10 indicates the number of Water Quality journal publications published in the top 10% of journals, based on the JCR Immediacy Index. One hundred three (103) of the 867 papers appear in the top 10% of journals, representing 11.9% of the Water Quality journal papers. This indicates that nearly one-eighth of the Water Quality journal papers are published in the highest quality journals as determined by the JCR Immediacy Index, which is 1.2 times higher than the expected percentage.

Table 10. Water Quality Journal Publications in Top 10% of Journals by JCR Immediacy Index

Hot Papers

ESI establishes citation thresholds for hot papers, which are selected from the highly cited papers in different fields, but the time frame for citing and cited papers is much shorterpapers must be cited within 2 years of publication and the citations must occur in a 2-month time period. Papers are assigned to 2-month periods and thresholds are set for each period and field to select 0.1% of papers.

Using the hot paper thresholds established by ESI as a benchmark, 4 hot papers, representing 0.5% of the Water Quality papers, were identified in three fieldsEngineering, Environment/Ecology, and Plant & Animal Science. The number of Water Quality hot papers is 5 times higher than expected. The hot papers are listed in Table 11.

Table 11. Hot Papers Identified Using ESI Thresholds

Field	ESI Hot Papers Threshold	No. of Cites in 2-Month Period	Paper
Engineering	5	6 cites in August-September 2005	Cardoza LA, et al. Separations coupled with NMR detection. TRAC-Trends in Analytical Chemistry 2003;22(10):766-775.
Environment/ Ecology	7	11 cites in March-April 2008	Danz NP, et al. Integrated measures of anthropogenic stress in the US Great Lakes basin. Environmental Management 2007;39(5):631-647.
Plant & Animal Science	4	5 cites in October 2001	Burkholder JM, et al. Overview and present status of the toxic Pfiesteria complex (Dinophyceae). Phycologia 2001;40(3):186-214.
	4	10 cites in October 2001	Glasgow HB, et al. A second species of ichthotoxic Pfiesteria (Dinamoebales, Dinophyceae). Phycologia 2001;40(3):234-245.

Author Self-Citation

Self-citations are journal article references to articles from that same author (i.e., the first author). Because higher author self-citation rates can inflate the number of citations, the author self-citation rate was calculated for the Water Quality papers. Of the 8,697 total cites of the 867 journal publications, 482 are author self-citesa 5.5% author self-citation rate. Garfield and Sher³ found that authors working in research-based disciplines tend to cite themselves on the average of 20% of the time. MacRoberts and MacRoberts⁴ claim that approximately 10-30% of all the citations listed fall into the category of author self-citation. Kovacic and Misak⁵ reported a 20% author self-citation rate for medical literature. Therefore, the 5.5% self-cite rate for the Water Quality papers is well below the range for author self-citation.

Highly Cited Researchers

A search of Thomsons ISIHighlyCited.com revealed that 25 (1.5%) of the 1,716 authors of the Water Quality papers are highly cited researchers. ISIHighlyCited.com is a database of the worlds most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999. The highly cited researchers identified during this analysis of the Water Quality publications are presented in Table 12.

Table 12. Highly Cited Researchers Authoring Water Quality Journal Publications

Patents

There was one patent issued to investigators from 1998 to 2008 for research that was conducted under EPAs Water Quality Research Program. The patent was not cited by any other patents (see Table 13).

Table 13. Patents from the Water Quality Research Program (1998-2008)

Non-Journal Publications (Reports, Books, and Book Chapters)

One hundred eleven reports, book chapters, and other non-journal publications produced by the Water Quality Research Program from 1999 to 2008 were included in the analysis. The 109 non-journal publications were cited 279 times in journals and there were no author self cites. When applying the ESI benchmark for journal publications to these 109 non-journal publications, 2 (1.8%) EPA publications (i.e., The Stressor Identification Technical Guidance Document, EPA/822/B00/025; and Microbial Source Tracking Guide Document, EPA/600/R-05/064) were highly cited when using the ESI criteria for the top 10% of papers. None of the non-journal publications met the criteria for highly cited when using the ESI thresholds for the top 1%, 0.1%, or 0.01%.

¹ Thomson Scientifics Web of Science provides access to current and retrospective multidisciplinary information from approximately 8,830 of the most prestigious, high impact research journals in the world. Web of Science also provides cited reference searching.

² Scopus is a large abstract and citation database of research literature and quality Web sources designed to support the literature research process. Scopus offers access to 15,000 titles from 4,000 different publishers, more than 12,850 academic journals (including coverage of 535 Open Access journals, 750 conference proceedings, and 600 trade publications), 27 million abstracts, 245 million references, 200 million scientific Web pages, and 13 million patent records.

³Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. American Documentation 1963;18(July):195-210.

⁴ MacRoberts MH, MacRoberts BR. Problems of citation analysis: a critical review. Journal of the American Society of Information Science 1989;40(5):342-349.

⁵ Kovacic N, Misak A. Author self-citation in medical literature. Canadian Medical Association Journal 2004;170(13):1929-1930.

Top of page