Bibliometrics

2009 Bibliometric Analysis for the U.S. Environmental Protection Agency/Office of Research and Developments Endocrine Disrupting Chemicals (EDCs) Research Program
February 2009

This is a bibliometric analysis of the papers prepared by intramural and extramural researchers of the U.S. Environmental Protection Agency’s (EPA) Endocrine Disrupting Chemicals (EDCs) Research Program.  For this analysis, 698 (652 journal articles and 46 non-journal publications) were reviewed, and they were published from 1998 to 2008. The journal articles were cited 12,895 times and the non-journal publications were cited 180 times in the journals covered by Thomson’s Web of Science¹ and Scopus².Of the 652 journal articles, 598 (91.7%) have been cited at least once in a journal. Of the 46 non-journal publications, 20 (43.5%) have been cited at least once in a journal.

Searches of Thomson Scientific’s Web of Science and Elsevier’s Scopus were conducted to obtain times cited data for the EDCs journal publications.  The analysis was completed using Thomson’s Essential Science Indicators (ESI) and Journal Citation Reports (JCR) as benchmarks. ESI provides access to a unique and comprehensive compilation of essential science performance statistics and science trends data derived from Thomson’s databases. For this analysis, the ESI highly cited papers thresholds as well as the hot papers thresholds were used to assess the influence and impact of the EDCs papers. JCR is a recognized authority for evaluating journals.  It presents quantifiable statistical data that provide a systematic, objective way to evaluate the world’s leading journals and their impact and influence in the global research community. The two key measures used in this analysis to assess the journals in which the EPA EDCs papers are published are the Impact Factor and Immediacy Index. The Impact Factor is a measure of the frequency with which the “average article” in a journal has been cited in a particular year.  The Impact Factor helps evaluate a journal’s relative importance, especially when compared to other journals in the same field.  The Immediacy Index is a measure of how quickly the “average article” in a journal is cited.  This index indicates how often articles published in a journal are cited within the same year and it is useful in comparing how quickly journals are cited. 

The report includes a summary of the results of the bibliometric analysis, an analysis of the 652 EDCs journal papers analyzed by ESI field (e.g., Environment/Ecology, Pharmacology & Toxicology, Plant & Animal Science), an analysis of the journals in which the EDCs papers were published, a table of the highly cited researchers in the EDCs Research Program, an analysis of the non-journal publications, and any patents that have resulted from the program.

SUMMARY OF RESULTS

1. More than one-quarter of the EDCs publications are highly cited papers.  181 (27.8%) of the EDCs papers qualify as highly cited when using the ESI criteria for the top 10% of highly cited publications.  This is 2.8 times the 10% of papers expected to be highly cited.  27 (4.1%) of the EDCs papers qualify as highly cited when using the ESI criteria for the top 1%, which is 4.1 times the number expected.  3 (0.5%) of these papers qualify as very highly cited when using the criteria for the top 0.1%, which is 5 times the number anticipated.  2 (0.3%) papers actually meet the 0.01% threshold for the most highly cited papers, which is 30 times the expected number for this program.
2. The EDCs papers are more highly cited than the average paper.  Using the ESI average citation rates for papers published by field as the benchmark, in 12 of the 16 fields in which the 562 EDCs journal papers were published, the ratio of actual to expected cites is greater than 1, indicating that the EDCs papers are more highly cited than the average papers in those fields. For all 16 fields combined, the ratio of total number of cites to the total number of expected cites (12,895 to 6,667) is 1.9, indicating that the EDCs papers are more highly cited than the average paper.
3. Nearly one-half of the EDCs papers are published in high impact journals ranked by Impact Factor.  272 of the 652 papers were published in the top 10% of journals ranked by JCR Impact Factor, representing 41.7% of the EDCs journal papers. This number is 4.2 times higher than the expected 65 papers. 184 of the 652 papers appear in the top 10% of journals ranked by JCR Immediacy Index, representing 28.2% of the EDCs journal papers. This number is 2.8 times higher than the expected 65 papers.
4. Six of the EDCs journal papers qualify as hot papers.  Using the hot paper thresholds established by ESI as a benchmark, 6 hot papers, representing 0.9% of the EDCs papers, were identified in the analysis.  Hot papers are papers that were highly cited shortly after they were published. The number of EDCs hot papers identified is 9 times higher than the expected 0.6 hot papers.
5. The authors of the EDCs papers cite themselves much less than the average author.  544 of the 12,895 cites are author self-cites. This 4.2% author self-citation rate is well below the accepted range of 10-30% author self-citation rate.
6. Twenty-one of the authors of the EDCs papers are included in ISIHighlyCited.com,which is a database of the world’s most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999.
7. No patents were issued and no patent applications were filed by investigators from 1997 to 2007 for research that was conducted under EPA’s EDCs research program.
8. The 46 nonjournal publications were cited 180 times in journals.  20 (43.5%) were cited at least once in a journal and the authors cited themselves 9 times (5.0% self-citation rate), which is much less than the literature-reported 10-30% range for author self-citation.  2(4.4%) of the nonjournal publications were highly cited when using the ESI criteria for the top 10% of highly cited publications.

Highly Cited EDCs Publications

All of the journals covered by ESI are assigned a field, and to compensate for varying citation rates across scientific fields, different thresholds are applied to each field.  Thresholds are set to select highly cited papers to be listed in ESI.  Different thresholds are set for both field and year of publication. Setting different thresholds for each year allows comparable representation for older and younger papers for each field.

The 652 EDCs research papers reviewed for this analysis were published in journals that were assigned to 16 of the 22 ESI fields.  The distribution of the papers among these 16 fields and the number of citations by field are presented in Table 1.

Table 1. EDCs Papers by ESI Fields

There are 181 (27.8% of the papers analyzed) highly cited EPA EDCs papers in 12 of the 16 fields—Agricultural Sciences, Biology & Biochemistry, Chemistry, Clinical Medicine, Computer Science, Engineering, Environment/Ecology, Multidisciplinary, Neuroscience & Behavior, Pharmacology & Toxicology, Plant & Animal Science, and Social Sciences—when using the ESI criteria for the top 10% of papers.  Table 2 shows the number of EPA papers in those 12 fields that meet the top 10% threshold in ESI.  Twenty-seven (4.1%) of the papers analyzed qualify as highly cited when using the ESI criteria for the top 1% of papers.  These papers cover 6 fields—Biology & Biochemistry, Clinical Medicine, Engineering, Environment/Ecology, Multidisciplinary, and Pharmacology & Toxicology.  Table 3 shows the 27 papers by field that meet the top 1% threshold in ESI.  The citations for these 27 papers are provided in Tables 4 through 9.  There were 3 (0.5%) very highly cited EDCs papers in the fields of Environment/Ecology and Multidisciplinary (see Table 10).  These papers, which met the top 0.1% threshold in ESI, are listed in Table 11.  Two (0.3%) of the EDCs papers met the top 0.01% threshold in ESI (see Table 12), which is 30 times the expected number of papers that should meet this threshold for this size program. These papers are listed in Table 13.

Table 2. Number of Highly Cited EDCs Papers by Field (top 10%)

Table 3. Number of Highly Cited EDCs Papers by Field (top 1%)

Table 4. Highly Cited EDCs Papers in the Field of Biology & Biochemistry (top 1%)

Table 5. Highly Cited EDCs Papers in the Field of Clinical Medicine (top 1%)

Table 6. Highly Cited EDCs Papers in the Field of Engineering (top 1%)

Table 7. Highly Cited EDCs Papers in the Field of Environment/Ecology (top 1%)

Table 8.  Highly Cited EDCs Papers in the Field of Multidisciplinary (top 1%)

Table 9.  Highly Cited EDCs Papers in the Field of Pharmacology & Toxicology (top 1%)

Table 10. Number of Very Highly Cited EDCs Papers by Field (top 0.1%)

Table 11.  Very Highly Cited EDCs Papers (top 0.1%)

Table 12.  Number of Extremely Highly Cited EDCs Papers by Field (top 0.01%)

Table 13.  Extremely Highly Cited EDCs Papers (top 0.01%)

Ratio of Actual Cites to Expected Citation Rates

The expected citation rate is the average number of cites that a paper published in the same journal in the same year and of the same document type (article, review, editorial, etc.) has received from the year of publication to the present.  Using the ESI average citation rates for papers published by field as the benchmark, in 12 of the 16 fields in which the EPA EDCs papers were published, the ratio of actual to expected cites is greater than 1, indicating that the EDCs papers are more highly cited than the average papers in those fields (see Table 14).  For all 16 fields combined, the ratio of total number of cites to the total number of expected cites (12,895 to 6,667) is 1.9, indicating that the EDCs papers are more highly cited than the average paper.

Table 14.  Ratio of Actual Cites to Expected Cites for EDCs Papers by Field

JCR Benchmarks

Impact Factor.  The JCR Impact Factor is a well known metric in citation analysis.  It is a measure of the frequency with which the “average article” in a journal has been cited in a particular year.  The Impact Factor helps evaluate a journal’s relative importance, especially when compared to others in the same field.  The Impact Factor is calculated by dividing the number of citations in the current year to articles published in the 2 previous years by the total number of articles published in the 2 previous years. 

Table 15 indicates the number of EDCs papers published in the top 10% of journals, based on the JCR Impact Factor.  Two hundred seventy-two (272) of 652 papers were published in the top 10% of journals, representing 41.7% of EPA’s EDCs papers. This indicates that nearly one-half of the EDCs papers are published in the highest quality journals as determined by the JCR Impact Factor, which is 4.2 times higher than the expected percentage.

Table 15.  EDCs Papers in Top 10% of Journals by JCR Impact Factor

Immediacy Index.  The JCR Immediacy Index is a measure of how quickly the average article in a journal is cited.  It indicates how often articles published in a journal are cited within the year they are published.  The Immediacy Index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year.

Table 16 indicates the number of EDCs papers published in the top 10% of journals, based on the JCR Immediacy Index.  One hundred eighty-four (184) of the 652 journal articles appear in the top 10% of journals, representing 28.2% of the EDCs papers. This indicates that more than one-quarter of the EDCs papers are published in the highest quality journals as determined by the JCR Immediacy Index, which is 2.8 times higher than the expected percentage.

Table 16.  EDCs Papers in Top 10% of Journals by JCR Immediacy Index

Hot Papers

ESI establishes citation thresholds for hot papers, which are selected from the highly cited papers in different fields, but the time frame for citing and cited papers is much shorter—papers must be cited within 2 years of publication and the citations must occur in a 2-month time period.  Papers are assigned to 2-month periods and thresholds are set for each period and field to select 0.1% of papers.  Using the hot paper thresholds established by ESI as a benchmark, 6 hot papers, representing 0.9% of the EDCs papers, were identified in three fields—Environment/Ecology, Multidisciplinary, and Pharmacology & Toxicology.  The number of EDCs hot papers is 9 times higher than expected. The hot papers are listed in Table 17.

Table 17.  Hot Papers Identified Using ESI Thresholds

Author Self-Citation

Self-citations are journal article references to articles from that same author (i.e., the first author).  Because higher author self-citation rates can inflate the number of citations, the author self-citation rate was calculated for the EDCs papers.  Of the 12,895 total cites, 544 are author self-cites—a 4.2% author self-citation rate.  Garfield and Sher³ found that authors working in research-based disciplines tend to cite themselves on the average of 20% of the time. MacRoberts and MacRoberts⁴ claim that approximately 10% to 30% of all the citations listed fall into the category of author self-citation. Kovacic and Misak⁵ recently reported a 20% author self-citation rate for medical literature.Therefore, the 4.2% self-cite rate for the EDCs papers is well below the range for author self-citation.

Highly Cited Researchers

A search of Thomson’s ISIHighlyCited.com revealed that 21 (1.5%) of the 1,379 authors of the EDCs papers are highly cited researchers.  ISIHighlyCited.com is a database of the world’s most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999. The highly cited researchers identified during this analysis of the EDCs publications are presented in Table 18.

Table 18.  Highly Cited Researchers Authoring EDCs Publications

Patents

No patents have been issued or patent applications filed by investigators from 1998 to 2008 for research that was conducted under EPA’s EDCs Research Program.

Nonjournal Publications (Books, Book Chapters, Reports, and Proceedings)

Forty-six nonjournal publications (books, book chapters, reports, and proceedings) produced by the program from 1998 to 2008 were included in the analysis. Of these 46 nonjournal publications, 20 (43.5%) have been cited at least once in a journal. The 46 publications were cited 180 times in the journals covered by Thomson Reuter’s ISI Web of Science. The authors of the nonjournal publications cited themselves 9 times, a 5.0% self citation rate, which is well below the 10-30% range report in the literature for author self-citation. Application of the ESI fields and highly cited benchmarks used for journal papers to the nonjournal publications, indicated that 2 (4.4%) of the nonjournal publications were highly cited when using the ESI criteria for the top 10% of highly cited publications.  None of the nonjournal publications met the ESI criteria for the top 1%, 0.1%, or 0.01% highly cited publications.

¹Thomson Scientific’s Web of Science provides access to current and retrospective multidisciplinary information from approximately 8,830 of the most prestigious, high impact research journals in the world. Web of Science also provides cited reference searching.

²Scopus is a large abstract and citation database of research literature and quality Web sources designed to support the literature research process. Scopus offers access to 15,000 titles from 4,000 different publishers, more than 12,850 academic journals (including coverage of 535 Open Access journals, 750 conference proceedings, and 600 trade publications), 27 million abstracts, 245 million references, 200 million scientific Web pages, and 13 million patent records.

³ Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. American Documentation 1963;18(July):195-210.

⁴ MacRoberts MH, MacRoberts BR. Problems of citation analysis: a critical review. Journal of the American Society of Information Science 1989;40(5):342-349.

⁵ Kavaci N, Misak A. Author self-citation in medical literature. Canadian Medical Association Journal 2004;170(13):1929-1930.

Top of page