Durham Leading Research Programme: Academic Impact

Contact Details
Academic Liaison Librarian (Researcher Support)
james.bisset@durham.ac.uk
Phone: +44 (191) 334 2961
Leading Research Module 15:
Academic Impact: Research Indicators

Learning Outcomes for the session
• Increased awareness of the various journal and author
metrics available.
• Understanding of how some key indicators (JIF, H-index)
are calculated
• Understanding of the key issues around the use of these
metrics and what research behaviours might be
incentivised.
• Awareness of the potential opportunities for exploring
wider academic and non-academic impact of publications
from altmetric tools available.

Session outline
• Introduction: What can you measure?
• Citations
• Author metrics
• Journal metrics
• Metrics and the REF
• Altmetrics (in brief)

Quick Survey (1)
• How many of you have been advised to publish
in certain journals (eg “high impact journals”)?
• How many of you are aware of if and how any
citation metrics were used in the last REF?

Quick Survey (2)
• What does it mean if an article has a citation
count of 97?
• What does it mean if an author has an h-index
of 14?
• What does it mean if a journal has a Journal
Impact Factor of 1.317?
• What does it mean if an article has an altmetric
score (on altmetric.com) of 2734?

What can you measure?
• Article/Book impact
• One measure of the impact of
individual journal articles,
conference proceedings or books,
is the number of times they are
cited by other works.

• Publication (Journal) impact
• The perceived impact of a specific
academic journal might be
assessed by the number of times
their articles are cited (on average)
and where they are cited.

• Researcher impact
• The number of outputs and citation
a researcher generates can be an
indicator for the impact of an
individual researcher.

• Institutional impact
• The prestige of a department or
area of research within an
institution compared to those at
other institutions can be measured
by the sum of individual
researchers ‘impact’.

Are bibliometrics
effective & reliable
measures of
academic impact?

Part 1
Citations
and
Citation
Indices

Citations
• Links between papers that have
something in common

Citations
something in common
• Building upon, or challenging, research

Citations
something in common
• Help make a judgement about impact
an article has made

Citations
something in common
• Help make a judgement about impact
an article has made
• Sum of citations can be an indication of
the ‘impact’ of an author’s work / a
journal as a collection of articles

... The problem...
• How do you identify who has cited a
publication you have written or read?

Citation indices
• 1955: Eugene Garfield proposed
creating a citation index for science to...

Citation indices
• 1955: Eugene Garfield proposed
creating a citation index for science to...
“eliminate the uncritical citation of fraudulent,
incomplete or obsolete data by making it
possible for the conscientious scholar to be
aware of criticisms of earlier papers”
Garfield, E (1955) ‘Citation Indexes for Science’
Science, New Series, Vol. 122, No. 3159, pp. 108-111

Citation indices
• 1955: Eugene Garfield - presents the idea of
measuring the ‘impact’ of journal articles
using citations

Citation indices
using citations
• 1960s: Science Citation Index developed to
highlight “formal, explicit linkages between
papers that have particular points in common”

Citation indices
using citations
• 1960s: Science Citation Index developed to
highlight “formal, explicit linkages between
papers that have particular points in common”
• 1975: Journal Citation Reports use Web of
Science data to rank journals within
disciplines

Citation indices
(2008) Taylor and Francis LibSite Newsletter,
issue 9. p. 5
Science subjects
Social-science subjects

Citation indices
“reference lists are held under copyright by
academic publishers which makes tracking
citations impossible”
The death of the reference and the re-use factor (2013)
http://figshare.com/blog/The_Death_Of_The_Reference_and_the_reuse_factor/103

Citation indices
• Web of Science (Thomson Reuters)
• Sci-verse Scopus (Elsevier)
• Google Scholar

Things you can do
• Count citations to an article
• Link to other related articles
• Citation mapping
• Set up citation alerts
• Search for cited references
• See citation reports for authors and journals

Demo
Web of Science /
Google Scholar

Over to you
• Check Google Scholar and Web of Science for an
article you have already written/read or are aware
of, or try:
Zong, Y. and Tooley, M. J. (2003) 'A historical record of coastal
floods in Britain : frequencies and associated storm tracks.',
Natural hazards, 29 (1). pp. 13-36.
• Explore and compare the number of citations (if
any) listed
Citation searching
Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available
at http://www.flickr.com/photos/theblackcanvas/2945878325/

Citation Metrics
h-index (Hirsch, 2005)
• an author’s number of articles (h) that have
received at least h citations
• a researcher with an h-index of 10 has published
10 articles that have each been cited 10+ times
g-index (Egghe, 2006)
• the highest number (g) of papers that together
received g2 or more citations
• a researcher with a g-index of 10 has published 10
papers that, in total, have been cited at least 100
times

h-index
Author: Smith, J
Has written and published 9 articles (a-i),
which have been cited as follows:
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
“no. of articles (n) that have received at least n citations”

h-index
Author: Smith, J
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
H-index: 3 (at least 3 References with 3 or
more citations)

h-index
Author: Smith, J
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
H-index: not 4 (only 3 References with 4 or
more citations)

h-index: what’s in a number?
• Nobel Prize Winner 2013, Peter W
Higgs
• H-index (Google Scholar) = 12
• H-index (Web of Science) = 11

g-index
Author: Smith, J
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
“the highest number (g) of papers that together have received
g2 or more citations”

g-index
Author: Smith, J
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
G-index: 5 (5x5 = 25... Total of top 5 cited
articles citations = 31)

g-index
Author: Smith, J
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
G-index: not 6 (6x6 = 36... Total of top 6 cited
articles citations = 34)

Over to you
• If you have published 1 or more papers, and if all
are indexed on Web of Science or Google Scholar,
see if you can calculate and compare your h-index
(or try to calculate your g-index)
• ... or try the same for a colleague, supervisor or
head of department.
Author metrics

Google Scholar – My Citations
• Track citations to your publications
• Check who is citing your publications. Graph your
citations over time. Calculate citation metrics.
• View publications by colleagues or co-authors
• Keep up with their work, view their citation
metrics.
• Appear in Google Scholar Search Results
• Create a public profile that can appear in Google
Scholar when someone searches for your name.

Author metrics – Issues?
• Author identification
e.g. Professor Gordon Love
A name is not unique
o Prof. Gordon Love, University of California (Earth Science)
o Dr Gordon L Love, Sacramento (Medicine and Health)
o Prof. Gordon Love, Durham University (Physics)

Author metrics – Issues?
• Author identification
e.g. Professor Gordon Love
... So you need a unique identifier (or 3)
o ORCID profile (0000-0001-5137-9434)
o Researcher ID profile (A-3071-2011)
o Google Scholar profile (3xJXtlwAAAAJ)

Journal Citation Reports (JCRs)
• JCRs – annual publication of journals and
their impact factors.
• Over 10,800 titles, across 232 disciplines
have JIFs in 2015 editions
• A journal that is cited once, on average, for
each article published has an JIF of 1.

Journal Citation Reports (JCRs)
• 2015 edition (published June 2015):
• is JCR year 2014 (providing 2014 Journal
Impact Factor (JIF) metrics)
• counting citation data from 2014
• for articles published in 2012/13

Journal Impact Factor (JIF)
Citations in 2014 to all
articles published by Journal
X in 2012 & 2013
Journal
X’s
2014
impact
factor
=

Citations in 2014 to all
articles published by Journal
X in 2012 & 2013
Number of articles that
were published in Journal X
in 2012 & 2013
Journal
X’s
2014
impact
factor
=

Citations in 2014 (in journals
indexed in Web of Science) to
all articles published by
Journal X in 2012 & 2013
Number of articles (of a type
deemed to be citeable) that
were published in Journal X
in 2012 & 2013
Journal
X’s
2014
impact
factor
=

Demo
JCRs
Journal Impact Factors

Other journal impact metrics
• Eigenfactor
• Calculates weightings based on ‘where’
the citation came from
• Also takes into account the ‘size’ of a
journal, to measure the ‘total importance of
a journal’
• http://www.eigenfactor.org/ (1997-2013)

Other journal impact metrics
• SCImago
• Based on data from Scopus (rather than
Web of Science)
• http://www.scimagojr.com/journalrank.php

Over to you
• Explore the JCRs to identify relevant subject
categories for your field of research.
• Identify journals and compare JIFs and Eigenfactor
Scores
Journal metrics

Issues with journal metrics
• No comprehensive data source
• Citation cultures vary across disciplines
• Publication cultures vary across disciplines
• Research careers have different stages
• Citations ≠ confirmation of excellence
• Scholarly communication is evolving…
• Blogs, twitter, data and open access
repositories

Metrics for Research Assessment
• REF2014
• 11 of 36 panels assessing submissions looked at
citation data
• Benchmarked against data for comparable articles
• Did not take into account Journal Impact Factors in
assessment (?)
• REF(2020?)
• Consulting on use of metrics
• “The Metric Tide” (9th July 2015)

“Placing too much emphasis on narrow,
poorly-designed indicators – such as journal
impact factors (JIFs) – can have negative
consequences”
“… correlation analysis of the REF2014
results ... Has shown that individual metrics
give significantly different outcomes from the
REF peer review process.”

“At an institutional level, HEI leaders should develop
a clear statement of principles … including the role
of quantitative indicators.”
“Individual researchers should be mindful of the
limitations of particular indicators in the the way they
present their own CVs and evaluate the work of
colleagues.”
“Publishers should reduce the emphasis on journal
impact factors as a promotional tool … [and] should
also make available a range of article-level metrics”

Journal Impact Factors and Citations
• “… the JIF is the main determinant of article citation
impact”
Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?
Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at
http://dx.doi.org/10.1016/j.joi.2013.08.006
• “… the impact factor of the journal in which articles
appeared was the primary predictor of the
citations that they accrued”
Haslam, N. and Koval, P. (2010) “Predicting long-term citation impact in articles in social and personality
psychology” Psychological Reports 106(3) Jun 2010. Available at http://dx.doi.org/10.2466/pr0.106.3.891-900

• “…a small but significant correlation between journal
rank and future citations can be observed … stems
from visibility effects due to the influence of the IF on
reading habits (Lozano et al., 2012), rather than from
factors intrinsic to the published articles … [but] the
correlation is so weak that it cannot alone account
for the strong correlation between retractions and
journal rank”
Björn Brembs, Katherine Button and Marcus Munafò (2013) “Deep impact: unintended consequences of journal
rank” Frontiers in Human Neuroscience 7: 291. Available at http://dx.doi.org/10.3389/fnhum.2013.00291
Citing
Lozano, G. A., Larivière, V., and Gingras, Y. (2012). “The weakening relationship between the impact factor and
papers' citations in the digital age” J. Am. Soc. Inf. Sci. Technol. 63, 2140–2145. Available at
http://dx.doi.org/10.1002/asi.22731

Fang, F. C., & Casadevall, A. (2011). Retracted Science and the Retraction Index
. Infection and Immunity, 79(10), 3855–3859. http://doi.org/10.1128/IAI.05661-11

Place of Publication: Open Access
• Boost the potential visibility and access of research,
to a wider audience
• Removes research from behind paywall barriers for
existing audience
• Enables authors to retain their rights and more easily
share via social media, email etc.

• 4633 articles across ecology, applied
mathematics, sociology and economics.
• 2280 were open access, and had an
average citation count of 9.04
• 2353 were subscriptions journals, and had
an average citation count of 5.76.
Norris, M. (2008) “The citation advantage of open access articles” Thesis. Available at
https://dspace.lboro.ac.uk/2134/4089

SPARC Europe: Open Access Citation Advantage Service
http://sparceurope.org/oaca/
Total number of studies so far 70
Studies that found a citation advantage 46
Studies that found no citation advantage 17
Studies that were inconclusive, found non-
significant data or measured other
things than citation advantage for articles
7

SPARC Europe: Open Access Citation Advantage Service
http://sparceurope.org/oaca/
Total number of studies so far 70
Studies that found a citation advantage 46
Studies that found no citation advantage 17
Studies that were inconclusive, found non-
significant data or measured other
things than citation advantage for articles
7
Correlation ≠ Causation
evidence of a positive correlation between citation count
and open access (gold or green)…
but…
• [Cause] Because research is more discoverable and
accessible
• [Cause] Because authors choose to make their best
(most citable) research open access

Altmetrics
“Unlike the JIF, altmetrics reflect the impact
of the article itself, not its venue. Unlike
citation metrics, altmetrics will track impact
outside the academy, impact of influential
but uncited work, and impact from sources
that aren’t peer-reviewed.“
http://altmetrics.org/manifesto/

Image Credits
[Slide 6] Via Flickr Creative Commons, by demonsub. Original available at
http://www.flickr.com/photos/66397367@N05/6993902269
[Slide 13] Via Flickr Creative Commons, by Kirsty Andrews. Original
available at http://www.flickr.com/photos/47745961@N08/5169765739
[Slide 31,44,59] Via Flickr Creative Commons, by © Stuti Sakhalkar.
Original at http://www.flickr.com/photos/theblackcanvas/2945878325/
[Slide 49] Via Flickr Creative Commons, by Alan Cleaver. Original available
at http://www.flickr.com/photos/11121568@N06/4122176776/
[Slide 22] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC)
Limited‘ Available at www.vitae.ac.uk/rdf

Measuring
Researcher
DevelopmentVitae Researcher Development Framework [see image credits]

Durham University Library: Supporting your research
Contact Details
Academic Liaison Librarian (Researcher Support)
james.bisset@durham.ac.uk
Phone: +44 (191) 334 2961

Durham Leading Research Programme: Academic Impact

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (8)

Ähnlich wie Durham Leading Research Programme: Academic Impact

Ähnlich wie Durham Leading Research Programme: Academic Impact (20)

Mehr von Jamie Bisset

Mehr von Jamie Bisset (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Durham Leading Research Programme: Academic Impact