There is an ever-increasing need to make your research more visible as you establish your career, and metrics to measure your research performance when it comes to thinking about promotion and probation. This session will focus on bibliometric research indicators (such as the Journal Impact Factor and SCImago, author metrics such as the h-index and g-index) and sources for accessing citation data (Web of Science, Journal Citation Reports and Google Scholar). These may be one of several factors to consider when thinking about where to submit an article manuscript for publication to maximise the potential academic impact of the research, and tools useful to be familiar with if they form part of any research evaluation you and your authored journal papers may be subject to. An additional section will also look at tips to consider when writing an article abstract to maximise its discoverability and cite-ability. Learning Outcomes: • Understanding of meaning and intended uses of bibliometric research indicators • Understanding of how some key indicators (JIF, H-index) are calculated • Ability to make a judgement as to the appropriateness and limitations of such indicators • Ability to use online datasets to view and calculate key bibliometric measures • Awareness of some factors which can increase the visibility and discoverability of your own research in bibliographic databases. Previous participants have said: "The session has helped provide me with the basic information on Journal Impact and where to find information such as an author's h-index. It will be useful for future journal submission consideration." "This session was very useful for me to become familiar with the topic."

1. Contact Details
Academic Liaison Librarian (Researcher Support)
james.bisset@durham.ac.uk
Phone: +44 (191) 334 2961
Measuring Research Impact:
Bibliometric Research Indicators

2. Learning Outcomes for the session
• Understanding the original purpose and intended uses of
bibliometric research indicators
• Understanding of how some key indicators (JIF, H-index)
are calculated
• Ability to make a judgement as to the appropriateness and
limitations of such indicators
• Ability to use online datasets to view and calculate key
bibliometric measures
• Awareness of some factors which can increase the
visibility and discoverability of your own publications.

3. Session outline
• Introduction: What can you measure?
• Citations
• Author metrics
• Journal metrics
• Optimising cite-ability

4. Quick Survey (1)
• How many of you have already published
research papers in journals conference
proceedings?
• How many of you are expected to during the
course of your studies?
• How many of you have already been advised
where to publish, or where not to publish, based
on what are the “best” journals?

6. What can you measure?
• Article/Book impact
• One measure of the impact of
individual journal articles,
conference proceedings or books,
is the number of times they are
cited by other works.

7. What can you measure?
• Publication (Journal) impact
• The perceived impact of a specific
academic journal might be
assessed by the number of times
their articles are cited (on average)
and where they are cited.

8. What can you measure?
• Researcher impact
• The number of outputs and citation
a researcher generates can be an
indicator for the impact of an
individual researcher.

9. What can you measure?
• Institutional impact
• The prestige of a department or
area of research within an
institution compared to those at
other institutions can be measured
by the sum of individual
researchers ‘impact’.

10. Impact ?

11. Are bibliometrics
effective & reliable
measures of
academic impact?

12. Quick Survey (2)
• Web of Science
• JCR
• Sciverse Scopus
• PoP software
• JIF
• Eigenfactor
• SCImagoJR /
SNIP
• H-index / g-index
• altmetrics
• Sherpa ROMEO
• Sherpa JULIET
• Sherpa FACT

13. Part 1
Citations
and
Citation
Indices

14. Citations
• Links between papers that have
something in common
• Building upon, or challenging, research
• Help make a judgement about impact
an article has made
• Sum of citations can be an indication of
the ‘impact’ of an author’s work / a
journal as a collection of articles

15. Citation indices
• 1955: Eugene Garfield proposed
creating a citation index for science to...
“eliminate the uncritical citation of fraudulent,
incomplete or obsolete data by making it
possible for the conscientious scholar to be
aware of criticisms of earlier papers”
Garfield, E (1955) ‘Citation Indexes for Science’
Science, New Series, Vol. 122, No. 3159, pp. 108-111

16. Citation indices
• 1955: Starts with the idea of measuring the
‘impact’ of journal articles using citations
• 1960s: Science Citation Index developed to
highlight “formal, explicit linkages between
papers that have particular points in common”
• 1975: Journal Citation Reports use Web of
Science data to rank journals within
disciplines

17. Citation indices
(2008) Taylor and Francis LibSite Newsletter,
issue 9. p. 5
Science subjects
Social-science subjects

18. Citation indices
“reference lists are held under copyright by
academic publishers which makes tracking
citations impossible”
The death of the reference and the re-use factor (2013)
http://figshare.com/blog/The_Death_Of_The_Reference_and_the_reuse_factor/103

19. Citation indices
• Web of Science
• Provided by Thomson Reuters.
• Includes the Sciences, Social Sciences, Arts &
Humanities & Books Citation Indexes.
• Indexes about 11,800 journals, plus conference
proceedings.
• Approximately 5,000 journals covering arts,
humanities and social sciences.

20. Citation indices
• Sciverse Scopus
• Launched by Elsevier in 2004.
• Main commercial competition to Web of Science
• Currently indexes c19,000 ‘active’ journals plus
conference proceedings
• Many titles covered by both WoS and Scopus

21. Citation indices
• Google Scholar
• Pulls data from a much broader range of documents
(eg books, reports, blogs, wider range of publishers)
• Useful for some subjects not covered by Web of
Science
• Citation data: quality and accuracy often
questionned.

22. Things you can do
• Count citations to an article
• Link to other related articles
• Citation mapping
• Set up citation alerts
• Search for cited references
• See citation reports for authors and journals

23. Demo
Web of Science /
Google Scholar

25. Over to you
• Check Google Scholar and Web of Science for an
article you have already read or are aware of (or
try:
Zong, Y. and Tooley, M. J. (2003) 'A historical record of coastal
floods in Britain : frequencies and associated storm tracks.',
Natural hazards, 29 (1). pp. 13-36.
• Explore and compare the number of citations (if
any) listed
Citation searching
Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available
at http://www.flickr.com/photos/theblackcanvas/2945878325/

27. Citation Metrics
h-index (Hirsch, 2005)
• an author’s number of articles (h) that have
received at least h citations
• a researcher with an h-index of 10 has published
10 articles that have each been cited 10+ times
g-index (Egghe, 2006)
• the highest number (g) of papers that together
received g2 or more citations
• a researcher with a g-index of 10 has published 10
papers that, in total, have been cited at least 100
times

28. h-index
Author: Smith, J
Has written and published 9 articles (a-i),
which have been cited as follows:
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
“no. of articles (n) that have received at least n citations”

29. h-index: what’s in a number?
• Nobel Prize Winner 2013, Peter W
Higgs
• H-index (Google Scholar) = 12
• H-index (Web of Science) = 11

31. g-index
Author: Smith, J
Has written and published 9 articles (a-i),
which have been cited as follows:
a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3
“the highest number (g) of papers that together have received
g2 or more citations”

32. Demo
Web of Science

33. Over to you
• If you have published 1 or more papers, and if all
are indexed on Web of Science or Google Scholar,
see if you can calculate and compare your h-index
(or try to calculate your g-index)
• ... or try the same for a colleague, supervisor or
head of department.
Author metrics
Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available
at http://www.flickr.com/photos/theblackcanvas/2945878325/

34. Google Scholar – My Citations
• Track citations to your publications
• Check who is citing your publications. Graph your
citations over time. Calculate citation metrics.
• View publications by colleagues or co-authors
• Keep up with their work, view their citation
metrics.
• Appear in Google Scholar Search Results
• Create a public profile that can appear in Google
Scholar when someone searches for your name.

35. Demo
Google Scholar
– My citations

36. Author metrics – Issues?
• Author identification
e.g. Professor Gordon Love
A name is not unique
o Prof. Gordon Love, University of California (Earth Science)
o Dr Gordon L Love, Sacramento (Medicine and Health)
o Prof. Gordon Love, Durham University (Physics)

37. Author metrics – Issues?
• Author identification
e.g. Professor Gordon Love
... So you need a unique identifier (or 3)
o ORCID profile (0000-0001-5137-9434)
o Researcher ID profile (A-3071-2011)
o Google Scholar profile (3xJXtlwAAAAJ)

38. Demo
ResearcherID
ORCiD

39. Publish or Perish (PoP) software
• Anne-Wil Harzing (2006), current version
4.25.1 (17th January 2016)
• Aimed at individual researchers
• Analyse own performance using a range of
metrics
• FREE TO DOWNLOAD (Windows, Apple OS
X, GNU/Linux) and FREE TRAINING
MATERIAL
• http://www.harzing.com/pop_win.htm

40. Part 3
Journal
Metrics

41. Journal Citation Reports (JCRs)
• JCRs – annual publication of journals and
their impact factors.
• Over 10,800 titles, across 232 disciplines
have JIFs in 2015 editions
• A journal that is cited once, on average, for
each article published has an JIF of 1.

42. Journal Citation Reports (JCRs)
• 2015 edition (published June 2015):
• is JCR year 2014 (providing 2014 Journal
Impact Factor (JIF) metrics)
• counting citation data from 2014
• for articles published in 2012/13

43. Journal Impact Factor (JIF)
Citations in 2014 to all
articles published by Journal
X in 2012 & 2013
Number of articles that
were published in Journal X
in 2012 & 2013
Journal
X’s
2014
impact
factor
=

44. Journal Impact Factor (JIF)
Citations in 2014 (in journals
indexed in Web of Science) to
all articles published by
Journal X in 2012 & 2013
Number of articles (of a type
deemed to be citeable) that
were published in Journal X
in 2012 & 2013
Journal
X’s
2014
impact
factor
=

45. Journal Ranking

46. Journal Ranking

47. Demo
JCRs
Journal Impact Factors

48. Other journal impact metrics
• Eigenfactor
• Calculates weightings based on ‘where’
the citation came from
• Also takes into account the ‘size’ of a
journal, to measure the ‘total importance of
a journal’
• http://www.eigenfactor.org/ (1997-2013)

49. Other journal impact metrics
• SCImago
• Based on data from Scopus (rather than
Web of Science)
• http://www.scimagojr.com/journalrank.php

50. Over to you
• Explore the JCRs to identify relevant subject
categories for your field of research.
• Identify journals and compare JIFs and Eigenfactor
Scores
Journal metrics
Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available
at http://www.flickr.com/photos/theblackcanvas/2945878325/

51. Issues with journal metrics
• Author identification
• Citation cultures vary across disciplines
• Publication cultures vary across disciplines
• Research careers have different stages
• Citations ≠ confirmation of excellence
• Scholarly communication is evolving…
• Blogs, twitter, data and open access
repositories

53. Optimising the cite-ability of an article
• Title & Abstract
• Discoverability
• Length
• Reference List
• Journal / means of publication
• Journal Impact Factors
• Open Access Publication

54. Title and abstract - Discoverability
“Construct a clear, descriptive title”
• describe what the research is about
• consider what readers might be looking for
“Re-iterate key phrases in the abstract”
• improves potential search ranking
• ‘aids’ human decision-making
Publisher guidelines (Wiley-Blackwell)

55. Title and abstract - Discoverability
Easier to find
=
More likely to be read
=
May translate to increased citations (but still
depends on the quality of the research)

56. “Ever since the British colonists in Australia became aware of the
disappearance of the indigenous peoples in the 1830s, they have
contrived to excuse themselves by pointing to the effects of disease and
displacement. Many colonists called for the extermination of Aborigines
when they impeded settlement by offering resistance, yet there was no
widespread public acknowledgement of this as a policy until the later
1960s, when a critical school of historians began serious investigations
of frontier violence. Their efforts received official endorsement in the
1990s, but profound cultural barriers prevent the development of a
general awareness of this. Conservative and right-wing figures continue
to play down the gravity of what transpired. These two aspects
of Australian public memory are central to the political humanisation of
the country.”
Australia’s Forgotten
Victims

57. Genocide and Holocaust
Consciousness in Australia
“Ever since the British colonists in Australia became aware of the
disappearance of the indigenous peoples in the 1830s, they have contrived
to excuse themselves by pointing to the effects of disease and
displacement. Yet although genocide was not a term used in the nineteenth
century, extermination was, and many colonists called for the extermination
of Aborigines when they impeded settlement by offering
resistance. Consciousness of genocide was suppressed during the
twentieth century until the later 1960s, when a critical school of historians
began serious investigations of frontier violence. Their efforts received
official endorsement in the 1990s, but profound cultural barriers prevent the
development of a general genocide consciousness. One of these
is Holocaust consciousness, which is used by conservative and right-wing
figures to play down the gravity of what transpired in Australia. These two
aspects of Australian public memory are central to the political
humanisation of the country. ”

58. Title and abstract - Length
• Didegah, F. and Thelwall, M. (2013)
- Looked at 16,058 Biology/Biochemistry articles, 16,378
Chemistry articles and 15,392 Social Sciences articles all
covered by Web of Science.
• “abstract length significantly associates with
increased citation impact in all fields”
• “the number of keywords and the title length
statistically associate with decreased citations”
Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?
Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at
http://dx.doi.org/10.1016/j.joi.2013.08.006

59. Title and abstract - Length

60. Reference List
• “The impact [factor] and the number of cited
references are … significant determinants of
increased citation impact”
Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?
Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at
http://dx.doi.org/10.1016/j.joi.2013.08.006
• Consistent with studies in fields including
psychology, medicine, chemistry, physics,
engineering…

61. Place of Publication: JIF
• “… the JIF is the main determinant of article citation
impact”
Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?
Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at
http://dx.doi.org/10.1016/j.joi.2013.08.006
• “… the he impact factor of the journal in which
articles appeared was the primary predictor of the
citations that they accrued”
Haslam, N. and Koval, P. (2010) “Predicting long-term citation impact in articles in social and personality
psychology” Psychological Reports 106(3) Jun 2010. Available at http://dx.doi.org/10.2466/pr0.106.3.891-900

62. Place of Publication: JIF
• “…a small but significant correlation between journal
rank and future citations can be observed … stems
from visibility effects due to the influence of the IF on
reading habits (Lozano et al., 2012), rather than from
factors intrinsic to the published articles … [but] the
correlation is so weak that it cannot alone account for
the strong correlation between retractions and
journal rank”
Björn Brembs, Katherine Button and Marcus Munafò (2013) “Deep impact: unintended consequences of journal
rank” Frontiers in Human Neuroscience 7: 291. Available at http://dx.doi.org/10.3389/fnhum.2013.00291
Citing
Lozano, G. A., Larivière, V., and Gingras, Y. (2012). “The weakening relationship between the impact factor and
papers' citations in the digital age” J. Am. Soc. Inf. Sci. Technol. 63, 2140–2145. Available at
http://dx.doi.org/10.1002/asi.22731

63. Place of Publication: JIF
Fang, F. C., & Casadevall, A. (2011). Retracted Science and the Retraction Index
. Infection and Immunity, 79(10), 3855–3859. http://doi.org/10.1128/IAI.05661-11

64. Impact ?

65. Place of Publication: Open Access
• Boost the potential visibility and access of research,
to a wider audience
• Removes research from behind paywall barriers for
existing audience
• Enables authors to retain their rights and more easily
share via social media, email etc.

66. Place of Publication: Open Access
• 4633 articles across ecology, applied
mathematics, sociology and economics.
• 2280 were open access, and had an
average citation count of 9.04
• 2353 were subscriptions journals, and had
an average citation count of 5.76.
Norris, M. (2008) “The citation advantage of open access articles” Thesis. Available at
https://dspace.lboro.ac.uk/2134/4089

67. Place of Publication: Open Access
SPARC Europe: Open Access Citation Advantage Service
http://sparceurope.org/oaca/
Total number of studies so far 70
Studies that found a citation advantage 46
Studies that found no citation advantage 17
Studies that were inconclusive, found non-
significant data or measured other
things than citation advantage for articles
7

68. Place of Publication: Open Access
Terras, M. (2011) “What happens when you tweet an Open Access Paper” Melissa
Terras’ Blog. Available at http://melissaterras.blogspot.co.uk/2011/11/what-happens-
when-you-tweet-open-access.html

69. Place of Publication: Open Access
Terras, M. (2011) “What happens when you tweet an Open Access Paper” Melissa
Terras’ Blog. Available at http://melissaterras.blogspot.co.uk/2011/11/what-happens-
when-you-tweet-open-access.html
By October 2012, the OA
version had seen nearly 3
times more downloads than
the version sitting behind a
subscription paywall.

70. Demo
Sherpa Romeo
Sherpa FACT

71. Over to you
• (very briefly) For any of the journals you have
previously looked at (eg to se their JIF), use
Sherpa Romeo (or Sherpa FACT if your research
is funded) to identify your open access options (if
you can interpret Sherpa!)
Checking OA options
Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available
at http://www.flickr.com/photos/theblackcanvas/2945878325/

72. Other suggestions…
• Dr Michael Taylor, Dept. Earth Sciences (Bristol)
- Discusses strengths and weaknesses of his already
published article titles.
- http://tinyurl.com/k6dhcac
- http://tinyurl.com/k7o9msc
- avoid vague words / weak puns
- NEGATIVE “it’s 12 characters too long to tweet”
- POSITIVE “ the title strongly implies the conclusion”
- POSITIVE “ Short, appealing and (hopefully) funny.”

73. Beyond Bibliometrics…
• Altmetrics…

74. Learning Outcomes for the session
• Understanding the original purpose and intended uses of
bibliometric research indicators
• Understanding of how some key indicators (JIF, H-index)
are calculated
• Ability to make a judgement as to the appropriateness and
limitations of such indicators
• Ability to use online datasets to view and calculate key
bibliometric measures
• Awareness of some factors which can increase the
visibility and discoverability of your own publications.

75. Image Credits
[Slide 5] Via Flickr Creative Commons, by demonsub. Original available at
http://www.flickr.com/photos/66397367@N05/6993902269
[Slide 13] Via Flickr Creative Commons, by Kirsty Andrews. Original
available at http://www.flickr.com/photos/47745961@N08/5169765739
[Slide 23, 35, 50, 71] Via Flickr Creative Commons, by © Stuti Sakhalkar.
Original at http://www.flickr.com/photos/theblackcanvas/2945878325/
[Slide 40] Via Flickr Creative Commons, by Alan Cleaver. Original available
at http://www.flickr.com/photos/11121568@N06/4122176776/
[Slide 22] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC)
Limited‘ Available at www.vitae.ac.uk/rdf

76. Measuring
Researcher
DevelopmentVitae Researcher Development Framework [see image credits]

77. Durham University Library: Supporting your research
Contact Details
Academic Liaison Librarian (Researcher Support)
james.bisset@durham.ac.uk
Phone: +44 (191) 334 2961