Whilst passage indexing may seem like a small tweak to search ranking, it is potentially much more symptomatic of the beginning of a fundamental shift in the way that search engines understand unstructured content, determine relevance in natural language, and rank efficiently and effectively. It could also be a means of assessing overall quality of content and a means of dynamic index pruning. We will look at the landscape, and also provide some takeaways for brands and business owners looking to improve quality in unstructured content overall in this fast changing landscape.

1. Passage Indexing – It is
Probably a Bigger ‘Thing’
Than You Think

2. Hello… I am
Dawn Anderson
from Bertey

3. I’m a nosey crow SEO who likes to
follow happenings in the information
retrieval research space… a lot

4. Information
retrieval is the
arm of computer
science behind
web search

5. It’s ‘complicated’ but
it’s not ‘magic’

6. It’s just maths and
programming after all

7. Research, you say… Pffft

8. “That’s not
‘real life’
‘production
search’

9. True… There’s a
lot of web
search research
which may not
go anywhere

10. But...There are
(themes)
‘directions’
research heads
in – ‘Hot topics’

11. Plus… Google takes a ‘Hybrid Approach’ to
research
https://research.google/pubs/pub38149/

12. Research &
engineering
are closely
aligned
Research designed to meet needs of
users
Research designed to make it to
production in a relatively short space
of time often
Engineers are often researchers and
vice versa
Able to research and deploy iteratively
and quickly

13. Plus… There are
research applications
built to align IR
research with ‘real life’
industry – e.g. Anserini
(Yang, Fang & Lin, 2017)

14. Designed to build ‘reproducible’ code,
tests & studies to avoid the below:

15. I’m going to talk
about a ‘Hot topic’

16. Passage retrieval,
ranking & re-ranking

17. You may have
heard of
Google’s new
‘Passage
indexing
Algorithm’

18. Live in US
Search
already since
Feb 2021

19. It will
‘eventually’
impact 7% of
queries globally

20. Groan… You all know
all about it… right?

21. Humour
me…

22. ‘Passage
indexing
algorithm’ is
just the tip of
the iceberg

23. Being aware of
how important
passage research
in IR is might help
your SEO efforts
in other ways

24. Passage research is a big
deal in information
retrieval right now

25. It’s much bigger
than just this
passage
indexing
algorithm

26. Passage Indexing is Part of
a “Bigger Breakthrough”

27. “A Breakthrough in Ranking”
• “We’ve recently made a breakthrough in ranking and are now able to
not just index web pages, but individual passages from the pages,”
“By better understanding the relevancy of specific passages, not just
the overall page, we can find that needle-in-a-haystack information
you’re looking for.” (Raghavan, P, 2020)

28. That kind of sounds
BIG

29. So, just what is Google’s
‘Passage Indexing
Algorithm’ about?

30. Spoiler
I’m not going to tell you to ’optimize’ for the passage indexing algorithm

31. It’s not
something
you can
‘optimize’ for

32. If anyone offers to ‘optimise
passages’ – Run for the hills

33. It’s kind of a new
‘clarifying algorithm’

34. Google first announced its
‘passage indexing algorithm’ in
October 2020

35. Passage indexing is
about better
understanding of
contextual language
use by search
engines using AI

36. More specifically better
‘natural language
understanding’

37. E.g. Understanding when ‘bank’ in queries or
content means ‘river bank’ versus ‘financial bank’

38. It’s a way to judge parts of a document
individually for relevance to a query
(passages), rather than as just one very
small part of a full document

39. Passage
indexing is
actually about
passage
ranking

40. So let’s just
call it
‘passage
ranking’

41. It’s primarily for long form
content

42. LARGE documents

43. Which cover multiple
topics (scope creep)

44. OR… the author just used more
words than needed (verbosity)

45. Likely over a threshold >
‘x’ unique ‘uncommon’
words

46. i.e. other than … Common English Words
a,able,about,across,after,all,almost,also,am,among,an,and,any,are,as,at,
be,because,been,but,by,can,cannot,could,dear,did,do,does,either,else,ev
er,every,for,from,get,got,had,has,have,he,her,hers,him,his,how,however,i,
if,in,into,is,it,its,just,least,let,like,likely,may,me,might,most,must,my,neithe
r,no,nor,not,of,off,often,on,only,or,other,our,own,rather,said,say,says,she,
should,since,so,some,than,that,the,their,them,then,there,these,they,this,t
is,to,too,twas,us,wants,was,we,were,what,when,where,which,while,who,w
hom,why,will,with,would,yet,you,your

47. ASIDE: These words nowadays actually
add ‘contextual glue’ in natural language
models, but were traditionally removed
‘stop words’

48. They still add
very little
cumulative
information
gain (value)

49. It’s probably for
LARGE
documents
without
semantic
structure

50. Literally like a loose
‘Bag of Words’

51. Maybe
disorganised
topics too –
‘Jumble sale
content’

52. It won’t impact
ecommerce /
transactional pages

53. Not directly anyway

54. But… Understanding
passages in long
form content ‘might’
help ecommerce
pages indirectly
(more on that later)

55. Why not? – Transactional pages are
different beasts to informational pages
Not that many words on
an ecommerce page
(either product page or
category page)
Ecommerce pages tend
to have an ‘out of the
box’ organised content
structure template
Ecommerce pages will
likely have lots of other
'value add’ features too
Ecommerce content is
not conversational in
nature
Ecommerce pages tend
to have a descending
'importance' hierarchy
in the page content 'out
of the box'

56. Lots of clues
as well
‘beyond the
words’
Internal links with
indicative anchors
Strong
categorization &
subcategorization
‘Probably’ some
product schema
markup too

57. So, it’s probably not that difficult for search
engines to understand ecommerce pages

58. Passage indexing / ranking
is a ‘leg-up’ to pages which
likely don’t rank for much
currently

59. Unoptimised
long form
content

60. They don’t look like this
<h1>Main theme</h1>
<p>paragraph</p>
<h2>Key point - Section – Subtopic</h2>
<p>paragraph</p>
<h3>Further point heading connected to h2 section</h3>
<p>paragraph</p>
<h2>Key point – Section – Subtopic</h2>

61. They could
be randomly
‘tagged’ with
nonsense

62. Real topic
experts
‘probably’
don’t optimise
pages

63. There may be some very
relevant expert points in a
long document they author

64. Sometimes there can be
‘diamonds’ of knowledge

65. Buried… and not
obvious at first sight

66. But… how to
dig them
out?

67. AI / NLP &
passage
research has
makes finding
‘diamonds’
possible

68. Passage
research is
about wider
ranking
‘challenges’ too

69. Longstanding ranking
‘problems’

70. It mostly comes down to
thr ‘stages’ of ranking &
ranking efficiency

71. Balancing
costs versus
potential
returns

72. Multiple
Stages of
Ranking

73. Two (or multi) stage ranking
Retrieval (first
stage)
Re-ranking
(refinement of
the initial fetch
for optimal
search results)

74. Retrieval stage
From all possible text (or synonym)
matching candidates above an x
quality threshold fetch e.g. 1000
Pass to the next stage for
further consideration

75. Re-ranking stages (Cascading stages?)
From e.g. initial
fetched 1000,
fine tune an
optimal set of
combined
results to meet
query
Present in
response to
query in search

76. Why more than
one stage?

77. Efficiency &
Effectiveness
Considerations

78. Importance

79. Zipfs Law in
everything

80. Not ‘EVERYTHING’ is worth
seriously even considering

81. Only a relatively few pages
& sites will be important

82. The ‘fine-tuning’ is
expensive

83. The ‘fine-tuning’ part uses heavy
machine learning to ‘meet needs in
Top-K search positions’

84. The aim is to build ‘the
perfect set of results in a
perfectly ranked order’

85. The Probability
Ranking
Principle in
IR(PRP)
(Robertson,
1977)

86. But PRP only
considers
documents
independently

87. Doesn’t consider what else
might make up a ‘rich
SERPs experience’

88. And how to rank with a
range of ‘intent aspects’

89. Because it’s the richness
of the results which meet
‘possible information
needs’

90. Query aspects – ‘Harry
Potter’ – Did you mean the
book or the film or images?

91. Diverse results matter to users

92. Also
Computers
Were (Are)
Hopeless at
Ambiguity
“Computers are hopeless at disambiguation – at
understanding which of multiple meanings is
correct – because they don’t have our world
knowledge.” Dr Stephen Clark, Cambridge
University
https://www.cam.ac.uk/research/features/our-
ambiguous-world-of-words

93. Different
types of
ambiguous
queries
Genuine ambiguous queries –
‘bank’, ‘rose’, ‘bond’
Underspecified queries – ‘Harry
Potter’ (is it the book or the
film?)
Fully specified queries – ’Harry
Potter film’ (aspect is fully
defined)

94. Queries can
be
ambiguous –
e.g. Apple
‘Apple’
(fruit)
‘Apple’
(Computer)

95. Which is more
relevant?

96. It depends… On many
factors

97. Also…The results should
match the probable
audience needs

98. Who is the
user?
What the niche
is? (Do videos
matter more?)
Do more people
search for apple
fruit there?
Where is the
user? (location)
Are images or
video a good fit?
When is it?
(Query intent
shift) (

99. What’s the probability x
user wanted x information
need meeting?

100. Perhaps it’s best to show a
range of possible intent
matching results?

101. Relevance Ranking & Result Diversification
A list of documents ranked
in order of descending
relevance to a query (PRP)
But…Diversity matters in
results returned to account
for ambiguity and
redundancy. Better to have
broad diversity and novelty
to cover all query aspects

102. Modern fine-tuning is NOT
document independent

103. ‘Search’ is about
the ‘WHOLE’
experience

104. This is ‘The
Search Result
Diversification
Problem’

105. In addition to Probability Ranking Principle several
other approaches have been explored
Portfolio Theory (PT)
(Wang and Zhu, 2009)
Quantum Probability Ranking Principle (QPRP)
(Zuccon, G., Azzopardi, L.A. and Van Rijsbergen, K., 2009)
Maximal Marginal Relevance (MMR)
(Carbonell and J. Goldstein, 1998)

106. Seeking to
find the
‘BEST’ blend
for the
moment

107. It’s this…
and it’s not
‘magic’

108. It’s ‘Learning-to-
Rank’ (LeToR)

109. Learning-to-
Rank
(learning
‘current
relevance’)
Feed a model with click data / queries (learn current
relevance)
Train & lab evaluation
Send to human quality raters scoring (human in the loop)
Aggregate the scores from raters 'on the whole evaluation'
Use NDCG (Non-discounted cumulative gain) to adjust the
whole test algorithm based on scores from human raters

110. It’s like this…
Predicting what
people want in
given scenarios

111. So…SERP
Stages
First stage - Retrieve initial fetch of
e.g. 1000 documents to get a top-k
(e.g. 20) candidates to re-order
Final Stage - Refine to build an
optimal page dynamically refined
for relevancy & diversity

112. Dynamic Search Result
Diversification in IR is an NP-
Hard problem

113. i.e. – No-one has
solved it

114. Here’s a great book on the subject

115. Also… there
are…
pipeline
problems

116. The first stage (retrieval)
has been ‘lacking’ for a
long, long time

117. Classical
Information
Retrieval (first
stage) was ‘lexical’ –
matching ‘keywords’
present in both
documents and
queries

118. i.e. Are the
words in the
document?

119. BM25
TF:IDF
Example common first
stage algorithms

120. But does having
word frequency
make something
automatically
relevant?

121. Some
semantics
were likely
added
Synonyms
Boolean OR, OR, OR
parameters added perhaps
Query expansion

122. But nothing too
expensive initially

123. Later stages
of ranking
are for the
‘magic’

124. Precision refinement
Novelty in results
Subtopic retrieval
Reduce redundancy in results
Determine ‘the best mix of results for the query’
Determine the optimal position for each candidate type

125. Re-ranking uses more expensive methods
COMPUTATIONALLY FINANCIALLY ENVIRONMENTALLY

126. When the first
stage of ranking
is based on
relative
keyword (term)
frequency
matching to e.g.
document
length
Lexical matching
E.g. BM25 (Best
match 25
algorithm)
commonly used
in information
retrieval

127. Large
documents
are a BIG
problem

128. Plus there’s also an
increasing number
of long documents
specifically
and an increasing
number of
documents to
choose from

129. Large documents can contain ‘too much information’
Verbosity Dilution Scope creep
Vagueness Waffle
Content for
the sake of
content

130. So much
information about
too many topics.
Makes a document
irrelevant for
everything

131. Those ‘diamonds’
buried can be missed

132. Not just long
documents either…
Just any full
documents too
(i.e. any text filled
URI)

133. Can be hard to determine
off ‘word matching’ alone

134. Lots of gems of diverse
knowledge / query aspect
meeting info can be missed

135. The long document
problem is well known

136. E.g. Relevance
feedback with
too much data
(Allan, 1995)

137. So passage retrieval
research is not new

138. Indeed… Every snippet in
the SERPs in a passage of
sorts

139. Designed for
summarization
& search result
diversification
in re-ranking

140. IR Researchers have been working
on understanding relevance in
passages for decades

141. And there
are many,
many more

142. Some past approaches
included… ignoring them
completely

143. It looks like…
Several
different
approaches,
including:
Ignore Them
• Ignore them completely to save on efficiency. As long
as there are other smaller relevant documents to rank
Trim Them
• Trim the document by selecting one good passage
Identify a passage
• Identify one passage only on large documents over x
size and for everything else (smaller documents) use
normal full document ranking
Use TextTiling
• Try to identify subtopics in natural paragraph breaks
Use only The First 'n' Terms
• Anything after 'n' terms is ignored

144. No wonder
‘diamonds’
could be
missed

145. But judging
individual
passages versus
full documents
can increase
perceived
relevance

146. So… Divide &
Conquer

147. Divide the
problem into
parts &
solve each

148. Divide the document
into ‘passages’

149. And then
apply the
same
principles

150. Multi-Stage
Ranking
Systems on
passages
too
Full
ranking –
document
Re-
ranking –
document
Full
ranking –
passage
Re-
ranking -
passage

151. Recently
breakthroughs…
NLP language
models came
along

152. Specifically BERT
(Devlin, 2018) &
Better BERTS

153. What is BERT (Bi-
Directional
Encoder
Representations
from
Transformers?
A PLM (Pre-trained language model)
Used for neural matching to understand word’s meaning in
‘context’
So, not just ‘keyword matching’ to documents on a page
But understanding multiple meanings of a word in different
contexts
BERT is pre-trained on millions of words so natural language
researchers / engineers have a ‘starter for ten’ language
model which already understands the contextual glue /
nuance in words

154. BERT can dig
out ‘meaning’ in
informational /
conversational
content

155. By providing context
& disambiguating

156. There’s now LOADS of BERTs
EVERYWHERE

157. There’s even
a BERT-lang
Street

158. Many
researchers
jumped on
board

159. That’s the
nature of
‘open source’
research
after all

160. And Big
Tech
extended or
segwayed
BERT
ELECTRA (Google)
T5 (Google) (Huge compared with BERT)
RoBERTa (Facebook)
ERNIE (Baidu)
Turing-NLG
MT-DNN (Google)
GPT, GPT-2, GPT-3 (Open AI)

161. Building ‘SuperBERTS’

162. ’Model Ensembling’

163. Plus ‘Student’ &
‘Teacher’ BERTs

164. But BERT’s
were mostly
‘nice to
have’s’

165. BERT has architectural
limitations

166. It has a 512 token
capacity – so not full
document length

167. But then… BERT was repurposed
as a passage re-reranker
(Nogueira, R. and Cho, K., 2019)

168. Divide & Conquer at
scale

169. Google researchers utilized this too

170. An
Ensemble -
Taking the
best parts of
other
language
models &
features
RoBERTa (Facebook)
ELECTRA (Google)
BERT (Google)
Tensorflow (Google)
DeepCT (Dhai)
Learning to Rank (LeToR)

171. DeepCT (Dhai,
2019)
‘Deep
Contextualized
Term-
Weighting
Framework’
tfDeepCT – An alternative to term frequency which
replaces tf with tfDeepCT
DeepCT-Index – Alternative weights added to an
original index, with no additional postings. Weighting
is carried out offline, and therefore does not add any
latency to search engine online usage
DeepCT-Query – An updated bag-of-words query
which has been adapted using the deep contextual
features from BERT to identify important terms in a
given text context or query context.

172. “A Breakthrough in Ranking”
• “We’ve recently made a breakthrough in ranking and are now able to
not just index web pages, but individual passages from the pages,”
“By better understanding the relevancy of specific passages, not just
the overall page, we can find that needle-in-a-haystack information
you’re looking for.” (Raghavan, P, 2020)

173. It’s ‘probably’ this or
something better still

174. Using passage rerankers on long
form content suddenly makes long
form content understandable
(even in the first stage of retrieval)

175. ‘Passage
indexing
algorithm’ is
just the tip of
the iceberg

176. Passage research has NOT
slowed down

177. Passages
are
everywhere
too
Every long conversational query
In long form content
In forums, UGC, tweets,
Facebook posts
Every People Also Ask is a
passage (question)

178. Passages are data & data is like oil for NLP
researchers

179. Transfer
learning

180. Search engines like Google and Bing ‘gift’ training
data to NLP / IR researchers to progress

181. MSMARCO is
one of the
biggest
drivers
(Real Bing
queries)
1,000,000 question dataset
Natural language generation dataset
Passage ranking dataset
Keyphrase extraction dataset
Crawling dataset
Conversational search dataset

182. MSMARCO

183. When you
combine
passage data
with BERT type
models… rocket

184. Understanding
passages is the key to
understanding
language, intent & user
tasks it seems

185. And
improving the
whole ranking
pipeline

186. Break the
tasks down
Open domain question answering
Greater understanding in expert
disorganized content
Greater conversational search
Greater understanding about how
everything 'fits together'

187. Divide & Conquer

188. Passages &
subtopics
are VERY
connected

189. Further progress in
passage research

190. The NLP
Research
Community is
‘ALL IN’ on
passage ranking
& re-ranking

191. Obviously fueled by MS-MARCO Passage
Ranking Leaderboard

192. Move over BERT… VERY different
models are appearing
T5 (Google)
DocTTTTTQuery (Nogueira & Lin, 2019)
Expando-mono-duo (Pradeep, R., Nogueira, R. and Lin, J., 2021)
RocketQA + Ernie (Qu et al., 2020, (Baidu))
COIL (Gao, L., Dai, Z. and Callan, J., 2021
DeepCT (Dai, Z and Callan, 2019)

193. And many are
focused on
improving the
first stage of
ranking
dramatically

194. First stage
retrieval has
been a
‘bottleneck’

195. Hot Topic - Improving
first stage retrieval

196. Passage research
seems to be very
connected to this

197. Alternatives
to BM25 &
TF:IDF are
being found
in passages
Dense Passage Retrieval
ANCE
DocTTTTTQuery
COIL
ColBERT
DeepCT

198. Dense Passage Retrieval – ‘Packing’ sparse initial
stage data with training passages
ANCE – Nearest neighbour approach
DocTTTTTQuery – Query expansion in first stage
COIL – Scoring system to add ‘context’ to BM25 first
stage retrieval – adds semantic & lexical (hybrid)
ColBERT – Late stage BERT introduction
DeepCT – Index term weighting with DeepCT
framework for ‘importance weights’

199. 2020 & 2021
‘dense passage
retrieval’
research is on
fire

200. Including Google

201. Google are working on hybrids for first stage
too

202. Leveraging Semantic and Lexical
Matching to Improve the Recall of
Document Retrieval Systems: A Hybrid
Approach (Kuzi, S., Zhang, M., Li, C.,
Bendersky, M. and Najork, M., 2020)

203. And ‘that breakbrough’??

204. It’s in production

205. Advances in
TF-Ranking
(July, 2021)

206. They talked about the ensemble
success here in 2021

207. This rapidly evolving
research ‘may’ be
part of why Google’s
Passage Indexing
Algorithm has not
been rolled out
further yet

208. Things are
changing quickly.
‘Passage
Indexing’ could be
greatly improved
soon

209. So, what does
all this mean
for you?

210. Understanding passages &
subtopics more could greatly
improve search result
diversification

211. First stage
retrieval covers a
mountain of
unstructured text
/ unstructured
data

212. A better ranking
pipeline gives
more choice

213. Subtopics & passages
go hand-in-hand

214. Sub-topics +
intents +
tasks

215. They ‘co-occur’

216. ”You shall
know a word
by the
company it
keeps” (Firth,
1957)

217. Your page is
NOT ranked
independently

218. Your page is judged for the ‘added
value / diversity’ it can bring to a
mix of results ALREADY deemed
relevant

219. What you create
in informational
sections will
impact ‘value’ in
other ‘intent’
sections

220. You will be Judged
on The ‘Whole Site
Pie’

221. On all
sections of
the site
Informational content (blogs, guides,
FAQs etc)
Informational ‘collection’ pages (e.g.
blog category pages, guide hub
pages)
Transactional content (e.g. -
ecommerce product pages, local
listings)
Transactional ‘collection’ pages (e.g. -
ecommerce category pages, local
listing category pages)

222. Since meeting the needs of
under-specified queries
requires broad coverage

223. You will be judged on
“How many of the intents of this under-
specified query do you meet?”

224. It could be hard to rank for
big ‘head’ terms without
comprehensive stock
(‘information gain’ (value))
within your document
collection (site), for all
query aspects (intents) to
meet an under-specified
query

225. You need to
do everything
with ‘high
quality’ in
mind

226. Start with a
‘few’ GREAT
pages

227. ‘Overflow
SEO’

228. 'Grow your
site up’

229. Always think
“Where is the
information gain
beyond what is
already in this site?”

230. ‘Build Relevance
Layers’

231. Always think “Where
is the information
gain beyond what is
already in the
SERPs?”
Or do it much better

232. i.e. Make sure you have
content that meets
sufficient demand

233. There could be
‘Dynamic Index
Pruning’ by search
engines

234. But you
should move
and ‘prune’
with caution

235. Don’t just add more
content

236. Think…
Are we bringing a new
perspective?
Is the page substantively different
to what we already have
Does this page meet another
aspect of information need /
different intent?
Does this page substantively bring
another type of media into the
mix?
Don’t try to be the same as what is
already ranking… do something
more / different

237. Bring a new perspective
THOUGHT LEADERSHIP NEW DATA DRIVEN
REFRESHING
PERSPECTIVES
INTERACTIVE TOOLS ADDITIONAL FEATURES IN
YOUR CONTENT WHICH
OTHERS DON’T HAVE?

238. Improving
informational
content
Add semantic headings
Connect up topics via ‘relatedness’
Learn to structure content properly
Inverted pyramid approach to web content
Avoid verbosity
Move’ and ‘prune’ with caution
Don’t ‘prune’ away your semantic ‘relatedness’

239. It all comes
back to this…

240. Relevance
and diversity
in SERPs
combined

241. Unlocked by a better end-
to-end search ranking
pipeline

242. Across a range
of task intents &
media types
dynamically
suited to the
audience

243. Toward task-
driven
search

244. Maybe something like
this… ->

245. MUM
(Google)

246. From answers to journeys
A new visual way to meet
informational needs
From queries to queryless

247. Understanding user’s
information needs a
passage at a time

249. Thank you J - @dawnieando