The document summarizes an experiment conducted by eBay to measure the effectiveness of paid search advertising. It found that: 1) Paid search ads had little to no effect on frequent users and positively influenced new, infrequent, and lapsed users, leading to short-term sales increases. 2) For established brands, nearly all paid search click traffic was substituted by organic search traffic when ads were turned off, with most clicks from new/infrequent/lapsed users. 3) Paid search ads should not target frequent users searching for branded terms, as this has a negative impact on return on ad spend. It then outlines how this methodology could be adapted by GoCity, such as by analyzing

1. 1
GO CITY
KOFO MARY ARE

2. 2
The primary objective of the presentation is to summarise the Ebay
methodology for an experiment. Titled “Consumer Heterogeneity and Paid
Search Effectiveness: A Large Scale Field Experiment”
Designed to measure the causal effectiveness of paid search ads. The
premise of the experiment is encapsulated by the Null Hypothesis.
Null Hypothesis: Ho: β≥0
There is “hidden value” in “non-branded” user searches.
The Alternative Hypothesis: Ho: β<0
There is no “hidden value” in “non-branded” user searches.
The second objective is the adaptation of a series of experiments conducted
by eBay in 2014 for GoCity.
Reject the null hypothesis as there is limited value hidden in non-branded searches

3. 3
KEY
TAKAWAYS
E-BAY
HAVE A MATERIALLY NEGATIVE
INFLUENCE ON THE BEHAVIOURS OF
FREQUENT USERS . AND A POSITIVE
INFLUENCE ON NEW, INFREQUENT AND
LAPSED USERS. FOR SHORT –TERM SALES.
PAID FOR ADS
FOR ESTABLISHED BRANDS, PAID FOR CLICK TRAFFIC IS
SUBSTITUTED NEARLY ENTIRELY BY ORGANIC TRAFFIC.
DURING CONTROLLED EXPERIMENTS WHEN PAID FOR
ADS ARE SWITCHED OFF.
THE MAJORITY OF CLICKS ARE ATTRIBUTED TO NEW,
INFREQUENT AND LAPSED USERS. ALSO THERE IS A
SUBSTANTIAL FALL IN CLICK TRAFFIC.
CLICK TRAFFIC
FOR ESTABLISHED BRANDS, PAID FOR ADS SHOULD
NOT BE PRESENTED TO FREQUENT USERS THAT
DEPLOY “BRANDED KEYWORDS” IN SEARCH.
AS ACTIVITY BIAS WHICH IS THE LARGE SHARE OF
HEAVY USERS CLICKING ON PAID FOR ADS. HAS A
SIGNIFICANT NEGATIVE IMPACT ON ROAS. LEADING
TO NEGATIVE ROI IN CONTROLLED EXPERIMENTS.
FREQUENT USERS
SEM ACCOUNTED FOR A SIGNIFICANT INCREASE
IN NEW CUSTOMER ACQUISTIONS AND AN
INCREASE IN THE NUMBER OF PURCHASES
FROM INFREQUENT AND LAPSED BUYERS..
CHANNEL EFFECTIVENESS
USER ACQUISITIONS
SEM HAD A VERY SMALL OR INSIGNIFICANT INFLUENCE
ON SALES. ONCE PAID FOR ADS HAD BEEN SWITCHED
OFF.
CHANNEL EFFECTIVENESS
NEW SALES
INFREQUENT AND LAPSED USERS ARE THE WORSE
CUSTOMERS AS THEY ENGAGE IN A VERY LOW NUMBER
OF PURCHASES. FURTHER MORE THEY ACCOUNT FOR
THE MAJORITY OF CLICK TRAFFIC WHEN PAID FOR ADS
ARE SWITCHED OFF.
NEW, INFREQUENT AND
LAPSED USERS
The amount of money spent on an ad
campaign is a function, not only of the
advertiser’s campaign. But is also determined
by the behaviour and intent of the consumers
The endogeneity problem, which is related to
activity bias Posits that people who are more
active online will both see more display-ads and
click on more links. Leading to the concern that
attribution models may overestimate the
efficacy of such ads

4. 4
SUMMARY OF
EBAY
METHODOLOGY
HANDLING OF USERS
Expose a random group of users to the ads and a control
group of users who will not be exposed to the ad.
Types of User
1. Infra- marginal consumers: users unaffected by ads
2. Marginal consumers: users affected by ads
User Location Sourced From: Shipping Zip code
D3 = D1 – D2 = Difference between MSN paid and MSN natural search
D4 – D2 = Difference between Google Natural Search and MSN natural search
ATTRIBUTED SALES: fell by 72%
NB: All attributed sales within a 24-hour period correspond to a user
clicking on a Google paid search link
DATA: eBAY bids on over 100 million keywords, thus provides an ideal
environment to test the effectiveness of paid search ads for non-branded
keywords.
Branded Keywords: Advertising for “branded related terms” was, halted.
Duration: 60 days
Ad Networks: MSN and Yahoo!
GEOGRAPHC LOCATION: Determined using Googles geographic bid feature for
Nielsen Designate Market Area .
NUMBER OF DMA’S IN THE USA: 210 correspond to metropolitan areas.
PAID FOR ADS: Ads for 30% of the DMA’s were suspended .
SELECTION OF DMA’S: Random sample of DMA’s were selected, and segmented
into a “control” group and a “test” group
RESULT
99.5% of all forgone paid for clicks traffic from branded keyword paid search was
captured by organic search traffic from the ad network platform. In this case
“BING”.
Click Volume: 5.6% lower in the period.
Organic search essentially substitutes paid search for “branded keywords” in the
absence of paid ads.
Quantification of Substitution: Regressed the log or total daily
clicks from MSN to eBay. On an indicator for whether days were in the
period with ads turned off.
PAGE 6: The difference in difference estimation. MSN and Google Test. Fig
2.0 Brand Keyword Substitution.
When seasonality is accounted for
Click Traffic Lost: 0.529%
So 99.5% of traffic is retained.
RESULTS OF NON-BRANDED EXPERIEMENT
Advertising Clicks: Declined by 41%
Total Clicks: Declined by 2 %
TOTAL CLICK LOST: 58% of the total lost paid search clicks.
Page 9: Graph Attributed Sales By Region Fig 3 a & b
• 68 Test DMA’s where advertising ceased
• 65 Matched Control DMA’s
• 77 Control DMA’s
Page 10: Table 1.0 ROI
PAID SEARCH ADS ROI : only add 0.66% to sales at a 95% c
confidence interval of [-0.42%, 1.74%].
Consumer Response Heterogeneity
Page 12: Paid search effect by user segment
To quantify the impact of the different types of users in the DMA subgroups on sales in
eBay.
Duration: April 2011 - 2021
A treatment dummy was interacted with indicators for “the number of purchases” by
the users deploying historical data: from April 2011 to April 2012. In the New York Area
NULL HYPOTHESIS: Ho: β≥0 Users who type “eBay” are using search as navigation with the intent to go to ebay.com
ALTERNATIVE HYPOTHESIS: Hoβ<0 Users who type “eBay” are not using search as a navigation tool with the intent to go to ebay.com
the amount of money spent on an ad
campaign is a function, not only of the
advertiser’s campaign. But is also
determined by the behaviour and intent
of the consumers
BRANDED & NON-BRANDED SEARCH EXPERIMENTS
A 1% drop in paid search visits leads to a
0.5% increase in natural search visits and a
0.23 % increase in direct navigation visits.
There are 210 DMA’s in
the USA
RESULTS
The graphs demonstrates the effect that advertising has on influencing consumer behaviour. Pg 12
- Ads have little or no effect on active or moderately active users
- The effect of advertising steadily rises with the duration between a purchase
- The effect of advertising is statistically significant with users that have not made a purchase in one year
Where did the non-branded traffic go? Pg 14 Paid Search Attribution by User Segment
The majority of users that click on ads are frequent users that are not navigating directly to
the eBay website
Advertising clicks dropped:41% ; Total Clicks : dropped 2%: Total Clicks lost: 58%; Indicating that 42% of paid
clicks are newly acquired. i.e. new users of the brand.
NB: Need to make a distinction between the nature of visits: Hence distinguish between referring clicks and total
visits (clusters of page visits by the same user). Users will travel to eBay from Google multiple times in one sitting.
Quantification of ROI: Refer to equation in the notes section
Hence a 99.5% drop in paid search visits
leads to a 49..5% increase in natural search
visits and a 23% increase in direct
navigation visits.

5. 5
ADAPATION OF
EBAY
METHODOLOGY
HANDLING OF USERS
Expose a random group of users to the ads and a control
group of users who will not be exposed to the ad.
Types of Users
1. Infra- marginal consumers: users unaffected by ads
2. Marginal consumers: users affected by ads
User Location Sourced From: Shipping Zip code
Quantification of ROI: Refer to equation in the notes section
Data: The total number of keywords Go Pass bids on is unknown.
However, we need the keywords bided on by Go Pass for branded
and unbranded words for the New York area.
Total population is 18,823,000 in 2021.
Sample Size: in 30% of the total DMA’s allocated for the New York
area. The ads will be suspended, for 5,646,900 people living in New
York IN 2021.
SELECTION OF DMA’S SUB CATEGORIES: Random sample of
DMA’s should be selected, and segmented into a “control” group
and a “test” group.
Duration: 60 days.
Graphs: the creation of two graphs that show click traffic counts to
the Gocity website. Graphs 1.0 will compare “click traffic counts”
where paid search was, suspended for media net. Graph 2.0
displays click traffic counts for the ad-network “Google AdSense
where “click traffic counts” are, suspended and then resumed. We
should observe the following:
∙ Paid click should fall to zero.
∙ The substitution between paid and unpaid search should
be near complete
∙ Organic search replaces paid search when ads are
suspended. When ads are switched back on paid search
replaces organic search.
NULL HYPOTHESIS: Users who type “New York Pass” are using search as a navigation tool to go to the GoCity website. Alternatively Users that are new, infrequent or lapsed users may use a non –
branded search term such as; “New York City Tourist Attractions”.
To test this Hypothesis
We need to switch off paid for ads for “branded related search terms on the relevant Ad Networks commissioned by Go Pass to Distribute Ads. Such ad networks could be Media Net or Google
AdSense. In order to determine the percentage of forgone paid-for click traffic that will be, captured by organic traffic. On another Ad network for example “Monumetric”. If this figure is close to 100%.
Then the presence of paid for ads negatively influences frequent users of Go Pass.
The DMA Area is
New York City
.
It includes all five boroughs; New York State,
Connecticut, Pike County, New Jersey,
Pennsylvania, and Long Island.
BRANDED & NON-BRANDED SEARCH EXPERIMENTS
Quantification of Substitution: Regressed the log or total daily
clicks from MSN to eBay. On an indicator for whether days were in
the period with ads turned off.
Consumer Response Heterogeneity
To quantify the impact of the different types of users in the DMA
subgroups on sales in Go Pas.
A treatment dummy was interacted with indicators for “the
number of purchases” by users deploying historical data: from Jan
2018 to Jan 2019.
RESULTS
We should expect to see the following regarding the influence of
advertising on the behaviour of different visitors to the Go City website.
- Ads have little or no effect on active or moderately active users
- The effect of advertising steadily rises with the duration between a
purchase
- The effect of advertising is statistically significant with users that have
not made a purchase in one year
Where did the non-branded traffic go?
The majority of users that click on ads are frequent users
that are not navigating directly to the eBay website