Practical tips to conducting A/B tests on emails, starting from planning, designing, timing, executing and reporting

1. A/B email testing - Proposal / Recommendations
Menaka Seshakumar
Marketing Manager (Digital/CRM/Automation)
Singapore

2. Testing Tactics
• Subject line
• CTA
• Images vs Text
• Template design
• Timing of sends
• Test on communication types – newsletters vs invites vs edms
• Frequency testing –between emails
• Trigger date testing – prior to event start date
• Sender’s name inclusion
• Signatures
• Subject personalization
• Dynamic content placement

3. Getting Started..
1. Consensus on what to test.
2. Test group would consist of 10% of the pool. The winning
email to be sent to the remaining 90%
3. Testing would include only one variation, to clearly give a
cue on what element has affected the email performance
rate.
4. Most email responses are to come within 24-48 business
hours. Hence allow 2 business days for testing, prior to the
actual blast. The winning email would to be determined at
the end of 48 hours and will be blasted to the 90%
audience pool.

4. Subject line testing
• Long vs short subject lines
• Inclusion of a number in the subject line – does this get our audience to open
emails?
• Subject line personalization
• Subject line urgency
• Example scenario
– June Newsletter
• Subject Line A: Mobile opens take the lead: new market share stats
• Subject Line B: Mobile opens take the lead with 80% increase over 6 months
• What we tested: Using statistics vs. not using statistics in the subject line
– July Newsletter
• Subject Line A: Get your email opened: First impressions, subscriber engagement infographic +
Interactive Testing sneak peek
• Subject Line B: Get your email opened: First impressions + new infographic
• What we tested: Length of subject line
– Aug Newsletter
• Subject Line A: Your passcode to a plethora of Intel inside resources
• Subject Line B: Sign in to gain access to Intel inside resources
• What we tested: More aggressive/attention grabbing language vs. CTA language (Remove the word
“Marketing” from eDM subject lines)

5. Pre-work
Testing calendar
– Recommended to test one tactic every month
– Set-up and share a google calendar to allocate
dates for A/B test runs.
– Benefits of A/B test calendar
1. Very quick at-a-glance view of tests running
2. Easily shared with anyone
3. Low overhead

6. Testing tactic Description
Component Measure
Opens Clicks Unique
Visitors
Click to
Open
Open
rate
Email version A B A B A B A B A B
From Line
Subject Line
Day Sent
Time Sent
Layout / Design
Offer
Headline
Content /
Messaging
Personalization
Email Signature
Email Testing Matrix by individual campaign

7. Test tracker
A/B test by program (Essential Series)
Program plan Jun Jul Aug Sep Oct Nov Dec
Campaign
Title
(Essential
series)
Jun
newsletter
Jul
Newsletter
Aug
Newsletter
Sep
Newsletter
Oct
newsletter
Nov
newsletter
Dec
newsletter
Go-live date
Topic
Subject line
From name
Test element Subject line
(statistics)
From Name Landing page Email
template
CTA Offer Subject line
(Length)
Test objective Increase open
rates
Increase sign
ups
Increase web
traffic
Test audience
size

8. Email analysis via reporting
Email Name
Email Subject
line
Total
Sends
Total
Delivered
Total
Opens
Open
Rate
Total
Clickthrou
ghs
Clickthrou
gh Rate
Click To
Open
Rate
Existing
Visitor
Clickthrou
ghs
New
Visitor
Clickthrou
ghs
Total
Form
Submissio
ns
Form
Conversio
n Rate
from
Email
Email IT newsletter
Subject line A 2518 2472 409 16.55% 54 2.18% 0.00% 5 49 8 0.32%
Subject line B 2127 2094 360 17.19% 46 2.20% 0.00% 24 22 4 0.19%