This document discusses Salesforce's approach to Selenium testing at scale. Some key points: - Salesforce runs over 1 million browser tests per day across thousands of VMs. - They use WebDriver to test against a live application across many browser and OS combinations. - Proper use of page objects is important to encapsulate Selenium and allow tests to scale. - Challenges include assigning failures, non-deterministic tests, browser and Selenium version upgrades.

1. Selenium at
Salesforce Scale
David Louvton
Lead Engineer
dlouvton@salesforce.com
Sagar Wanaselja
Lead Engineer

3. • About Salesforce engineering
• Operating a selenium farm
• Selenium design patterns
• Overcoming test failures
• Q&A
Agenda

4. Salesforce Scale
• 3:2 ratio for developers : quality engineers
• everyone is a software engineer; role varies
• selenium committers on staff (Jim Evans, Luke Inman)
• Data centers with thousands of VMs for automation
• multimillion dollar investment
• using vmware (VShere), OpenStack and EC2
• 500 commits per day
• ~120 software engineering teams

5. Why Salesforce loves WebDriver
WebDriver is the only test framework in our current
arsenal that truly mimics an end-user’s experience
• We don’t mock anything
• We test against a live app
• We suffer the pain of latency
• We test end to end on almost every action
WebDriver tests can catch things other levels of tests
can’t
• Multi-platform consistency
• DOM consistency
• Rendering issues
• Catching inconsistency between
app layers
source: watirmelon.com © Alister Scott

6. UI Testing Portfolio
• 50,000 unique Selenium tests
• 7,500 unique WebDriver tests
• full “browser compatibility”: IE6-IE11, Firefox, Chrome, Safari
• some can run in headless browser like Phantom JS
• 2,000 javascript unit tests
• no DOM required
• 10ms average execution time
source: watirmelon.com © Alister Scott

7. 1,000,000 browser tests per day
• Click here to add bullets

8. The Selenium Infrastructure
● Automatically creating, assigning, closing and reopening
bug reports for test failures and flapping tests .
● Test jobs must meet failure thresholds (99%+). Code
lines are locked for teams under thresholds
● Support for all combinations of OS & Browser using
portable VM Templates, creating deterministic and identical
test environments

10. Issues of scale
• Assigning test failures to the responsible engineer
• Non deterministic tests (flappers) and re-runs
• Selenium jar upgrades are challenging
• Browser upgrades are challenging
• IT and automation ops provide different versions
• Large suites: adjust run frequency, deprecate old tests

11. Salesforce UI Testing Pattern
• Click here to add bullets

12. Proper Page Objects Encapsulate Selenium
• Make an API of your page for your test
• No imports for “openqa” or “selenium.org” in your test
• All references to driver & selenium in the page object
• Fail fast - call the page object factory in test setup
• No assertions in page objects (or any test utility)
• Share page objects across engineering teams
• developer complete UI should have a page object
• if you can’t make one it’s not a testable page
• dependency challenges at scale

13. Challenges for Mobile Platforms
• Adapting page objects to platform specific UI elements
• Screen sizes and adaptive layouts
• “stage left” - switching into contained apps
• lightweight page objects = widgets
• Avoid browser or platform specific code in actual tests
• establish context elsewhere in utils or page objects
• Automated tests in emulators (99%+)
• manual tests on devices before release
• exception: testing video

14. Sagar Wanaselja
Lead Engineer
swanaselja@salesforce.com

15. Problematic WebDriver Test

16. Better WebDriver Test

17. PageObject WebDriver Test

18. Diagnosing Test Failures
• Screenshot and HTML capture at failure point
• AssertionError or RuntimeException
• Differences between localhost and automation VM
• Optimistic timing assumptions
• javascript, rendering, server response, etc.
• Defensive “negative” testing
• Getting to 100% tests passing

19. Pro Tips
• Version tests and app code (and configuration) together
• Use the Page Object model
• Use the API to set up the test, not the UI
• Anticipate Selenium and Browser releases
• Allocate capacity to browsers most used by customers
• log the user agent on every page view

20. More Pro Tips
• Avoid using xpath selectors
• Complex DOM makes xpath evaluation much slower
• Some xpaths are not properly evaluated in WebDriver
• When working with popups make sure you return focus
to the main window
• For static/simple HTML pages consider the
HtmlUnitDriver

22. Mature Quality Strategies
• Selenium for dynamic security vulnerability analysis
• Pixel perfect image diffs
• Accessibility testing (tab completion)
• Testing javascript with unit tests
• Selenium is for testing browser compatibility and DOM