Keith Williams, Senior Software Engineer in Test at Brightcove presented at MassTLC's automated testing event on June 12, 2013 at Brightcove, Boston, MA

1. Balancing Test Automation
in a Continuous Delivery
World

2. kwilliams@brightcove.com
twitter: @bckwilliams
Keith Williams
Brightcove Automation Specialist

3. Composition of Effective Test Strategy
Component Test
API or Integration
Test
Webdriver

4. • Low level - testing within a component
• Developed in same language as product
• Written by developers
• Use TDD if it works for your team
• Heavily mocked, eliminating DB and other
service interactions
Component Testing

5. • Easy to develop tests - function/method
based
• Run as first step of every code checkin
• Very fast runtime with many tests
• Easiest tests for great code coverage
• Depending on language, some great tools
available (RSpec, JUnit, Midje,
CoffeeScript, etc)
Component Testing - Benefits

6. • Tests the exposed API of a running service
• Tests target the service's API, but may use
other APIs
• Define each service's API specification and
test cases
• May leverage mocking of other services or
DB
API Testing

7. • Developer and CD pipeline run after
deployment
• Fast running with many tests
• Catches server configuration and
deployment issues
• API_Test Framework for RESTful APIs
API Testing - Benefits

8. • Browser based testing of deployed stack
• Tests are a flow through web pages perhaps in
different services
• Few or no mocks
• Fewer tests that are slow running
• More difficult to develop
Webdriver Testing

9. • Full Integration tests
• Good for testing services with UI components
• Catches environment, networking,
configuration, deployment and service issues
Webdriver Testing Benefits

10. • Selenium 2.0 based with Ruby page_object
• Pages of each service are in separate gem
• Tests reside in the service source tree
Webdriver Framework

11. • API Specification
o Service
 Resource (base_url)
• Operation
o http verb, list of urls (versions)
• API Shortcuts - ease use from other tests
API_Test Framework

12. • API Tests - reside with service source
• Allow table based specification of test cases
• Easy to understand what tests are
present/missing for a resource/operation
• Automatically expanded for all operations
(versions) and urls
• Easy to add new positive/negative tests
API_Test Framework(cont'd)

13. ?
Questions