This Open Source Test Workshop is for senior IT and business executives needing visibility and management tools and methodology into all the demand for IT. Shows how to bring Open Source Testing into your organization.

1. Surface and Mitigate Performance Bottlenecks
Open Source Test Workshop
For CIOs, CTOs, Senior Managers
Frank Cohen, +01 (408) 871-0122, fcohen@pushtotest.com
June 8, 2011
Wednesday, June 8, 2011

2. Open Source Test Automation
Slides on Twitter
‣ http://twitter.com/fcohen
‣ Please retweet
‣ Please follow me
‣ #ost
2
Wednesday, June 8, 2011

3. Open Source Test Automation
About PushToTest
‣ Our Mission: We Test, For A More Reliable World
‣ Make It Easier To Move From Manual To Automated Testing
‣ Support Advanced Testing, including Agile, Load, Integration
‣ Open Source Testing (OST) For Everyone
3
Wednesday, June 8, 2011

4. XML, SOA, and Performance
Web 2.0 Success Driving Business
‣ Every Organization Wants Modern IT Functions and User Interfaces
‣ Rich User Interfaces (RIA, using Ajax, Flash, Flex)
‣ SOA Composite Applications
• Faster Time-To-Market
• Increase Revenue
• Cost Savings
‣ Service Enabled Applications
• Deliver More Integration Faster
• Always-On
• Less Cost
4
Wednesday, June 8, 2011

5. XML, SOA, and Performance
The Reality of Web 2.0 Development
‣ We Build Applications Differently Now
‣ Haze of New Protocols, Software Models, Data Formats
‣ Agile Often Means Shorter Build-Test-Deploy ‘Waterfall-style’ Cycles
‣ Many Still Trying “Test And Trash” Techniques
‣ Uptime Depends on Business Service Management (BSM) Testing
to Surface Functional and Performance Issues
Every 8 weeks 30% of the Application Changes
5
Wednesday, June 8, 2011
30% Tests Re-recorded

6. Open Source Test Automation
Testing Costs
‣ Proprietary Test Tools Offer:
Functional Test Load Test
Functional Test Platform $19,000 Load Test Platform $20,000
Functional Tests $8,000 Simulated User Licenses $120,000
Protocols: AJAX, SOAP $12,000 Recoding The Test $35,000
Support $5,000 Re-quality The Test $12,000
$40,000 $187,000
‣ QA Labs for 10,000 Users = $35,000 Per Year
‣ DoesCause Analysis Consultants
Root
Not Include: Cost of Test Employees,
‣ Does Not Guarantee Success
6
Wednesday, June 8, 2011

7. Open Source Test Automation
Proprietary Testing Falling Behind
‣ Adobe Releases Testing APIs for Flex/Flash in 2009
‣ JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010
‣ Appcelerator Titanium Makes 3 Releases in 2010
‣ Microsoft Draws 8,000 to Mix for StarLight in 2010
‣ StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC,
LinuxWorld Feature Keynote Talks On Ajax
‣ “HP Refundedcustomer in theFor The Ajax industry. Handler”,
a PushToTest
Our Payment
automotive
Protocol
7
Wednesday, June 8, 2011

8. Open Source Test Automation
Open Source Testing Emerges
W
tness
Lots Of Options
Selenium
HtmlUnit
soapUI
DBUnit
Sahi
Watir
FlexMonkey
FIT
Windmill Hudson TestMaker jMeter
‣ Signs of Maturity:Logger, Recovery Scenario Manager, Object
Iteration Engine,
Record/Playback, Data Generation/Importer,
8 Recognizer
Wednesday, June 8, 2011

9. XML, SOA, and Performance
Depend On A Service Partner
‣ Repurposing: One Test Script For
Functional Tests, Load and Performance
Tests, and Business Service Monitors
‣ Components: Tests Run Even After
Application Changes
‣ Deployment: Run Tests In Your Lab,
Ours, or Both!
‣ Services: Add Expertise, Training, Tools
To Your Organization
‣ Enthusiasm:Openparticipate and
contribute to
We
Source Test Projects
9
Wednesday, June 8, 2011

10. Open Source Test Automation
Manual Testing Process
Results
4 1
Recommend Changes
Explore
Manual
Testing
Tes
3 t
2 Plan
10 Run The Test Steps
Create Test Plans
Wednesday, June 8, 2011

11. Open Source Test Automation
Side Effects
‣ Exhaustion
‣ Fatigue
‣ Blurry Vision
‣ Night Sweats
‣ Insanity
‣ Hair Loss
‣ Acne
‣ Divorce
11
Wednesday, June 8, 2011

12. Open Source Test Automation
OST Architecture
Application Under Test PTTMonitor
Web, RIA, BPM, SOA CPU, Net, Memory
Functional
Test Injector Test Authoring
Test
Bundles ScriptRunners Data Production Libraries SoapUI (Pro)
MySQL, HtmlUnit Sahi, Selenium, SoapUI, Java csv files, rdbms queries Sahi (Pro)
JUnit Class Framework
Load Test Eclipse Java IDE
Fitness
Test Operation Console Languages: Java, Python
Wiki Collaboration
Production Designer Grid and Cloud Repository Results Analysis
Record/Playback Test Deployment MySQL RDBMS Business Intelligence Tool
Monitor
Continuous Test
Repository Bug Tracker Monitoring
Integration Management
12
Wednesday, June 8, 2011

13. Open Source Test Automation
An OST Cloud
Thousands of virtual users Stores results of every
TestNode perform real-life scenarios on Results DB transaction, step, and
the Application Under Test monitored resource
PTTMonitor
Application Under Test
Load Injectors
400+ charts identify the root
Analysis cause to performance
bottlenecks, functional issues
Central location from which a
TestMaker
tester, developer, IT manager, CI
Console service runs and monitors tests
13
Wednesday, June 8, 2011

14. Open Source Test Automation
An OST Cloud
Thousands of virtual users Stores results of every
TestNode perform real-life scenarios on Results DB transaction, step, and
the Application Under Test monitored resource
PTTMonitor
Application Under Test
Load Injectors
400+ charts identify the root
Analysis cause to performance
bottlenecks, functional issues
Central location from which a
TestMaker
tester, developer, IT manager, CI
Console service runs and monitors tests
14
Wednesday, June 8, 2011

15. Open Source Test Automation
Browser Compatibility Tests
Application Under Test
TestNode
Selenium RC SeleniumHtmlUnit
Selenese Test
TestMaker Console
15
Wednesday, June 8, 2011

16. Open Source Test Automation
ScriptRunners
ScriptRunner Type Name (langtype)
Designer Script IE, Htmlunit designerscript
Selenium HtmlUnit Browser selenium
Selenium RC IE, Firefox, Chrome seleniumrc
Sahi IE, HtmlUnit sahi
HTTP Archive Page Optimization HAR
Selenium RC IE, Firefox, Chrome seleniumrc
SoapUI SOAP and REST soapui
TestGen4Web Web 1.0 Tests testgen4web
Mozmill Mozilla XUL Tests mozmill
Unit Tests jUnit TestCase Java, Jython, JRuby,
PHP, Perl, Groovy
DotNet .NET Tests dotnet
Visual Basic VB tests vb
16
Command Line Desktop and utils commandline
Wednesday, June 8, 2011

17. Open Source Test Automation
Component Approach
TestScenario Component Application
Log-In
Load Test
100 Users 500 Users 1000 Users setUp
Data runTest
Test Use Case
Database Production
tearDown
Development RDBMS DPL Log-In
Database CSV DPL Product Search Component
QA Regression Order Product
Custom DPL
Database Confirm Order Component
Load Test
Log-Out
Database
Production
17
Wednesday, June 8, 2011

18. Open Source Test Automation
Lots of Possibilities TestMaker
Test Sent
Seattle, Datacenter
TestNode Generating Load
Dallas, Texas
Datacenter
You’re Such A Geek!
18
Wednesday, June 8, 2011

19. Open Source Test Automation
TestMaker Demonstration
‣ Fun And Easy Test Script Creation
‣ Selenium and TestMaker
‣ Repurposing A Single Test
‣ Functional Test, Load and Performance Test, Production Monitor
‣ Results Analysis
‣ Chart Engine
19
Wednesday, June 8, 2011

20. Open Source Test Automation
Root Cause Analysis
‣ PTTMonitor - Agent Based Monitor Gateway
‣ Glassbox - Performance Monitoring for Java Applications
‣ dynaTrace - Analysis In Application and DB Tiers
20
Wednesday, June 8, 2011

21. Open Source Test Automation
PTTMonitor
‣ Correlate Test Operation To Resource Usage
‣ Deploy as Web Application (WAR) and Service
‣ Observes CPU, Network, Memory Utilization As Test Runs
‣ Gateway to Glassbox, dynaTrace, Windows PerfMon
21
Wednesday, June 8, 2011

22. Open Source Test Automation
Results Analysis Engine
22
And 350 Additional Charts
Wednesday, June 8, 2011

23. Open Source Test Automation
Glassbox
‣ Correlate Test Operation To Application Tier Issues
‣ Memory Leaks
‣ Thread Deadlocks
‣ Slow DB Connections
‣ No Instrumentation of Applications
‣ Live Performance Alert View
‣ Run In Production
23
Wednesday, June 8, 2011

24. Open Source Test Automation
Glassbox Troubleshooter
24
Wednesday, June 8, 2011

25. Open Source Test Automation
dynaTrace PurePaths
‣ PurePath from TestMaker Virtual Users To Application & DB Tier
‣ Answers: What Are The Transactions and Why Are They Slow?
dynaTrace monitors
applications for
functional issues
dynaTrace
Database Agent
and performance
bottlenecks.
dynaTrace
Application Server Agent
PurePath Token
TestNode TestNode TestNode
test test test
PTTMonitor
Test RunTime
25
TestMaker Console
Wednesday, June 8, 2011

26. Open Source Test Automation
Pinpoint The Problem
‣ Down to the Line of Source Code
26
Wednesday, June 8, 2011

27. Open Source Test Automation
.NET Monitoring
‣ Correlate PerfMon Statistics with Test Operation
‣ PTTMonitor Gateways Data from PerfMon or MS SQL Server
.NET Application Tier PerfMon MS SQL Server
SOAP HTTP REST
TestNode TestNode TestNode
PTTMonitor
test test test
Test RunTime
TestMaker Console
27
Wednesday, June 8, 2011

28. Open Source Test Automation
Next let’s talk about
PushToTest Options
‣ OnDemand
‣ QuickStart
‣ POC Option
28
Wednesday, June 8, 2011

29. Open Source Test Automation
Immediate Benefits
‣ Product Solutions
‣ Better, Lower-Cost, More Efficient Test Tool
‣ Business Service Monitoring at the Application Tier (SLA
Compliance)
‣ Methodology Solutions
‣ Improved Test Methodology for my Organization and Team
‣ Services
‣ Turnkey Load Testing
‣ Integrate TestMaker Into My Build Environment
‣ Add A Protocol, Chart, Task to TestMaker To Meet My Needs
29
Wednesday, June 8, 2011

30. Open Source Test Automation
Dual License
‣ Functional Testing with TestMaker Object Designer ‣ TestMaker Enterprise expands on the success of
‣
TestMaker Community
Repurposing Selenium, soapUI Sahi and unit tests into
data-driven functional tests, load and performance ‣ Load Testing to Millions of Virtual Users
‣
tests, and business service monitors
Cloud and Grid Testing
‣ Load Testing To 50 Virtual Users
‣ Data-driven tests using Data Production Libraries for
‣ Data-driven tests using CSV files. CSV files, RDBMS, Data Generating Objects
‣ PTTMonitor Correlates CPU, Network, Memory ‣ Root cause analysis and mitigation using Glassbox
‣
Utilization As Tests Operate.
Mock services for agile service development
‣ 25 Charts and Reports
‣ Flex AMF Protocol Handler for Flex/Flash testing
‣ Community Support and access to all documentation
‣ 1000+ Charts and Reports
‣ Business service monitors, including email notification
‣ Support, QuickStart, Training, Consulting
30
Wednesday, June 8, 2011

31. Open Source Test Automation
Pricing
‣ Annual Subscription Model Based On:
‣ Seats for TestMaker Console Installation
‣ Seats for SOAP and REST Test Authoring
‣ Virtual User Levels for Load Testing
‣ Count of Business Service Monitor
‣ Technical support: 1messagesday or faster response to emails,
phone calls, instant
business
31
Wednesday, June 8, 2011

32. Open Source Test Automation
‣ As-Needed, On-Demand Basis
‣ Test In Your Environment,
Or Our Cloud Test Environment,
Or Both
‣ Ideal For Consulting Organizations
Where Test Needs Are Unknown
‣ AMI Distribution
‣ Pay Only For What You Use
‣ Annual Cap On Total Costs
32
Wednesday, June 8, 2011

33. Open Source Test Automation
The QuickStart Option
‣ Your First Test, Built and Operated by PushToTest Experts
‣ Ideal For Organizations With Little or No Load Testing Experience
‣ PushToTest Experts Build Your First Test
‣ PushToTest Experts Operate The Test,
Provide Root Cause Analysis and Mitigation
‣ PushToTest Trains Your Team To Build Their Own Tests
33
‣ $9,750 USD
Wednesday, June 8, 2011

34. Open Source Test Automation
The POC Option
‣ PushToTest Experts Help Your TestMaker Evaluation
‣ Ensures TestMaker Works In Your Application/System
‣ PushToTest Issues Price Proposal
‣ Applies Engineers To Build and Operate Your Test
‣ PushToTest Trains Your Team To Use TestMaker
‣ Success of POC Confirms Intention To Purchase
34 ‣ Cost: $0
Wednesday, June 8, 2011

35. Open Source Test Automation
PushToTest Global Services
‣ Put Our Experts To Work For Your Projects
‣ Methodology, Solutions, Support, Enthusiasm
‣ Offices: USA (Texas and California,) Bangalore, London
‣ The Near Shore Option
‣ 150 Test Experts
‣ Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours
‣ http://www.pushtotest.com/services
“PushToTest Uniquely Built Load Tests For Our
Customers, Including Pepsi, And Worked With Our
Engineers To Surface and Solve Performance Bottlenecks.”
Alan Rodriguez, Director of Testing,
35 Tribal DDB Dallas
Wednesday, June 8, 2011

36. Open Source Test Automation
Slides on Twitter
‣ http://twitter.com/fcohen
‣ Please retweet
‣ Please follow me
‣ #ost
36
Wednesday, June 8, 2011

37. Open Source Test Automation
Where To Go From Here
‣ Please Complete The Evaluation Form
‣ And We Will Send You The Slides
‣ Watch A Screencast and Tutorial
‣ http://www.pushtotest.com/screencastcentral
‣ Attend An Open Source Test Workshop
‣ http://workshop.pushtotest.com
‣ Ask For A Proposal on Licenses, Training, Support, Consulting
‣ Call +01 408 871 0122, sales@pushtotest.com
37
Wednesday, June 8, 2011