The document discusses leveraging DevOps practices to improve mainframe application delivery. It describes how traditional mainframe development and testing causes delays due to shared, restricted resources and inefficient processes. The solution presented uses DevOps tools and practices like continuous integration/delivery, dependency virtualization, and automated quality testing to enable more efficient mainframe application development and testing. This allows development and operations teams to work in parallel, validate code quality earlier, and deploy applications more frequently.

1. DevOps for the Mainframe
Leveraging Continuous Integration, the cloud, and beyond to
deliver z/OS applications
David Myers, Product Manager, Continuous Delivery for
Mainframe and Mobile Software
Rosalind Radcliffe, Distinguished Engineer, Enterprise
Modernization Solution Architect
© 2012 IBM Corporation

2. Please note
IBM’s
statements
regarding
its
plans,
direc3ons,
and
intent
are
subject
to
change
or
withdrawal
without
no3ce
at
IBM’s
sole
discre3on.
Informa3on
regarding
poten3al
future
products
is
intended
to
outline
our
general
product
direc3on
and
it
should
not
be
relied
on
in
making
a
purchasing
decision.
The
informa3on
men3oned
regarding
poten3al
future
products
is
not
a
commitment,
promise,
or
legal
obliga3on
to
deliver
any
material,
code
or
func3onality.
Informa3on
about
poten3al
future
products
may
not
be
incorporated
into
any
contract.
The
development,
release,
and
3ming
of
any
future
features
or
func3onality
described
for
our
products
remains
at
our
sole
discre3on.
Performance
is
based
on
measurements
and
projec3ons
using
standard
IBM
benchmarks
in
a
controlled
environment.
The
actual
throughput
or
performance
that
any
user
will
experience
will
vary
depending
upon
many
factors,
including
considera3ons
such
as
the
amount
of
mul3programming
in
the
user’s
job
stream,
the
I/O
configura3on,
the
storage
configura3on,
and
the
workload
processed.
Therefore,
no
assurance
can
be
given
that
an
individual
user
will
achieve
results
similar
to
those
stated
here.
2

3. New era systems integrate existing operational systems
with rapid delivery of new client-facing apps
Social, Local, Mobile
Smart Infrastructure Analytics
CRM
HR
Legacy
ERP
DB
Manage workloads Rapid innovation User experience and
and maintain security in the cloud mobile management
3
3

4. Evolving customer and market expectations
Capabilities and User Experience Today Emerging
Systems of Record Systems of Engagement (+ Record)
Primary Workload Types Transactional Big Data, Analytics, Mobile/Social Channels
Time to Value Planned Opportunistic
Delivery Model Planned Incremental (DevOps)
Development and Operations Team Sizes 100s and Costly 10s with built-in DevOps automation
Release Frequency Months to Years Hours to Days, based on business opportunity
Integration Frequency Weeks Continuous
Service Sourcing Develop Consume and Assemble (Public and Private)
Operational Model Systems Management Built in to application, Recovery Oriented Computing, Continuous Availability
Infrastructure Deployment Days Minutes
4
Risk Profile Big-Bang (High Risk) Incremental

5. DevOps is…
A set of principles and values that facilitate
collaboration across disciplines to…
1. Enable rapid evolution of deployed business services
2. Reduce risk, decrease cost, and improve quality across
the portfolio
People
Process
DevOps Principles
§ Collaborate across disciplines Tools
§ Develop and test against a production-like system
§ Deploy frequently using repeatable and reliable
processes
§ Continuously monitor and validate operational quality
characteristics
§ Amplify feedback loops
5

6. Solution: A Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Ensure applications are production-ready throughout the lifecycle
and can be released at any time while minimizing rollback due to quality issues
• Validate
on
more
produc3on-­‐like
• Standardiza3on
on
processes
and
assets
condi3ons
earlier
between
Dev
and
Ops
• Automate
hand-­‐offs/promo3ons
to
• Automated
monitoring
and
dashboarding
increase
velocity
through
the
different
of
quality
and
performance
against
service
stages
level
agreements
at
mul3ple
stages
6

7. Mainframe Delivery Pains…
§ Multiple teams working across § Too much bad code going into
restricted dev and test capacity test and production causes crit
lead to conflict, delays, or bad sits and emergency fixes
test results in shared
environments
§ Bottlenecks due to inefficient
§ Complex and manual
communications between
management and configuration disparate platforms and teams
tasks result in errors and delays (Dev/Test - System Programmers;
mobile – distributed-mainframe)
7

8. …solutions from IBM
Play
Defense
§ Provide
cheap,
isolated,
development
§ Enforce
base
quality
standards
and
test
environments
for
project
automa3cally
prior
to
promo3on
teams
– Ra3onal
Development
and
Test
Environment
– Ra3onal
Test
Workbench
– Ra3onal
Test
Virtualiza3on
Server
– Ra3onal
Quality
Manager
– SmartCloud
Provisioning
– SmartCloud
Applica3on
Monitoring
– Cloud
Ready
for
Linux
on
System
z
– Omegamon
§ Automate
consistent
build,
configure,
§ Improve
communica3on
and
and
deploy
processes
across
all
stages
collabora3on
with
cross-­‐plaVorm
– Ra3onal
Team
Concert
release
planning
– SmartCloud
Con3nuous
Delivery
– IBM
Collabora3ve
Lifecycle
Management
– SmartCloud
Orchestrator
– Smart
Cloud
Control
Desk
Go
on
Offense
8

9. DevOps for the common mainframe delivery cycle
1
2
2
Dev
Test
Dev
Test
Dev
Test
Staging
Prod
4
3
Development" Test" System
Programmer"
9

10. Typical z/OS Testing Architecture
Organized by project team, vertically scaled, sharing resources, limited automation
z/OS
Problems Encountered
Test LPAR
Project
Team 1. Shared resources combined
[April Maintain] with overlapping schedules can
App elicit conflicts, impede
innovation and slow code
Project delivery
Team
[Prototype SOA]
2. Coordination of environmental
Test
changes and releases cause
App bottlenecks, delays and
Project Data
Team additional overhead
[June New Func]
3. Shared test data is difficult to
…
manage and can lead to over
testing or incorrect test results
Project
Team App
[Dec Sys Upgrade]
10
10 10

11. Rational Development and Test Environment for
System z
The ultimate in modern application development for System z
COBOL,
PL/I,
C++,
Java,
EGL,
Batch,
Assembler,
Debug
Tool
IMS
DB2
CICS
WAS MQ
RDz user
RDz user z/OS
x86
PC
running
Linux
RDz user
ISPF user
RDz user
RDz & ISPF user
§ Liberate
developers
to
rapidly
prototype
new
applica3ons
§ Develop
and
test
System
z
applica3ons
anywhere,
any3me!
§ Free
up
mainframe
development
MIPS
for
produc3on
capacity
§ Eliminate
costly
delays
by
reducing
dependencies
on
opera3ons
staff
Note:
This
Program
is
licensed
only
for
development
and
test
of
applica3ons
that
run
on
IBM
z/OS.
The
Program
may
not
be
used
to
run
produc3on
workloads
of
any
kind,
nor
more
robust
development
workloads
including
without
limita3on
produc3on
module
builds,
pre-­‐produc3on
tes3ng,
stress
tes3ng,
or
performance
tes3ng.
11

12. Decouple subsystem teams using Green Hat
Improve
tes+ng
with
Green
Hat
service
simula+on
Simulate subsystem dependencies
– Test impact of latency
– Test application response to unresponsive
services
Application changes
Create a stable test environment being tested
– Simulate “public” services
– Reduce capacity requirements on
infrastructure
– Improve security by reducing access from
external services
Databases Mainframe Third-party
applications Services
virtualized services
Ra3onal
Test
Virtualiza3on
Server
12

13. Testing with dependency virtualization
Controlled large system testing by isolating components under test
• Easier problem determination
• Lower test environment capacity requirements
• Improved component quality
Phase 1 Phase 2 Phase 3
Test
Case
Test
Case
Test
Case
IMS
Data 3rd Party
IMS Data 3rd Party
Access Call IMS Data 3rd Party
App
Access
App
Call
Access Call
App
CICS
CICS
Commarea CICS
Commarea
Call Commarea
Call
Call
Virtual
Virtual
Virtual
Services
Services
Services
13

14. Provision new dev / test systems in minutes
• Fast
VM
provisioning
for
near-­‐instant
deployment
of
100s
of
virtual
machines
and
scale
based
on
business
needs
• Rich
set
of
web
interfaces
into
the
cloud
that
can
be
driven
by
a
user
or
scripts
for
complete
automa3on
• Reliable,
non-­‐stop
cloud
capable
of
automa3cally
tolera3ng
and
recovering
from
soeware
and
hardware
failures
• Near-­‐zero
down=me
due
to
faults,
hypervisor/
management
soeware
upgrades
or
addi3on/
removal
of
hardware
Func3on
Test
Performance
Test
Requested
VMs
will
be
up
and
running
under
a
minute
using
standard
HW
14

15. Fail fast and drive better quality downstream
Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)
Automated, Reusable
continuous testing integration
with isolated Continuous Integration for z tests simplify
development and testing across
test environment layers
Check
In
+
Build
+
Deploy
+
Run
Code
Automated
Tests
Change
Promote
Request
To
QA
Frequent, rapid
feedback earlier Developer Higher quality
to improve
going into QA
quality Rapid
Feedback
§ Fast, dependable, automatic feedback speeds time to market
§ Lower cost of application testing using off-mainframe z/OS test environment
§ Enables confidence by automatically tracking and promoting code health
15

16. Continuous Integration for System z Scenario
System z
Test
ests>
pp+zUnit T Environment
<Deploys A (RD&T or LPAR)
<Drives>
SCM/CI <Starts Tests> Test
Server Execution
(Rational Team Manager
Concert) (Ra3onal
Quality
<Results> Manager)
Test
Automation
(Rational Test
1. Check-in code 5. Kick-off Automated Workbench)
2. Build code and zUnit Test Plan
6. Run automated
tests interface tests against
3. Deploy build results 8. Report test results Test Environment
and test data to Test in dashboard/build 7. Mark execution
Environment results/defect records Pass/Fail in
4. Execute zUnit Tests records in CI server. Test Execution 16
Manager
16

17. Separate tools makes collaboration inefficient or
difficult...
§ Each tool came with its own Tool A
Dev work Items Tool B
• UI - Web and desktop
presentations of views and DB
Release
Planning
UI
tasks
• Logic – Workflow, process, LOGIC
DB
UI
search, query, scale, security Tool E
and collaboration LOGIC
• Storage – Availability,
SCM
traceability DB
UI
• Privacy, backup/archive
LOGIC
§ Resulting in... DB
UI
• Brittle integrations
LOGIC
• Silos everywhere DB
UI Tool C
• High cost to maintain and
Tool D LOGIC Reporting
administer
• Proprietary API's Incident
Management
17

18. Tie development release to operational release
Ra3onal
Team
Concert
Release
Plan
and
work
items
SmartCloud
Control
Desk
18

19. Delivery Organized for Flexibility and Speed
Organized by application team, horizontally sliced, dedicated resources, highly automated
z/OS LPAR
Problems Encountered
Project 1. Shared resources
Team
[April Maintain] LPAR combined with
overlapping schedules
Data
can elicit conflicts,
QA/Integra3on
Test
LPAR
impede innovation and
Project
Team slow code delivery
[Prototype SOA] RD&T 2. Coordination of
Data
Integr.
environmental changes
Test
and releases cause
Project Data
bottlenecks, delays and
Team
[June New Func] RD&T
additional overhead
3. Shared test data is
…
Data
difficult to manage and
can lead to over testing
Project or incorrect test results
Team
[Dec Sys Upgrade] LPAR 4. Provisioning, managing,
and synchronizing project
Data
test environments
including data 19
19 19

20. Deploy application and config changes to test environments in minutes
Opera+ons
team
Development
team
establishes
deployment
makes
applica+on
pa;erns
for
use
by
IBM code
and
development
and
test
SmartCloud
configura+on
Continuous
teams
Delivery
changes
Deliver
Environments 3x
Test against
Faster!!
production-like
environments with
Applications &
middleware
AUTOMATICALLY
installed and properly IBM
configured IBM SmartCloud Workload
20
Provisioning Deployer

21. What are we working on in
the lab…
21

22. …solutions from IBM
Play
Defense
§ Provide
cheap,
isolated,
development
§ Enforce
base
quality
standards
and
test
environments
for
project
automa3cally
prior
to
promo3on
teams
– Ra3onal
Test
Workbench
– Ra3onal
Development
and
Test
Environment
– Ra3onal
Quality
Manager
– Ra3onal
Test
Virtualiza3on
Server
– SmartCloud
Applica3on
Monitoring
– SmartCloud
Provisioning
– Omegamon
– Cloud
Ready
for
Linux
on
System
z
§ Automate
consistent
build,
provision,
§ Improve
communica3on
and
configure,
and
deploy
pof
zLinux
across
collabora3on
with
cross-­‐plaVorm
Cloud-­‐style
deploy
rocesses
all
stages
dev
and
test
paMerns
and
z/OS
release
planning
on
RD&T
or
zHardware
– Ra3onal
Team
Concert
– IBM
Collabora3ve
Lifecycle
Management
– SmartCloud
Con3nuous
Delivery
– Smart
Cloud
Control
Desk
– SmartCloud
Orchestrator*
Go
on
Offense
22

23. Provisioning using Cloud
#!/usr/bin/env ruby
class DevopsDeployer
def initialize(build_url, build_id)
@log = Logger.new(LOG_FILE)
@log.level = LOG_LEVEL
@iaas_gateway = IaasGateway.new(HsltProvider.new(),
LOG_FILE, LOG_LEVEL)
@server_instance = nil
.cbl
.cpy
rtc_build_system_provider = RtcBuildSystemProvider.new(
RTC_REPOSITORY_URL, RTC_USER_ID, RTC_PASSWORD_FILE)
@build = rtc_build_system_provider.resolve_build(
build_url, ENV['buildResultUUID'], build_id)
Infrastructure
as
Code
@build_system_gateway = BuildSystemGateway.new(
.asm
rtc_build_system_provider, LOG_FILE, LOG_LEVEL)
end
def add_build_stamp
template_file = WEB_APP_ROOT +
"/app/templates/pages/page.html"
@log.info "Adding build ID stamp #{@build.id} to
.rexx
#{template_file}" CLIST
# Read in the file's contents as a string, replace
# the build_id, then overwrite the original contents
# of the file
text = File.read(template_file)
new_text = text.gsub(/{{ build_id }}/,
Source
Ar3facts
"<a href="#{@build.uri}">#{@build.id}</a>")
File.open(template_file, "w") { |file|
}
end
Source
Control
file.puts new_text
# ... Management
23

24. Delivery Pipeline Fundamentals
Build,
Package,
&
Unit
Test
Applica+on
.cbl
.cpy
Binaries
&
PlaGorm
Configura+on
Deploy
.asm
.rexx
CLIST
Environment
Deployable
Ar3facts
Running
System
Source
Ar3facts
Source
Control
Library
Management
24

25. Consolidate, standardize, simplify deployment
Enterprise
Applica3ons
have
thousands
of
disparate
parts
• Currently
maintained
in
separate
systems,
if
maintained
at
all
• Limited
linkage
between
systems
for
applica3on
and
configura3on
dependencies
• Missing
assets
and
informa3on
is
rampant
DevOps
force
linkages,
automa3on,
and
standardized
packaging…
Copybooks
HTML
COBOL
JCL
Build,
Package,
WSDL
&
Unit
Test
MQ
Applica+on
Deploy
Java
Queue
Binaries
&
Def
PlaGorm
Configura+on
DB2
IMS
DBD/PSB
DDL
DB2
Plans
Environment
Deployable
Ar3facts
Running
System
Procs
Library
Source
Management
25

26. Create, secure, and deploy test data
Automated Build &
Deploy Test Data
-Compare z/OS
-Subset -Refresh UT
RD&T
2TB 1 GB
Test Data
z/OS
Production or
UT
LPAR
Production Clone
1 GB
Masked Test Data
Automated
Testing
Create “right-size” De-identify sensitive information Automated test data
production-like environments De-identify sensitive information
with realistic but fictional data for deployment for each
for application testing with realistic purposes
testing & development but fictional data for
testing & development purposes build and test
Optim RTC 26

27. Standardize z/OS region configurations to ease
deployment
Configura3on
Applica3on
• Standard topologies exist today
(production LPARs)
• Standardized regions which can be
Data
repeatedly deployed are rare
• Standardized/Automated deployment of
COMPLETE system is spotty
CICS
5.1
IMS
11.1
Standard
Middleware
Palern
To adopt the cloud for testing…
Standardize and automate provisioning of
Z/OS
1.12
DB2
10.1
everything …
27

28. Standardize z/OS region configurations to ease
deployment
RTC/CIz Available
Application
Capabilities
Configura3on
Applica3on
load modules
Data
JCL and REXX
Test Data
CICS
5.1
IMS
11.1
Pattern/Platform
Standard
Middleware
Palern
Z/OS
1.12
DB2
10.1
28

29. Evolving to deploying regions … cloud-style
“Cloud”
CICS+DB2
RD&T
CICS+
IMS
Or
DB
LPAR
Region/Middleware
RD&T
Palerns
Or
Batch
LPAR
RD&T
Or
LPAR
RD&T
Or
LPAR
SmartCloud
Orchestrator
+
<SCD
for
Mainframe
Project>
29

30. Acknowledgements and Disclaimers:
Availability.
References
in
this
presenta3on
to
IBM
products,
programs,
or
services
do
not
imply
that
they
will
be
available
in
all
countries
in
which
IBM
operates.
The
workshops,
sessions
and
materials
have
been
prepared
by
IBM
or
the
session
speakers
and
reflect
their
own
views.
They
are
provided
for
informa3onal
purposes
only,
and
are
neither
intended
to,
nor
shall
have
the
effect
of
being,
legal
or
other
guidance
or
advice
to
any
par3cipant.
While
efforts
were
made
to
verify
the
completeness
and
accuracy
of
the
informa3on
contained
in
this
presenta3on,
it
is
provided
AS-­‐IS
without
warranty
of
any
kind,
express
or
implied.
IBM
shall
not
be
responsible
for
any
damages
arising
out
of
the
use
of,
or
otherwise
related
to,
this
presenta3on
or
any
other
materials.
Nothing
contained
in
this
presenta3on
is
intended
to,
nor
shall
have
the
effect
of,
crea3ng
any
warran3es
or
representa3ons
from
IBM
or
its
suppliers
or
licensors,
or
altering
the
terms
and
condi3ons
of
the
applicable
license
agreement
governing
the
use
of
IBM
soeware.
All
customer
examples
described
are
presented
as
illustra3ons
of
how
those
customers
have
used
IBM
products
and
the
results
they
may
have
achieved.
Actual
environmental
costs
and
performance
characteris3cs
may
vary
by
customer.
Nothing
contained
in
these
materials
is
intended
to,
nor
shall
have
the
effect
of,
sta3ng
or
implying
that
any
ac3vi3es
undertaken
by
you
will
result
in
any
specific
sales,
revenue
growth
or
other
results.
©
Copyright
IBM
Corpora6on
2013.
All
rights
reserved.
§ U.S.
Government
Users
Restricted
Rights
-­‐
Use,
duplica6on
or
disclosure
restricted
by
GSA
ADP
Schedule
Contract
with
IBM
Corp.
§ Please
update
paragraph
below
for
the
par+cular
product
or
family
brand
trademarks
you
men+on
such
as
WebSphere,
DB2,
Maximo,
Clearcase,
Lotus,
etc
IBM,
the
IBM
logo,
ibm.com,
[IBM
Brand,
if
trademarked],
and
[IBM
Product,
if
trademarked]
are
trademarks
or
registered
trademarks
of
Interna3onal
Business
Machines
Corpora3on
in
the
United
States,
other
countries,
or
both.
If
these
and
other
IBM
trademarked
terms
are
marked
on
their
first
occurrence
in
this
informa3on
with
a
trademark
symbol
(®
or
™),
these
symbols
indicate
U.S.
registered
or
common
law
trademarks
owned
by
IBM
at
the
3me
this
informa3on
was
published.
Such
trademarks
may
also
be
registered
or
common
law
trademarks
in
other
countries.
A
current
list
of
IBM
trademarks
is
available
on
the
Web
at
“Copyright
and
trademark
informa3on”
at
www.ibm.com/legal/copytrade.shtml
If
you
have
men3oned
trademarks
that
are
not
from
IBM,
please
update
and
add
the
following
lines:
[Insert
any
special
3rd
party
trademark
names/alribu3ons
here]
Other
company,
product,
or
service
names
may
be
trademarks
or
service
marks
of
others.
30

31. www.ibm.com/software/rational
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
31