Although Workflow Management Systems (WfMSs) are a key component in workflow technology, research work for assessing and comparing their performance is limited. This work proposes the first micro- benchmark for WfMSs that can execute BPMN 2.0 workflows. To this end, we focus on studying the performance impact of well-known workflow patterns expressed in BPMN 2.0 with respect to three open source WfMSs (i.e., Activiti, jBPM and Camunda). We executed all the experiments under a reliable environment and produced a set of meaningful metrics. This paper contributes to the area of workflow technology by defining building blocks for more complex BPMN 2.0 WfMS benchmarks. The results have shown bottlenecks on architectural design decisions, resource utilization, and limits on the load a WfMS can sustain, especially for the cases of complex and parallel structures. Experiments on a mix of workflow patterns indicated that there are no unexpected performance side effects when executing different workflow patterns concurrently, although the duration of the individual workflows that comprised the mix was increased.

1. Research
Marigianna Skouradaki, Frank Leymann
Institute of Architecture of Application Systems
University of Stuttgart, Germany
Vincenzo Ferme, Cesare Pautasso
Faculty of Informatics,
University of Lugano, Switzerland
André van Hoorn
Institute of Software Technology,
University of Stuttgart, Germany
Micro – Benchmarking BPMN 2.0
Workflow Management Systems
with Workflow Patterns

2. 22
Research
© Marigianna Skouradaki
Motivation
*the appearance and company selection are random

3. 77
Research
© Marigianna Skouradaki
Motivation
Workload
Mix
Results
2. What is the impact of the core BPMN 2.0 language constructs
on the performance of the Workflow Management Systems?
3. Can simple workload mix reveal performance bottlenecks?
1. How to define meaningful workload models candidates?
Benchmark

4. 88
Research
© Marigianna Skouradaki
Agenda
n Background
n Workload Models
n BenchFlow Environment
n Experiments Setup
n Experiments Methodology
n Results
n Discussion
n Conclusions and Future Work

5. 99
Research
© Marigianna Skouradaki
What is a Workflow Management System (WfMS)?
4.Navigator
3.Web Services
6.Application
Server
Workflow
Engine
5.Instance
Database
2.Users
& IT tools
…
1.Process
Models

6. 1010
Research
© Marigianna Skouradaki
What is a Workflow Management System (WfMS)?
4.Navigator
3.Web Services
6.Application
Server
Workflow
Engine
5.Instance
Database
2.Users
& IT tools
…
1.Process
Models

7. 1313
Research
© Marigianna Skouradaki
Micro-Benchmark
n Simple workload and targets the evaluation of atomic
operations1
1Waller, J., Hasselbring, W.: A benchmark engineering methodology to measure the overhead of application-level monitoring.
In: KPDAYS. CEUR Workshop Proceedings, vol. 1083, pp. 59–68. CEUR-WS.org (2013)
2Oracle, “Java Microbenchmark Harness”, (2013). Online resource: http://openjdk.java.net/projects/code-tools/jmh/
3Van Der Aalst, W.M.P., Hofstede, T., et al.: Workflow patterns. Distrib. Parallel Databases 14(1), 5–51 (Jul 2003)
HelloWorld() micro-benchmark measures “nothing”.
It is a good showcase for the overheads in infrastructure2
à For Workflow Management Systems
The basic control flow
workflow patterns can be considered
as atomic operations.
Contribution
Introduce & Execute
the first Micro-Benchmark on
BPMN 2.0 WfMS using
Workflow Patterns

8. 1818
Research
© Marigianna Skouradaki
Workflow Patterns as Workload Models (Constraints)
1. Maximize the simplicity of the process models
2. Omit interactions with external systems (i.e., stress the
Process Navigator)
3. Most script tasks are empty, unless the
implementation of the workflow pattern requires
otherwise
4. Define equal probability for passing the control flow to
the outgoing branches
5. Combine:
- Simple merge + exclusive choice
- Parallel split + synchronization

9. 2323
Research
© Marigianna Skouradaki
Workload Models
Empty
Script 1
Empty
Script 2
Sequence Pattern [SEQ]
parallelSplitPattern
Empty
Script 1
Empty
Script 2
Parallel Split & Synchronization [PAR]
tern
Empty
Script 1
Wait 5
Sec
Explicit Termination Pattern [EXT]ryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
Arbitrary Cycle Pattern [CYC]
exclusiveChoicePattern
generate
number
Empty
Script 1
Empty
Script 2case_2
case_1
Simple Merge & Exclusive Choice [EXC]

10. 2424
Research
© Marigianna Skouradaki
Three Open-Source WfMS Under Test
1. Widely used in the industry
2. Tested against BPMN 2.0 conformance1
3. Vendor provided configurations are used
1.Geiger, M., Harrer, S., Lenhard, J., Casar, M., Vorndran, A., Wirtz, G.: BPMN conformance in open source engines. In: Proc. of the 9th IEEE International
Symposium on Service-Oriented System Engineering (SOSE 2015). SOSE 2015, IEEE, San Francisco Bay, CA, USA (March 30 – April 3 2015)

11. 2525
Research
Vincenzo Ferme
The BenchFlow Environment5
5Ferme, Vincenzo; Ivanchikj, Ana and Pautasso Cesare: "A Framework for Benchmarking BPMN 2.0 Workflow Management Systems",
In: Proceedings of BPM’15, Innsbruck, Austria, Springer, August, 2015.

12. 2626
Research
Vincenzo Ferme
Metrics / KPIs

13. 2727
Research
Vincenzo Ferme
Experiments Methodology
n Experiment 1:
Max(1500) concurrent Instance Producers starting
instances for each workflow pattern [SEQ, EXC, EXT,
PAR, CYC]
n Experiment 2:
1500 concurrent Instance Producers starting an equal
number instances of the workflow patterns (20% [MIX]
of [SEQ, EXC, EXT, PAR, CYC])
à Three rounds per experiment
à Each experiment lasts for 10 minutes
à Discarded the data of the warm-up state (first 1min)

14. 2828
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Empty
Script 2
[SEQ]

15. 2929
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
[EXC]
Pattern
generate
number
Empty
Script 1
Empty
Script 2case_2
case_1

16. 3030
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Wait 5
Sec
[EXT]

17. 3131
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
Empty
Script 1
Empty
Script 2
[PAR]

18. 3232
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)
CYC
Instance Producers:
Camunda: 600
WfMS A: 600
WfMS B: 600
ryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
[CYC]

19. 3333
Research
© Marigianna Skouradaki
Results: Mean Duration (ms)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
8.16
6.39
9.3
2,622
10.06
38.65
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
Activiti jBPM Camunda
SEQ EXC E
0
20
40
60
80
100
43.21
57.42
5.83
5.73
36.75
44.62
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b)
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
(b)MeanCPU(%)Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CPU (%) U
per workflow pattern
Mean duration (ms)

20. 3434
Research
© Marigianna Skouradaki
Results: Mean CPU (%) and RAM (MB) usage
munda
SEQ EXC EXT PAR CYC MIX
0
20
40
60
80
100
43.21
57.42
60.2
66.1
70.09
77.69
5.83
5.73
0.24
5.64
4.67
0.66
36.75
44.62
33.34
41.76
41.67
48.53
(b)MeanCPU(%)
rkflow pattern and (b) Mean CPU (%) Usage
SEQ EXC EXT PAR CYC MIX
0
2,000
4,000
12,074.2
12,215.9
12,025.54
12,201.16
12,340.65
12,310.88
2,936.6
2,976.37
2,747.34
2,935.81
2,851.9
2,786.54
807.81
824.96
794.92
828.37
933.31
991.86
MeanRAM(MB)
WfMS A WfMS B Camunda
Fig. 3. (a) Mean RAM (MB) Usage per workflow
per workflow pattern in [MIX]
SEQ EXC EXT PAR CYC MIX
0
5
10
15
20
0.39
0.48
14.1
13.29
6.23
8.16
6.39
9.3
2,622
10.06
39.36
540.02
0.74
0.85
0.4
0.7
3.06
1.22
(a)MeanDuration(ms)
WfMS A WfMS B Camunda
SEQ EXC EXT PAR
0
20
40
60
80
100
43.21
57.42
60.2
66.1
5.83
5.73
0.24
5.64
36.75
44.62
33.34
41.76
(b)MeanCPU(%)
Fig. 2. (a) Mean Duration (ms) per workflow pattern and (b) Mean CP
per workflow pattern
mentary material. The behaviour of all the WfMSs is discussed th
CPU(%) usage
Memory (MB) Usage

21. 3737
Research
© Marigianna Skouradaki
Results Throughput (wi/s)
SEQ EXC EXT PAR CYC MIX
WfMS A 1,447.66 1,436.03 1,426.35 1,429.65 644.02 1,402.33
WfMS B 63.31 49.04 0.38 48.89 13.46 1.78
Camunda 1,456.79 1,417.17 1,455.68 1,455.68 327.99 1,061.27
CYC Instance Producers: WfMS A: 800 WfMS B: 1500 Camunda: 600
WfMS B uses a synchronous instantiation API.
Therefore, load drivers must wait for the workflow instances to end
Before instantiating the next one

22. 4646
Research
© Marigianna Skouradaki
Findings
Even though the defined workload is simple
SEQ EXC EXT PAR CY
0
5
10
15
20
0.39
0.48
14.1
13.29
2.91
6.39
9.3
2,622
10.06
38.65
0.74
0.85
0.4
0.7
(a)MeanDuration(ms)
Activiti jBPM
Fig. 2. (a) Mean Duration (ms)
per workflow pattern
shows that the duration times
those of [SEQ].
More particularly, we have
and 0.85 ms for Camunda. Co
a slight increase with respect
CPU and 12, 215 MB RAM for
jBPM takes approximately the
workflow instances. For this, it
of RAM, and Camunda 43.21%
765, 274 workflow instances in
4.2.3 Explicit Terminatio
[EXT] executes concurrently an
3. Differences in
the duration
of the
workflow instances
2. Discovered
Bottleneck on WfMS B
1. Differences in
resource utilization
show space for
improvement
4. EXT pattern is
not implemented
consistently on WfMS B
5. Sequential patterns
are better candidates
for discovering the
maximum throughputsequencePattern
Empty
Script 1
Empty
Script 2
6. Complex and parallel
workflows
are better candidates
for stressing the WfMS on
Resource Utilization
parallelSplitPattern
Empty
Script 1
Empty
Script 2
attern
Empty
Script 1
Wait 5
Sec
7. The CYC pattern
caused
time out errors
in Camunda
arbitaryCyclesPattern
generate
1 or 2
Empty
Script 1
Empty
Script 2
i++
i < 10
i >= 10
case_1
case_2
8. The execution
of the MIX
had the expected
behavior

23. 4747
Research
© Marigianna Skouradaki
Future Work
n Finalize and release the BenchFlow framework in an open-source
model on Github and DockerHub:
n Generate synthetic process models w.r.t. the present results
n Execute a macro benchmark with the synthetic, complex,
representative workflow mix
n Execute different tests with realistic load functions and test data
n Production-like configurations
benchflowhttps://github.com/benchflow
goo.gl/Olw32m skourama@iaas.uni-stuttgart.de