In the Melbourne edition of a 4-city Technology Radar roadshow, ThoughtWorks Australia's Head of Technology Scott Shaw and senior consultant Jen Smith cover topics from all 4 quadrants of the latest edition of the ThoughtWorks Technology Radar. This presentation covers Reactive Architectures, Hamms, Spring Boot vs. Nancy, and Impala.

1. TECHNOLOGY
RADAR
May 2015 — Our thoughts on the technology and trends that are shaping the future
1

2. 2

3. 3
TECHNOLOGYADVISORYBOARD

4. 4

5. 5

6. 6

7. THEMES FOR THIS ISSUE
7

8. TECHNIQUES
8

9. TECHNIQUES
8

10. 9
ADOPT
1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery
3. Generated infrastructure diagrams NEW
4. Structured logging
TRIAL
5. Canary builds
6. Datensparsamkeit
7. Local storage sync
8. NoPSD
9. Offline-first web applications NEW
10. Products over projects NEW
11. Threat Modelling NEW
ASSESS
12. Append-only data store
13. Blockchain beyond Bitcoin
14. Enterprise Data Lake
15. Flux NEW
16. “git-based CMS” NEW
17. Phoenix environments NEW
18. Reactive architectures NEW
HOLD
19. Long lived branches with Gitflow
20. Microservice envy
21. Programming in your CI/CD tool
22. SAFe™
23. Security sandwich
24. Separate DevOps team
TECHNIQUES

11. 10
TECHNIQUES
Architectures for the digital world
(theme: innovations in architecture)

12. 10
TECHNIQUES
18
15
14
12
REACTIVE ARCHITECTURES
FLUX
ENTERPRISE DATA LAKE
APPEND-ONLY DATA STORE
Architectures for the digital world
(theme: innovations in architecture)

13. Attrition
Acquisition
Retention
Activation
Referral
Ads
ARCHITECTURES FOR THE DIGITAL JOURNEY
11
Emails
Google

14. Legacy Systems
RIGHT DATA, RIGHT PLACE, RIGHT TIME
12
Legacy SystemsLegacy Systems Web Analytics Operational Metrics Insights

15. 13
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
Microservices
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)

16. 13
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
Microservices
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)

17. 14
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)Microservices
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)

18. 14
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)Microservices
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)

19. 15
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)Microservices

20. 15
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Business
Logic
(functions)
Legacy
Systems
Archivers
File Store
(S3)Microservices

21. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
Microservices
Append-
only
Database
Flux-based Web
Application
User Actions
View Rendering
(react.js)
Transactions,
Web Analytics,
Operational Logs
Subscribed
Events
Business
Logic
(functions)
Legacy
Systems
Archivers

22. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)

23. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE

24. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE
CONSUMER

25. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE
CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS
COMPLEXITY

26. 16
Insights
Analytics
Reports,
Model
Parameters
Spark, Hadoop
File Store
(S3)
Event Queue
(Time Series Database,
Apache Kafka,
AWS Kinesis,
Eventstore, …)PUSHES RESPONSIBILITY FOR DATA QUALITY BACK ON
THE SOURCE SYSTEMS
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE
CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS
COMPLEXITY

27. 17
ADOPT
1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery
3. Generated infrastructure diagrams NEW
4. Structured logging
TRIAL
5. Canary builds
6. Datensparsamkeit
7. Local storage sync
8. NoPSD
9. Offline-first web applications NEW
10. Products over projects NEW
11. Threat Modelling NEW
ASSESS
12. Append-only data store
13. Blockchain beyond Bitcoin
14. Enterprise Data Lake
15. Flux NEW
16. “git-based CMS” NEW
17. Phoenix environments NEW
18. Reactive architectures NEW
HOLD
19. Long lived branches with Gitflow
20. Microservice envy
21. Programming in your CI/CD tool
22. SAFe™
23. Security sandwich
24. Separate DevOps team
TECHNIQUES

28. TOOLS
18

29. TOOLS
18

30. 19
TOOLS
ADOPT
48. Composer
49. Go CD
50. Mountebank
51. Postman
TRIAL
52. Boot2docker
53. Brighter NEW
54. Consul
55. Cursive
56. Gitlab
57. Hamms NEW
58. IndexedDB
59. POLLY NEW
60. Rest-assured NEW
61. Swagger
62. Xamarin
63. ZAP NEW
ASSESS
64. Apache Kafka NEW
65. Blackbox
66. Bokeh/Vega NEW
67. Gor NEW
68. NaCL NEW
69. Origami NEW
70. Packet beat
71. pdfmake NEW
72. PlantUML NEW
73. Prometheus NEW
74. Quick NEW
75. Security Monkey NEW
HOLD
76. Citrix for development

31. 20
TOOLS

32. 20
TOOLS
57
HAMMS

33. WE BUILD DISTRIBUTED SYSTEMS BY DEFAULT
21

34. 22

35. 22
FAILUREISTHENORM…
but testing it is tough

36. HAMMS
Simulating badly behaved servers
https://github.com/kevinburke/hamms

37. STARTING
24

38. STARTING
24

39. IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6

40. IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6

41. NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25

42. NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25

43. OTHER FAILURE MODES
27
Invalid content lengths and types Non-http responses
Slow responses Partial responses

44. WHEN WOULD YOU USE HAMMS?
28
Automated testing Usability testing

45. BENEFIT OF COMMON PROTOCOLS
29
HTTP, TCP/IP
Well-known failure cases
Build on other people’s hard work!

46. GOING BEYOND UNIT TESTING…
30
ZED ATTACK PROXY
CONSUMER-DRIVEN CONTRACT TESTING
REST-ASSURED
GORHAMMSTEST.GENERATIVE
QUICKCHECK
SIMULANT

47. RESOURCES
31
https://github.com/kevinburke/hamms

48. 32
TOOLS
ADOPT
48. Composer
49. Go CD
50. Mountebank
51. Postman
TRIAL
52. Boot2docker
53. Brighter NEW
54. Consul
55. Cursive
56. Gitlab
57. HAMMS NEW
58. IndexedDB
59. POLLY NEW
60. Rest-assured NEW
61. Swagger
62. Xamarin
63. ZAP NEW
ASSESS
64. Apache Kafka NEW
65. Blackbox
66. Bokeh/Vega NEW
67. Gor NEW
68. NaCL NEW
69. Origami NEW
70. Packet beat
71. pdfmake NEW
72. PlantUML NEW
73. Prometheus NEW
74. Quick NEW
75. Security Monkey NEW
HOLD
76. Citrix for development

49. LANGUAGES &
FRAMEWORKS
33

50. LANGUAGES &
FRAMEWORKS
33

51. 34
LANGUAGES &
FRAMEWORKS
ADOPT
77. Nancy
TRIAL
78. Dashing
79. Django Rest
80. Ionic Framework
81. Nashorn
82. Om
83. React.js
84. Retrofit
85. Spring Boot
ASSESS
86. Ember.js NEW
87. Flight.js
88. Haskell Hadoop library
89. Lotus
90. Reagent
91. Swift
HOLD
92. JSF

52. 35
LANGUAGES &
FRAMEWORKS

53. 35
LANGUAGES &
FRAMEWORKS
85 SPRING BOOT
NANCY77

54. A TALE OF TWO WEB FRAMEWORKS
36
Java/Spring C#/.NET
Lightweight ✓ ✓
Low-ceremony ✓ ✓
Self-hosted ✓ ✓
Opinionated ✓ ✓
boot

55. ON THE SURFACE, VERY SIMILAR
37

56. ON THE SURFACE, VERY SIMILAR
37

57. BUT WHAT’S UNDER THE COVERS?
38
Spring Boot’s pom.xml
1847 lines in total!

58. BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page

59. BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page

60. FRAMEWORKS VS. COMPOSITION
39
Spring Framework
Your Spring Boot App
Jetty
Your
App
Code
Owin
Nancy.Owin
Nancy
Composes
Calls higher-order functions

61. 40
LANGUAGES &
FRAMEWORKS
ADOPT
77. Nancy
TRIAL
78. Dashing
79. Django Rest
80. Ionic Framework
81. Nashorn
82. Om
83. React.js
84. Retrofit
85. Spring Boot
ASSESS
86. Ember.js NEW
87. Flight.js
88. Haskell Hadoop library
89. Lotus
90. Reagent
91. Swift
HOLD
92. JSF

62. PLATFORMS
41

63. PLATFORMS
41

64. 42
PLATFORMS
ADOPT
TRIAL
25. Apache Spark NEW
26. Cloudera Impala NEW
27. DigitalOcean
28. TOTP Two-Factor Authentication
HOLD
45. Application Servers NEW
46. OSGi
47. SPDY NEW
ASSESS
29. Apache Kylin NEW
30. Apache Mesos
31. CoreCLR and CoreFX NEW
32. CoreOS
33. Deis NEW
34. H2O NEW
35. Jackrabbit Oak
36. Linux security modules
37. MariaDB
38. Netflix OSS Full stack
39. OpenAM
40. SDN
41. Spark.io
42. Text it as a service / Rapidpro.io
43. Time-series Databases NEW
44. U2F

65. 43
PLATFORMS

66. 43
PLATFORMS
26CLOUDERA IMPALA

67. 44
IMPALA
Elegant, nimble querying of
big, lumbering data

68. 45
HADOOP
Fast processing
Scalable
Commodity hardware
Elegant failure handling
HDFS

69. MAPREDUCE
46
• Powerful … but complicated
• Not always the right model
Input
Splitting Mapping
Sorting
Reducing

70. SQL
47
SELECT …
FROM …
WHERE …
GROUP BY …
Declarative
Simple
Well known

71. RELATIONAL DATABASES
48
Not always good choice
for adhoc analysis:

72. RELATIONAL DATABASES
48
Not always good choice
for adhoc analysis:
• Data needs to be
optimised for
consumption
ETL
?
?

73. RELATIONAL DATABASES
48
Not always good choice
for adhoc analysis:
• Data needs to be
optimised for
consumption
• Scaling is tough
?
?
?
?
ETL
?
?

74. IMPALA
49
An SQL engine that runs on Hadoop

75. IMPALA IN ACTION
50
External tables
create EXTERNAL TABLE books( id BIGINT, isbn
STRING, category STRING, publish_date TIMESTAMP,
publisher STRING, price FLOAT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LOCATION '/data/books/';

76. IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
Real-time querying of large datasets

77. IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
+---------------------------+----------+
| category | count(*) |
+---------------------------+----------+
| COMPUTERS | 312604 |
| SPORTS-RECREATION | 312766 |
| PETS | 313039 |
| CRAFTS-HOBBIES | 312438 |
| FAMILY-RELATIONSHIPS | 312435 |
| HOUSE-HOME | 313365 |
Runtime : 5 seconds
(~15 million rows)
Real-time querying of large datasets

78. IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity)
as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON
books.id =
transactions.book_id
GROUP BY category;

79. IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity)
as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON
books.id =
transactions.book_id
GROUP BY category;
+---------------------------+----------------+
| category | total_quantity |
+---------------------------+----------------+
| STUDY-AIDS | 10171749 |
| SELF-HELP | 10238182 |
| TECHNOLOGY-ENGINEERING | 10207003 |
| SOCIAL-SCIENCE | 10177075 |
| HISTORY | 10238243 |
| LAW | 10229825 |
| PERFORMING-ARTS | 10231744 |
Runtime : 19 seconds

80. WHEN WOULD YOU USE IMPALA?
53
Adhoc/exploratory data
analysis
Existing Business
Intelligence tools via
ODBC/JDBC

81. DIVERSIFY CONSUMERS OF YOUR DATA
54
?
?
?
?
Our Data
Curious
Person
Financial
Analyst
Customer
Experience
Guru
Dashboard
Whizz

82. DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI
Tools
?
?
?
?
Our Data
Curious
Person
Financial
Analyst
Customer
Experience
Guru
Dashboard
Whizz

83. DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI
Tools
Data
Scientists
Graph
Fans
Map
Reducers
?
?
?
?
Our Data
Curious
Person
Financial
Analyst
Customer
Experience
Guru
Dashboard
Whizz

84. MADE POSSIBLE BY V2.0 DESIGN CHOICES
55
Diagram taken from: http://hortonworks.com/blog/apache-hadoop-2-is-ga/

85. FURTHER RESOURCES
56
http://impala.io/
Tutorials from Cloudera: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/
topics/impala_tutorial.html
Impala and Amazon EMR: http://docs.aws.amazon.com/ElasticMapReduce/latest/
DeveloperGuide/emr-impala.html
Hadoop: http://hadoop.apache.org/
Yarn + hadoop v2: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/
YARN.html

86. 57
PLATFORMS
ADOPT
TRIAL
25. Apache Spark NEW
26. Cloudera Impala NEW
27. DigitalOcean
28. TOTP Two-Factor Authentication
HOLD
45. Application Servers NEW
46. OSGi
47. SPDY NEW
ASSESS
29. Apache Kylin NEW
30. Apache Mesos
31. CoreCLR and CoreFX NEW
32. CoreOS
33. Deis NEW
34. H2O NEW
35. Jackrabbit Oak
36. Linux security modules
37. MariaDB
38. Netflix OSS Full stack
39. OpenAM
40. SDN
41. Spark.io
42. Text it as a service / Rapidpro.io
43. Time-series Databases NEW
44. U2F

87. 58
Scott Shaw
@scottwshaw
Jen Smith
@JenniferSmithCo
thoughtworks.com/radar