The document discusses how modern software architectures can help tame big data. It introduces the speakers and provides an overview of WidasConcepts. The agenda includes a discussion of how big data can help businesses, an example of big data applied in the CarbookPlus platform, and new software architectures for big data. Real-time systems and architectures like lambda architecture are presented as ways to process big data at high velocity and volume. The conclusion emphasizes that big data improves business efficiency but requires tailored implementations and new skills.
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Taming Big Data With Modern Software Architecture
1. Software Architecture
Taming Big Data
With
Modern Software Architecture
Big Data User Group
Stuttgart, 26.09.2013
Thomas Widmann
Managing Director, WidasConcepts
Vinoth Kannan
Senior Consultant, WidasConcepts
2. About the Speakers
2
Thomas Widmann
Vinoth Kannan
! Senior Consultant in WidasConcepts
! Working primarily around Big Data ecosystems of Hadoop and Storm
! Experienced in IT industries of Europe, North America and Asia
! Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France
! Contact: vinoth.kannan@widas.de
! Managing Director of WidasConcepts
! Expert in modern Sales- and Service-Solutions
! More than two decades of consulting experience in Web X.0
concepts, continuous delivery and IT-strategies
! Big Data Evangelist
! Contact: thomas.widmann@widas.de
3. Who we are
! Innovative IT-Consulting Company
founded in 1997
! Headquaters in Wimsheim with
offices in Munich and Essen
! Specialized in Software for Modern
Business in the areas of
! Strategic IT Consulting
! Next Business Intelligence
! Software Development
! 35 expert consultants specializing in
! Data Science
! Software Architecture
! Mobile & Web Development
! Business Consulting
WidasConcepts GmbH
3
4. Agenda
4
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software
Architectures for Big Data
Q & A
5. How big data can help business
5
Big challenges of todays business
eCommerce Financial Services Telecommunication
HealthCare Marketing National Security
o Increase in Sales
o Determine pricing
o Avoid Shop cart abandoners
o Manage logistics and inventory
o Fraud detection
o Real Time analytics on
financial data
o Managing mobile payments
o Manage network traffic
overload
o Hardware cost optimisation
based on traffic data
o Real time monitoring of
patients vitals for predictive
intervention and care.
o Cost savings on patient care
o Mood prediction for market
research
o Real time bidding
o Location based marketing
o Monitor real time feed from
CCTV, traffic data, mobile data
o Find patterns/cluster for
people, places, products, things
Building sector, all manufacturers, energy, media and many more
6. Getting the Right Data
to the Right People
at the Right Time
6
Mobile Payment
Money Cards / POS
Banking Portal
Adviser / Agency
Banking
Operations
Event
Database
Business
Activity
Monitoring
CEP
Making Use of Events
Real Time Analysis
Cross-Up-Selling
Fraud Detection
Product Innovation
7. Impact of Big Data on Business
Results on Europe-wide survey by TU-Berlin, Sept 2013
Source: European Smart Data Survey, Sept 2013
7
8. Need for Big Data
4Vs concept of Big Data
Volume
To handle the huge
amount of data genereted
by businesses
Velocity
To store, analyse and
retrive huge dataset in
good speed
Variety
To process data from
multiple sources, majorly
unstructured data
Source: Oracle
Value
To ask the right questions
to generate maximum
value
8
9. What makes data so big?
Combination of structured, unstructured data
created by sensors, actors, humans
9
10. New Software Architecture
Need of the hour for modern architecture
analyse
huge
volume
of
data
fast
consolidate
data
from
mul;ple
sources
Intelligent
predic;ve
10
11. Agenda
11
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software
architectures for Big Data
Q & A
12. Datawarehouse – Classic BI
Online System
Database n
Classic Software Architectures
Main goal: keep business records
<Client>
Browser
Presentation
Business Logic
Database 2
StagingETL ETL
BI
Reporting
Database 1
Batch
Processing
Main goal: Reporting
12
13. Need for real time system
for enterprise
13
The Approach
Impact of analytics, event processing and decision management in building a real
time big data system for enterprise
Acquire Data:
• Business Trx./Events
• Observational Events
from anywhere
Observe
Orient
Decide
Act
Systems:
• Various systems
• Preferrable: MQ
• put in context
• Evaluate data, predict
• determine actions
• Initiate actions
• manage data
Systems:
• complex event processing,
rule management
• realtime analysis with new
big data technologies
• select action, if
necessary
• Perform action
(manually or
automated)
Systems:
• Business software
• Decision support
software
• Reporting
(John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich
14. Phases of realising
real time system
Step by step approach for the architecture
• Create structured data from raw data
• Extract, merge, filter etcData Extraction
• Aggregate and select relevant data
• Speed and flexibilityModel development
• Testing the model
• Comparing the result
Evaluation &
deployment
• Triggered by action in the decision layer
• Score to validate outputReal-time Scoring
• Need to refresh the ever changing data
• Real time data viewModel Refresh
14
15. Big Data Architecture
Service Layer CQRS
Distributed DB
Distributed Messaging System
Recommendations
Realtime-Processing Batch-Processing
Lambda architecture
Business Transactional Data
Business Analytics Data
Recommendation Data
Real Time Event Alert Data
Big Data Eco System
Distributed Search
Engine
Business
Data Formats
Contextual Data
Business Transactions
Reporting
Notifications
Notification SystemData Retrieval
In-Memory Technology
15
Distributed DBNoSQL DB
16. Basic idea behind
Lambda Architecture
query = function(all data)
Nathan Marz
Big Data - principles and best practices
of scalable realtime data systems
16
17. Basic Idea behind Lambda
𝑓( 𝑎↓0 … 𝑎↓𝑚 )
Perform some function from real-time data “0“ to the history data “n“
Real Time Big Data
𝑓( 𝑎↓0 .. 𝑎↓𝑛 … 𝑎↓𝑚 ) = 𝑓( 𝑎↓0 .. 𝑎↓𝑛 ) + 𝑓( 𝑎↓𝑛+1 … 𝑎↓𝑚 )
Lambda Architecture
Hadoop ProcessStorm
Process
Real Time Big
Data
}
}
}
Letting the History data processed by Hadoop makes process faster
17
18. Agenda
18
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software architectures
for Big Data
Q & A
20. Why
Carbookplus?
Benefits
of
being
a
Carbookplus
member
Track trips
Maintain Logbook
Get Notified about best gas stations & snipers
Manage and compare vehicle cost
Fleet management
Social platform connecting drivers
21. Big
Data
in
Carbookplus
Where
do
we
use
it
Trip
Transac;on
Data
Recommend
possible
trips
Create
trips
Update
logbook
entry
No;fy
user
for
discrepancy
in
the
logbook
entry
No;fy
user
for
best
gas
sta;on
Update
vehicle
mileage
Generate
analy;c
reports
Real Time !!!
22. Big
Data
in
Carbookplus
Event
Driven
Model
Use Case : Track trips on Real Time
Start
address
Intermediate
address
1
End
address
Event 1
Event 2
Event n Intermediate
address
n
…..
Event n+1
Object Model
Event …
23. Big
Data
in
Carbookplus
How
do
we
use
it
Trip
Transac;on
Data
Real Time Storm Cluster
In
Memory
Data
Processing
MongoDB
Trip
Data
Hadoop
Cluster
Map
&
Reduce
Analy;cal
Data
Decision
Management
Recommender
Engine
Rule
Engine
KaYa
ETL
Output
Mobile
Web
BI
User
Data
ETL
Vehicle
Data
ETL
Historical Data
Real Time Data
24. Conclusion
Ø Big Data clearly improves business efficiency and answers lot of the
questions which were previously unanswerable in this data driven world.
Ø Need to adapt our IT architecture to remain competitive and see the big
picture
Ø Flipside : No standardised implementation architecture. Need to be
tailor-made for individual business needs.
Ø Road Ahead : Need for phased integration into Big Data ecosystem
Ø We are hiring !!
§ Data Scientists
§ Big Data Engineers
§ Interns
Visit us : www.widas.de
Email : contact@widas.de
25. Agenda
25
How BigData can help
business
BigData applied in
CarbookPlus Platform
New Software architectures
for
Big Data
Q & A