SlideShare a Scribd company logo
1 of 26
Download to read offline
Software Architecture
Taming Big Data
With
Modern Software Architecture
Big Data User Group
Stuttgart, 26.09.2013
Thomas Widmann
Managing Director, WidasConcepts
Vinoth Kannan
Senior Consultant, WidasConcepts
About the Speakers
2	
  
Thomas Widmann
Vinoth Kannan
!   Senior Consultant in WidasConcepts
!   Working primarily around Big Data ecosystems of Hadoop and Storm
!   Experienced in IT industries of Europe, North America and Asia
!   Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France
! Contact: vinoth.kannan@widas.de
!   Managing Director of WidasConcepts
!   Expert in modern Sales- and Service-Solutions
!   More than two decades of consulting experience in Web X.0
concepts, continuous delivery and IT-strategies
!   Big Data Evangelist
! Contact: thomas.widmann@widas.de
Who we are
!   Innovative IT-Consulting Company
founded in 1997
! Headquaters in Wimsheim with
offices in Munich and Essen
! Specialized in Software for Modern
Business in the areas of
!   Strategic IT Consulting
!   Next Business Intelligence
!   Software Development
!   35 expert consultants specializing in
!   Data Science
!   Software Architecture
!   Mobile & Web Development
!   Business Consulting
WidasConcepts GmbH
3	
  
Agenda
4	
  
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software
Architectures for Big Data
Q & A
How big data can help business
5
Big challenges of todays business
eCommerce Financial Services Telecommunication
HealthCare Marketing National Security
o  Increase in Sales
o  Determine pricing
o  Avoid Shop cart abandoners
o  Manage logistics and inventory
o  Fraud detection
o  Real Time analytics on
financial data
o  Managing mobile payments
o  Manage network traffic
overload
o  Hardware cost optimisation
based on traffic data
o  Real time monitoring of
patients vitals for predictive
intervention and care.
o  Cost savings on patient care
o  Mood prediction for market
research
o  Real time bidding
o  Location based marketing
o  Monitor real time feed from
CCTV, traffic data, mobile data
o  Find patterns/cluster for
people, places, products, things
Building sector, all manufacturers, energy, media and many more
Getting the Right Data
to the Right People
at the Right Time
6	
  
Mobile Payment
Money Cards / POS
Banking Portal
Adviser / Agency
Banking
Operations
Event
Database
Business
Activity
Monitoring
CEP
Making Use of Events
Real Time Analysis
Cross-Up-Selling
Fraud Detection
Product Innovation
Impact of Big Data on Business
Results on Europe-wide survey by TU-Berlin, Sept 2013
Source: European Smart Data Survey, Sept 2013
7	
  
Need for Big Data
4Vs concept of Big Data
Volume
To handle the huge
amount of data genereted
by businesses
Velocity
To store, analyse and
retrive huge dataset in
good speed
Variety
To process data from
multiple sources, majorly
unstructured data
Source: Oracle
Value
To ask the right questions
to generate maximum
value
8	
  
What makes data so big?
Combination of structured, unstructured data
created by sensors, actors, humans
9	
  
New Software Architecture
Need of the hour for modern architecture
analyse	
  
huge	
  
volume	
  of	
  
data	
  
fast	
  
consolidate	
  
data	
  from	
  
mul;ple	
  
sources	
  
Intelligent	
  
predic;ve	
  
10	
  
Agenda
11	
  
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software
architectures for Big Data
Q & A
Datawarehouse – Classic BI
Online System
Database n
Classic Software Architectures
Main goal: keep business records
<Client>
Browser
Presentation
Business Logic
Database 2
StagingETL ETL
BI
Reporting
Database 1
Batch
Processing
Main goal: Reporting
12	
  
Need for real time system
for enterprise
13	
  
The Approach
Impact of analytics, event processing and decision management in building a real
time big data system for enterprise
Acquire Data:
•  Business Trx./Events
•  Observational Events
from anywhere
Observe	
   Orient	
   Decide	
   Act	
  
Systems:
•  Various systems
•  Preferrable: MQ
•  put in context
•  Evaluate data, predict
•  determine actions
•  Initiate actions
•  manage data
Systems:
•  complex event processing,
rule management
•  realtime analysis with new
big data technologies
•  select action, if
necessary
•  Perform action
(manually or
automated)
Systems:
•  Business software
•  Decision support
software
•  Reporting
(John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich
Phases of realising
real time system
Step by step approach for the architecture
• Create structured data from raw data
• Extract, merge, filter etcData Extraction
• Aggregate and select relevant data
• Speed and flexibilityModel development
• Testing the model
• Comparing the result
Evaluation &
deployment
• Triggered by action in the decision layer
• Score to validate outputReal-time Scoring
• Need to refresh the ever changing data
• Real time data viewModel Refresh
14	
  
Big Data Architecture
Service Layer CQRS
Distributed DB
Distributed Messaging System
Recommendations
Realtime-Processing Batch-Processing
Lambda architecture
Business Transactional Data
Business Analytics Data
Recommendation Data
Real Time Event Alert Data
Big Data Eco System
Distributed Search
Engine
Business
Data Formats
Contextual Data
Business Transactions
Reporting
Notifications
Notification SystemData Retrieval
In-Memory Technology
15	
  
Distributed DBNoSQL DB
Basic idea behind
Lambda Architecture
query = function(all data)
Nathan Marz
Big Data - principles and best practices
of scalable realtime data systems
16	
  
Basic Idea behind Lambda
𝑓(​ 𝑎↓0 …​ 𝑎↓𝑚 )
Perform some function from real-time data “0“ to the history data “n“
Real Time Big Data
𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 …​ 𝑎↓𝑚 ) = 𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 ) + 𝑓(​ 𝑎↓𝑛+1 …​ 𝑎↓𝑚 )
Lambda Architecture
Hadoop ProcessStorm
Process
Real Time Big
Data
}
}
}
Letting the History data processed by Hadoop makes process faster
17	
  
Agenda
18	
  
How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software architectures
for Big Data
Q & A
What	
  is	
  Carbookplus? 	
  	
  
Change	
  the	
  way	
  you	
  drive.	
  
19	
  
Why	
  Carbookplus? 	
  	
  
Benefits	
  of	
  being	
  a	
  Carbookplus	
  member	
  
Track trips
Maintain Logbook
Get Notified about best gas stations & snipers
Manage and compare vehicle cost
Fleet management
Social platform connecting drivers
Big	
  Data	
  in	
  Carbookplus	
  
Where	
  do	
  we	
  use	
  it	
  
Trip	
  Transac;on	
  Data	
  
Recommend	
  possible	
  trips	
  
	
  
Create	
  trips	
  
	
  
Update	
  logbook	
  entry	
  
	
  
No;fy	
  user	
  for	
  discrepancy	
  in	
  the	
  
logbook	
  entry	
  
	
  
No;fy	
  user	
  for	
  best	
  gas	
  sta;on	
  
	
  
Update	
  vehicle	
  mileage	
  	
  
	
  
Generate	
  analy;c	
  reports	
  
Real Time !!!
Big	
  Data	
  in	
  Carbookplus	
  
Event	
  Driven	
  Model	
  
Use Case : Track trips on Real Time
Start	
  address	
  
Intermediate	
  address	
  1	
  
End	
  address	
  
Event 1
Event 2
Event n Intermediate	
  address	
  n	
  
…..	
  
Event n+1
Object Model
Event …
Big	
  Data	
  in	
  Carbookplus	
  
How	
  do	
  we	
  use	
  it	
  
Trip	
  Transac;on	
  
Data	
  
Real Time Storm Cluster
In	
  Memory	
  Data	
  Processing	
  
MongoDB	
  
Trip	
  Data	
   Hadoop	
  Cluster	
  
Map	
  &	
  Reduce	
  
Analy;cal	
  Data	
  
Decision	
  Management	
  
	
  
	
  
	
  
	
  
Recommender	
  
Engine	
  
Rule	
  
Engine	
  
KaYa	
  
ETL	
  
Output	
   Mobile	
   Web	
   BI	
  
User	
  Data	
   ETL	
  
Vehicle	
  Data	
   ETL	
  
Historical Data
Real Time Data
Conclusion	
  
Ø  Big Data clearly improves business efficiency and answers lot of the
questions which were previously unanswerable in this data driven world.
Ø  Need to adapt our IT architecture to remain competitive and see the big
picture
Ø  Flipside : No standardised implementation architecture. Need to be
tailor-made for individual business needs.
Ø  Road Ahead : Need for phased integration into Big Data ecosystem
Ø  We are hiring !!
§  Data Scientists
§  Big Data Engineers
§  Interns
Visit us : www.widas.de
Email : contact@widas.de
Agenda
25	
  
How BigData can help
business
BigData applied in
CarbookPlus Platform
New Software architectures
for
Big Data
Q & A
Taming Big Data  With Modern Software Architecture

More Related Content

What's hot

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...Data Con LA
 
Data Virtualization: From Zero to Hero
Data Virtualization: From Zero to HeroData Virtualization: From Zero to Hero
Data Virtualization: From Zero to HeroDenodo
 
The Emerging Data Lake IT Strategy
The Emerging Data Lake IT StrategyThe Emerging Data Lake IT Strategy
The Emerging Data Lake IT StrategyThomas Kelly, PMP
 
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)Denodo
 
Data architecture for modern enterprise
Data architecture for modern enterpriseData architecture for modern enterprise
Data architecture for modern enterprisekayalvizhi kandasamy
 
Enterprise Data Lake - Scalable Digital
Enterprise Data Lake - Scalable DigitalEnterprise Data Lake - Scalable Digital
Enterprise Data Lake - Scalable Digitalsambiswal
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra
 
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...Capgemini
 
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...Denodo
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonJeffrey T. Pollock
 
Big Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyBig Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyNati Shalom
 
The Role of the Logical Data Fabric in a Unified Platform for Modern Analytics
The Role of the Logical Data Fabric in a Unified Platform for Modern AnalyticsThe Role of the Logical Data Fabric in a Unified Platform for Modern Analytics
The Role of the Logical Data Fabric in a Unified Platform for Modern AnalyticsDenodo
 
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationAccelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationDenodo
 
Data lake benefits
Data lake benefitsData lake benefits
Data lake benefitsRicky Barron
 
Top Big data Analytics tools: Emerging trends and Best practices
Top Big data Analytics tools: Emerging trends and Best practicesTop Big data Analytics tools: Emerging trends and Best practices
Top Big data Analytics tools: Emerging trends and Best practicesSpringPeople
 
Splunk Business Analytics
Splunk Business AnalyticsSplunk Business Analytics
Splunk Business AnalyticsCleverDATA
 
Enabling Cloud Data Integration (EMEA)
Enabling Cloud Data Integration (EMEA)Enabling Cloud Data Integration (EMEA)
Enabling Cloud Data Integration (EMEA)Denodo
 
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...Seeling Cheung
 
Logical Data Warehouse and Data Lakes
Logical Data Warehouse and Data Lakes Logical Data Warehouse and Data Lakes
Logical Data Warehouse and Data Lakes Denodo
 

What's hot (20)

Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
Big Data Day LA 2015 - Data Lake - Re Birth of Enterprise Data Thinking by Ra...
 
Data Virtualization: From Zero to Hero
Data Virtualization: From Zero to HeroData Virtualization: From Zero to Hero
Data Virtualization: From Zero to Hero
 
The Emerging Data Lake IT Strategy
The Emerging Data Lake IT StrategyThe Emerging Data Lake IT Strategy
The Emerging Data Lake IT Strategy
 
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)
Empowering your Enterprise with a Self-Service Data Marketplace (ASEAN)
 
Data architecture for modern enterprise
Data architecture for modern enterpriseData architecture for modern enterprise
Data architecture for modern enterprise
 
Enterprise Data Lake - Scalable Digital
Enterprise Data Lake - Scalable DigitalEnterprise Data Lake - Scalable Digital
Enterprise Data Lake - Scalable Digital
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...
EMC World 2014 Breakout: Move to the Business Data Lake – Not as Hard as It S...
 
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
Analyst View of Data Virtualization: Conversations with Boulder Business Inte...
 
Flash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lonFlash session -streaming--ses1243-lon
Flash session -streaming--ses1243-lon
 
Big Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case StudyBig Data Real Time Analytics - A Facebook Case Study
Big Data Real Time Analytics - A Facebook Case Study
 
The Role of the Logical Data Fabric in a Unified Platform for Modern Analytics
The Role of the Logical Data Fabric in a Unified Platform for Modern AnalyticsThe Role of the Logical Data Fabric in a Unified Platform for Modern Analytics
The Role of the Logical Data Fabric in a Unified Platform for Modern Analytics
 
Data Lake,beyond the Data Warehouse
Data Lake,beyond the Data WarehouseData Lake,beyond the Data Warehouse
Data Lake,beyond the Data Warehouse
 
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationAccelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and Visualization
 
Data lake benefits
Data lake benefitsData lake benefits
Data lake benefits
 
Top Big data Analytics tools: Emerging trends and Best practices
Top Big data Analytics tools: Emerging trends and Best practicesTop Big data Analytics tools: Emerging trends and Best practices
Top Big data Analytics tools: Emerging trends and Best practices
 
Splunk Business Analytics
Splunk Business AnalyticsSplunk Business Analytics
Splunk Business Analytics
 
Enabling Cloud Data Integration (EMEA)
Enabling Cloud Data Integration (EMEA)Enabling Cloud Data Integration (EMEA)
Enabling Cloud Data Integration (EMEA)
 
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...
Citizens Bank: Data Lake Implementation – Selecting BigInsights ViON Spark/Ha...
 
Logical Data Warehouse and Data Lakes
Logical Data Warehouse and Data Lakes Logical Data Warehouse and Data Lakes
Logical Data Warehouse and Data Lakes
 

Viewers also liked

Applying Agile Values to Enterprise Architecture Software Architectural Trend...
Applying Agile Values to Enterprise Architecture Software Architectural Trend...Applying Agile Values to Enterprise Architecture Software Architectural Trend...
Applying Agile Values to Enterprise Architecture Software Architectural Trend...Software Park Thailand
 
Enterprise Architecture
Enterprise ArchitectureEnterprise Architecture
Enterprise ArchitectureVikas Grover
 
3 top tools for taming big data
3 top tools for taming big data3 top tools for taming big data
3 top tools for taming big dataHPC Asia
 
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...Signal Chicago 2012
 
Taming Big Data in the Reverse Logistics Supply Chain
Taming Big Data in the Reverse Logistics Supply ChainTaming Big Data in the Reverse Logistics Supply Chain
Taming Big Data in the Reverse Logistics Supply ChainJonathan Pine
 
Taming the Big Data Beast - Together
Taming the Big Data Beast - TogetherTaming the Big Data Beast - Together
Taming the Big Data Beast - TogetherKennisalliantie
 
Creating Enterprise Value from Business Architecture
Creating Enterprise Value from Business ArchitectureCreating Enterprise Value from Business Architecture
Creating Enterprise Value from Business Architectureiasaglobal
 
Taming Big Data!
Taming Big Data!Taming Big Data!
Taming Big Data!Ian Foster
 
Taming Social Data: How Social Data Framing liberates analysis and accelerate...
Taming Social Data: How Social Data Framing liberates analysis and accelerate...Taming Social Data: How Social Data Framing liberates analysis and accelerate...
Taming Social Data: How Social Data Framing liberates analysis and accelerate...DataSift
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to ElasticsearchRuslan Zavacky
 
Spark: Taming Big Data
Spark: Taming Big DataSpark: Taming Big Data
Spark: Taming Big DataLeonardo Gamas
 
Introduction to Enterprise Architecture
Introduction to Enterprise ArchitectureIntroduction to Enterprise Architecture
Introduction to Enterprise ArchitectureMohammed Omar
 
Introduction to Enterprise Architecture and TOGAF 9.1
Introduction to Enterprise Architecture and TOGAF 9.1Introduction to Enterprise Architecture and TOGAF 9.1
Introduction to Enterprise Architecture and TOGAF 9.1iasaglobal
 
Implementing Effective Enterprise Architecture
Implementing Effective Enterprise ArchitectureImplementing Effective Enterprise Architecture
Implementing Effective Enterprise ArchitectureLeo Shuster
 
Tools and techniques adopted for big data analytics
Tools and techniques adopted for big data analyticsTools and techniques adopted for big data analytics
Tools and techniques adopted for big data analyticsJOSEPH FRANCIS
 
Taming the Big Data Beast
Taming the Big Data BeastTaming the Big Data Beast
Taming the Big Data BeastBaynote
 

Viewers also liked (20)

Applying Agile Values to Enterprise Architecture Software Architectural Trend...
Applying Agile Values to Enterprise Architecture Software Architectural Trend...Applying Agile Values to Enterprise Architecture Software Architectural Trend...
Applying Agile Values to Enterprise Architecture Software Architectural Trend...
 
In-Store Analysis with Hadoop
In-Store Analysis with HadoopIn-Store Analysis with Hadoop
In-Store Analysis with Hadoop
 
Apache Drill
Apache DrillApache Drill
Apache Drill
 
Enterprise Architecture
Enterprise ArchitectureEnterprise Architecture
Enterprise Architecture
 
Turning Information chaos into reliable data
Turning Information chaos into reliable dataTurning Information chaos into reliable data
Turning Information chaos into reliable data
 
3 top tools for taming big data
3 top tools for taming big data3 top tools for taming big data
3 top tools for taming big data
 
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...
Taming the Data Deluge: How Advertisers and Publishers Can Win at the Big Dat...
 
Taming Big Data in the Reverse Logistics Supply Chain
Taming Big Data in the Reverse Logistics Supply ChainTaming Big Data in the Reverse Logistics Supply Chain
Taming Big Data in the Reverse Logistics Supply Chain
 
Taming the Big Data Beast - Together
Taming the Big Data Beast - TogetherTaming the Big Data Beast - Together
Taming the Big Data Beast - Together
 
Creating Enterprise Value from Business Architecture
Creating Enterprise Value from Business ArchitectureCreating Enterprise Value from Business Architecture
Creating Enterprise Value from Business Architecture
 
Taming Big Data!
Taming Big Data!Taming Big Data!
Taming Big Data!
 
Hadoop-2.6.0 Slides
Hadoop-2.6.0 SlidesHadoop-2.6.0 Slides
Hadoop-2.6.0 Slides
 
Taming Social Data: How Social Data Framing liberates analysis and accelerate...
Taming Social Data: How Social Data Framing liberates analysis and accelerate...Taming Social Data: How Social Data Framing liberates analysis and accelerate...
Taming Social Data: How Social Data Framing liberates analysis and accelerate...
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to Elasticsearch
 
Spark: Taming Big Data
Spark: Taming Big DataSpark: Taming Big Data
Spark: Taming Big Data
 
Introduction to Enterprise Architecture
Introduction to Enterprise ArchitectureIntroduction to Enterprise Architecture
Introduction to Enterprise Architecture
 
Introduction to Enterprise Architecture and TOGAF 9.1
Introduction to Enterprise Architecture and TOGAF 9.1Introduction to Enterprise Architecture and TOGAF 9.1
Introduction to Enterprise Architecture and TOGAF 9.1
 
Implementing Effective Enterprise Architecture
Implementing Effective Enterprise ArchitectureImplementing Effective Enterprise Architecture
Implementing Effective Enterprise Architecture
 
Tools and techniques adopted for big data analytics
Tools and techniques adopted for big data analyticsTools and techniques adopted for big data analytics
Tools and techniques adopted for big data analytics
 
Taming the Big Data Beast
Taming the Big Data BeastTaming the Big Data Beast
Taming the Big Data Beast
 

Similar to Taming Big Data With Modern Software Architecture

Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...
Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...
Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...Matt Stubbs
 
Capturing big value in big data
Capturing big value in big data Capturing big value in big data
Capturing big value in big data BSP Media Group
 
The Double win business transformation and in-year ROI and TCO reduction
The Double win business transformation and in-year ROI and TCO reductionThe Double win business transformation and in-year ROI and TCO reduction
The Double win business transformation and in-year ROI and TCO reductionMongoDB
 
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationAccelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationDenodo
 
Big Data & Analytics, Peter Jönsson
Big Data & Analytics, Peter JönssonBig Data & Analytics, Peter Jönsson
Big Data & Analytics, Peter JönssonIBM Danmark
 
Why Everything You Know About bigdata Is A Lie
Why Everything You Know About bigdata Is A LieWhy Everything You Know About bigdata Is A Lie
Why Everything You Know About bigdata Is A LieSunil Ranka
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017SingleStore
 
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"MDS ap
 
Building the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseBuilding the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseDatabricks
 
Data Engineering Proposal for Homerunner.pptx
Data Engineering Proposal for Homerunner.pptxData Engineering Proposal for Homerunner.pptx
Data Engineering Proposal for Homerunner.pptxDamilolaLana1
 
It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...
 It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201... It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...
It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...Edgar Alejandro Villegas
 
Self-service analytics @ Leaseplan Digital: from business intelligence to int...
Self-service analytics @ Leaseplan Digital: from business intelligence to int...Self-service analytics @ Leaseplan Digital: from business intelligence to int...
Self-service analytics @ Leaseplan Digital: from business intelligence to int...webwinkelvakdag
 
Building the Cognitive Era : Big Data Strategies
Building the Cognitive Era : Big Data StrategiesBuilding the Cognitive Era : Big Data Strategies
Building the Cognitive Era : Big Data StrategiesKevin Sigliano
 
Top Business Intelligence Trends for 2016 by Panorama Software
Top Business Intelligence Trends for 2016 by Panorama SoftwareTop Business Intelligence Trends for 2016 by Panorama Software
Top Business Intelligence Trends for 2016 by Panorama SoftwarePanorama Software
 
Big Data, Big Thinking: Untapped Opportunities
Big Data, Big Thinking: Untapped OpportunitiesBig Data, Big Thinking: Untapped Opportunities
Big Data, Big Thinking: Untapped OpportunitiesSAP Technology
 
Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Amar Roy
 
Bitrock manufacturing
Bitrock manufacturing Bitrock manufacturing
Bitrock manufacturing cosma_r
 
Kudu Forrester Webinar
Kudu Forrester WebinarKudu Forrester Webinar
Kudu Forrester WebinarCloudera, Inc.
 

Similar to Taming Big Data With Modern Software Architecture (20)

Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...
Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...
Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...
 
Capturing big value in big data
Capturing big value in big data Capturing big value in big data
Capturing big value in big data
 
The Double win business transformation and in-year ROI and TCO reduction
The Double win business transformation and in-year ROI and TCO reductionThe Double win business transformation and in-year ROI and TCO reduction
The Double win business transformation and in-year ROI and TCO reduction
 
Accelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and VisualizationAccelerate Self-Service Analytics with Data Virtualization and Visualization
Accelerate Self-Service Analytics with Data Virtualization and Visualization
 
Big Data & Analytics, Peter Jönsson
Big Data & Analytics, Peter JönssonBig Data & Analytics, Peter Jönsson
Big Data & Analytics, Peter Jönsson
 
Why Everything You Know About bigdata Is A Lie
Why Everything You Know About bigdata Is A LieWhy Everything You Know About bigdata Is A Lie
Why Everything You Know About bigdata Is A Lie
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
 
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"
 
Building the Artificially Intelligent Enterprise
Building the Artificially Intelligent EnterpriseBuilding the Artificially Intelligent Enterprise
Building the Artificially Intelligent Enterprise
 
Data Engineering Proposal for Homerunner.pptx
Data Engineering Proposal for Homerunner.pptxData Engineering Proposal for Homerunner.pptx
Data Engineering Proposal for Homerunner.pptx
 
It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...
 It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201... It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...
It’s Not About Big Data – It’s About Big Insights - SAP Webinar - 20 Aug 201...
 
Identifying the new frontier of big data as an enabler for T&T industries: Re...
Identifying the new frontier of big data as an enabler for T&T industries: Re...Identifying the new frontier of big data as an enabler for T&T industries: Re...
Identifying the new frontier of big data as an enabler for T&T industries: Re...
 
Self-service analytics @ Leaseplan Digital: from business intelligence to int...
Self-service analytics @ Leaseplan Digital: from business intelligence to int...Self-service analytics @ Leaseplan Digital: from business intelligence to int...
Self-service analytics @ Leaseplan Digital: from business intelligence to int...
 
Building the Cognitive Era : Big Data Strategies
Building the Cognitive Era : Big Data StrategiesBuilding the Cognitive Era : Big Data Strategies
Building the Cognitive Era : Big Data Strategies
 
Big data Introduction by Mohan
Big data Introduction by MohanBig data Introduction by Mohan
Big data Introduction by Mohan
 
Top Business Intelligence Trends for 2016 by Panorama Software
Top Business Intelligence Trends for 2016 by Panorama SoftwareTop Business Intelligence Trends for 2016 by Panorama Software
Top Business Intelligence Trends for 2016 by Panorama Software
 
Big Data, Big Thinking: Untapped Opportunities
Big Data, Big Thinking: Untapped OpportunitiesBig Data, Big Thinking: Untapped Opportunities
Big Data, Big Thinking: Untapped Opportunities
 
Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0
 
Bitrock manufacturing
Bitrock manufacturing Bitrock manufacturing
Bitrock manufacturing
 
Kudu Forrester Webinar
Kudu Forrester WebinarKudu Forrester Webinar
Kudu Forrester Webinar
 

Recently uploaded

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Recently uploaded (20)

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Taming Big Data With Modern Software Architecture

  • 1. Software Architecture Taming Big Data With Modern Software Architecture Big Data User Group Stuttgart, 26.09.2013 Thomas Widmann Managing Director, WidasConcepts Vinoth Kannan Senior Consultant, WidasConcepts
  • 2. About the Speakers 2   Thomas Widmann Vinoth Kannan !   Senior Consultant in WidasConcepts !   Working primarily around Big Data ecosystems of Hadoop and Storm !   Experienced in IT industries of Europe, North America and Asia !   Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France ! Contact: vinoth.kannan@widas.de !   Managing Director of WidasConcepts !   Expert in modern Sales- and Service-Solutions !   More than two decades of consulting experience in Web X.0 concepts, continuous delivery and IT-strategies !   Big Data Evangelist ! Contact: thomas.widmann@widas.de
  • 3. Who we are !   Innovative IT-Consulting Company founded in 1997 ! Headquaters in Wimsheim with offices in Munich and Essen ! Specialized in Software for Modern Business in the areas of !   Strategic IT Consulting !   Next Business Intelligence !   Software Development !   35 expert consultants specializing in !   Data Science !   Software Architecture !   Mobile & Web Development !   Business Consulting WidasConcepts GmbH 3  
  • 4. Agenda 4   How Big Data can help business Big Data applied in CarbookPlus Platform New Software Architectures for Big Data Q & A
  • 5. How big data can help business 5 Big challenges of todays business eCommerce Financial Services Telecommunication HealthCare Marketing National Security o  Increase in Sales o  Determine pricing o  Avoid Shop cart abandoners o  Manage logistics and inventory o  Fraud detection o  Real Time analytics on financial data o  Managing mobile payments o  Manage network traffic overload o  Hardware cost optimisation based on traffic data o  Real time monitoring of patients vitals for predictive intervention and care. o  Cost savings on patient care o  Mood prediction for market research o  Real time bidding o  Location based marketing o  Monitor real time feed from CCTV, traffic data, mobile data o  Find patterns/cluster for people, places, products, things Building sector, all manufacturers, energy, media and many more
  • 6. Getting the Right Data to the Right People at the Right Time 6   Mobile Payment Money Cards / POS Banking Portal Adviser / Agency Banking Operations Event Database Business Activity Monitoring CEP Making Use of Events Real Time Analysis Cross-Up-Selling Fraud Detection Product Innovation
  • 7. Impact of Big Data on Business Results on Europe-wide survey by TU-Berlin, Sept 2013 Source: European Smart Data Survey, Sept 2013 7  
  • 8. Need for Big Data 4Vs concept of Big Data Volume To handle the huge amount of data genereted by businesses Velocity To store, analyse and retrive huge dataset in good speed Variety To process data from multiple sources, majorly unstructured data Source: Oracle Value To ask the right questions to generate maximum value 8  
  • 9. What makes data so big? Combination of structured, unstructured data created by sensors, actors, humans 9  
  • 10. New Software Architecture Need of the hour for modern architecture analyse   huge   volume  of   data   fast   consolidate   data  from   mul;ple   sources   Intelligent   predic;ve   10  
  • 11. Agenda 11   How Big Data can help business Big Data applied in CarbookPlus Platform New Software architectures for Big Data Q & A
  • 12. Datawarehouse – Classic BI Online System Database n Classic Software Architectures Main goal: keep business records <Client> Browser Presentation Business Logic Database 2 StagingETL ETL BI Reporting Database 1 Batch Processing Main goal: Reporting 12  
  • 13. Need for real time system for enterprise 13   The Approach Impact of analytics, event processing and decision management in building a real time big data system for enterprise Acquire Data: •  Business Trx./Events •  Observational Events from anywhere Observe   Orient   Decide   Act   Systems: •  Various systems •  Preferrable: MQ •  put in context •  Evaluate data, predict •  determine actions •  Initiate actions •  manage data Systems: •  complex event processing, rule management •  realtime analysis with new big data technologies •  select action, if necessary •  Perform action (manually or automated) Systems: •  Business software •  Decision support software •  Reporting (John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich
  • 14. Phases of realising real time system Step by step approach for the architecture • Create structured data from raw data • Extract, merge, filter etcData Extraction • Aggregate and select relevant data • Speed and flexibilityModel development • Testing the model • Comparing the result Evaluation & deployment • Triggered by action in the decision layer • Score to validate outputReal-time Scoring • Need to refresh the ever changing data • Real time data viewModel Refresh 14  
  • 15. Big Data Architecture Service Layer CQRS Distributed DB Distributed Messaging System Recommendations Realtime-Processing Batch-Processing Lambda architecture Business Transactional Data Business Analytics Data Recommendation Data Real Time Event Alert Data Big Data Eco System Distributed Search Engine Business Data Formats Contextual Data Business Transactions Reporting Notifications Notification SystemData Retrieval In-Memory Technology 15   Distributed DBNoSQL DB
  • 16. Basic idea behind Lambda Architecture query = function(all data) Nathan Marz Big Data - principles and best practices of scalable realtime data systems 16  
  • 17. Basic Idea behind Lambda 𝑓(​ 𝑎↓0 …​ 𝑎↓𝑚 ) Perform some function from real-time data “0“ to the history data “n“ Real Time Big Data 𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 …​ 𝑎↓𝑚 ) = 𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 ) + 𝑓(​ 𝑎↓𝑛+1 …​ 𝑎↓𝑚 ) Lambda Architecture Hadoop ProcessStorm Process Real Time Big Data } } } Letting the History data processed by Hadoop makes process faster 17  
  • 18. Agenda 18   How Big Data can help business Big Data applied in CarbookPlus Platform New Software architectures for Big Data Q & A
  • 19. What  is  Carbookplus?     Change  the  way  you  drive.   19  
  • 20. Why  Carbookplus?     Benefits  of  being  a  Carbookplus  member   Track trips Maintain Logbook Get Notified about best gas stations & snipers Manage and compare vehicle cost Fleet management Social platform connecting drivers
  • 21. Big  Data  in  Carbookplus   Where  do  we  use  it   Trip  Transac;on  Data   Recommend  possible  trips     Create  trips     Update  logbook  entry     No;fy  user  for  discrepancy  in  the   logbook  entry     No;fy  user  for  best  gas  sta;on     Update  vehicle  mileage       Generate  analy;c  reports   Real Time !!!
  • 22. Big  Data  in  Carbookplus   Event  Driven  Model   Use Case : Track trips on Real Time Start  address   Intermediate  address  1   End  address   Event 1 Event 2 Event n Intermediate  address  n   …..   Event n+1 Object Model Event …
  • 23. Big  Data  in  Carbookplus   How  do  we  use  it   Trip  Transac;on   Data   Real Time Storm Cluster In  Memory  Data  Processing   MongoDB   Trip  Data   Hadoop  Cluster   Map  &  Reduce   Analy;cal  Data   Decision  Management           Recommender   Engine   Rule   Engine   KaYa   ETL   Output   Mobile   Web   BI   User  Data   ETL   Vehicle  Data   ETL   Historical Data Real Time Data
  • 24. Conclusion   Ø  Big Data clearly improves business efficiency and answers lot of the questions which were previously unanswerable in this data driven world. Ø  Need to adapt our IT architecture to remain competitive and see the big picture Ø  Flipside : No standardised implementation architecture. Need to be tailor-made for individual business needs. Ø  Road Ahead : Need for phased integration into Big Data ecosystem Ø  We are hiring !! §  Data Scientists §  Big Data Engineers §  Interns Visit us : www.widas.de Email : contact@widas.de
  • 25. Agenda 25   How BigData can help business BigData applied in CarbookPlus Platform New Software architectures for Big Data Q & A