SlideShare ist ein Scribd-Unternehmen logo
1 von 67
Downloaden Sie, um offline zu lesen
Lambda Architecture
and Open Source Tools for
Real-time Big Data
● Concepts & Techniques “Thinking with Lambda”
● Case studies in Practice
Trieu Nguyen - http://nguyentantrieu.info or @tantrieuf31
Principal Engineer at eClick Data Analytics team, FPT Online
All contents and thoughts in this slide are my subjective ideas and compiled from
Communities
Just a little introduction
● 2008 Java Developer, developed Social
Trading Network for a small startup (Yopco)
● 2011 worked at FPT Online, software engineer
in Banbe Project, Restful API for VnExpress
Mobile App
● 2012 joined Greengar Studios in 6 months,
scaling backend API mobile games (iOS, Android)
● 2013 back to FPT Online, R&D about Big Data
& Analytics, developing the new core
Analytics Platform (on JVM Platform)
Contents for this talk
●
●
●
●
●
●
●
●

The lessons from history
Problems In Practice
What is the Lambda Architecture?
Why lambda architecture for real-time big
data ?
Open Source Technology Stack
Lambda in Practice (Mobile Data and Web Data)
Lessons I have learned
Questions & Answers
History ?
The best way to predict the future is
looking at the past and now ?
Big data is a buzzword for
old problems
Explaining Big Data
http://www.youtube.com/watch?v=7D1CQ_LOizA
Learning ?
Working ?
Big Data + Old History
http://www.youtube.com/watch?v=tp4y-_VoXdA
This is Big DATA

This is most valuable things!
We can't solve problems
by using the same kind of
thinking we used when we
created them.
Albert Einstein
Think more with
Lambda and Reactive
Where Big Data
can be used
BBC Horizon 2013 The Age of Big Data
http://www.youtube.com/watch?v=RE0ITQ7XQjM
Google’s mission is to

organize

the world’s information and make it
universally accessible and useful.
Organize the world’s
information?
How did Google scale their search engine ?
How does Hadoop really work ?
http://stackoverflow.com/questions/6087834/howscalable-is-mapreduce-in-the-original-functionallanguages
Trends of Now and the Future
MapReduce Programming
Reactive Programming
Functional Programming
Streaming Computation
=> All just the special cases of Lambda

●
●
●
●
So what is the λ
(Lambda)
Architecture ?
the Lambda Architecture:
● apply the (λ) Lambda philosophy in designing big data
system
● equation “query = function(all data)” which is the basis of
all data systems
● proposed by Nathan Marz (http://nathanmarz.com/), a
software engineer from Twitter in his “Big Data” book.
● is based on three main design principles:
○ human fault-tolerance – the system is unsusceptible to data loss or data
corruption because at scale it could be irreparable. (BUGS ?)
○ data immutability – store data in it’s rawest form immutable and for
perpetuity. (INSERT/ SELECT/DELETE but no UPDATE !)
○ recomputation – with the two principles above it is always possible to
(re)-compute results by running a function on the raw data.
Lambda In Practice
2 case studies from my experiences
Case Study 1:
Mobile Data
Monitor API Backend + System KPI
Problem:

Inside “mobile data”,
What's the most
valuable piece of
information
I applied
“Lambda”
here

Backend System for mobile app
Web vs Mobile App
Web
Visitors
Visits
Pageviews
Events

Mobile App
Users
Sessions
Events
Metrics: Cause and Effect
●
●
●
●
●
●
●

Screen Size => App Design, UI/UX, Usability
App version => Deployment, Marketing
Connectivity => Code, User Experience
Location => Marketing, User Behaviour
OS => Marketing, Cost, Development
Memory => User Experience
Feature Session => How to engage app users
The data and the size, not too big for a small
startup!

Where is the lambda ?
I used Groovy + GPars (Groovy Parallel Systems) + MongoDB for fast
parallel computation (actor model) on statistical data
http://gpars.codehaus.org/
The GPars framework offers Java developers intuitive and safe ways to handle
Java or Groovy tasks concurrently.
Support:
●
●
●
●
●
●
●
●

Dataflow concurrency
Actor programming model
CSP
Agent - an thread-safe reference to mutable state
Concurrent collection processing
Composable asynchronous functions
Fork/Join
STM (Software Transactional Memory)
Mobile Apps => Backend APIs =>
Statistics => Find the Trends & Insights?
Reactive Data
Analytics for
Mobile Apps
It means real-time recommendation
by:
➔ context (location, time)
➔ user profile (preferences, level,
...)
Big Data on Small Devices: Data Science goes Mobile
http://strataconf.com/strata2013/public/schedule/detail/27605
Case Study 2:
Web Data
● Real-time Data Analytics
● Monitoring Stream Data (Reactive)

http://eclick.vn
at eClick we must
check campaigns in
near-real-time
(seconds) !

at eClick we have
30~40 GB Logs in Stream
10~20 GB Bandwidth
just for tracking user
actions (click,
impression,...)
in ONE day !
at eClick we have many types of log (video, web,
mobile, system logs, ad-campaign, articles, … )
“lambda architecture”
proposed by @nathanmarz
Internet

Netty Http
Server

TCP Connection

Kafka
Akka Workers
Hadoop Tools

Storm

Redis

Redis

KPI Report
the open-source lambda architecture at eClick
The big-data technology stack
● Netty (http://netty.io/) a framework using reactive programming
pattern for scaling HTTP system easier, by JBoss http://www.jboss.org
● Kafka (http://kafka.apache.org/) a publish-subscribe messaging
rethought as a distributed commit log, open sourced by Linkedin
● Storm (http://storm-project.net/) the framework for distributed
realtime computation system, by Twitter
● Redis (http://redis.io/) a advanced key-value in-memory NoSQL
database, all fast statistical computations in here.
● Groovy for scripting layer on JVM, ad-hoc query on Redis
● Hadoop ecosystem: HDFS, Hive, HBase for batch processing
● RxJava https://github.com/Netflix/RxJava a library for composing
asynchronous and event-based programs
● Hystrix https://github.com/Netflix/Hystrix : for Latency and Fault
Tolerance for Distributed Systems
My new ideas for the future
Connecting the active functor pattern + reactive programming +
stream computation + in-memory computing to make:
● real-time data analytics easier
● better recommendation system
● build more profitable in big data
More Information:
● http://activefunctor.blogspot.com/ (a special case of Lambda
that actively search best connections to form optimal
topology) - from ideas when internship at DRD with my
advisor.
● Can a function be persistent (stored as data), distributed in a
cluster (cloud), reactive to right data (best value in network) ?
● http://www.reactivemanifesto.org/ (reactive pattern)
Lessons
What I have learned from Lambda and Big
Data World
What I have learned
●
●
●
●
●

Study about lambda and read some books
Ask questions=> analytics=> Profit & Value
Collect any data you can, learn inside !
Implement it! Just right tools for right jobs.
Turn your data into the things everyone can
"look & feel"
read papers
Study the “lambda”
I studied Haskell in 2007 with Dr.Peter Gammie http://peteg.org/ when
internship at DRD (a non-profit organization).
● Imperative programs will always be vulnerable to data races because
they contain mutable variables.
● There are no data races in purely functional languages because they
don't have mutable variables.
Reading some books
Improve your business knowledge !
=> read the Behavioral Economics Books

http://www.goodreads.com/shelf/show/behavioral-economics
Collect the data ?
Use your imagination is more than just
knowledge you have
Think more about Butterfly Effect!
Z;
om A to
fr
l get you you
il
“Logic w n will get
in
ginatio - Albert Einste
ima
.”
ywhere
ever
Use you
r
with da imagination
ta
just log analytics, not
ic

Learn Data
Visualization
Questions & Answers
The link of this slide is here:
● http://nguyentantrieu.info/blog/lambda-architecture-andopen-source-tools-for-real-time-big-data/
More useful resources:
● http://nguyentantrieu.info/blog
● http://www.mc2ads.com

Weitere ähnliche Inhalte

Was ist angesagt?

Analyzing 1.2 Million Network Packets per Second in Real-time
Analyzing 1.2 Million Network Packets per Second in Real-timeAnalyzing 1.2 Million Network Packets per Second in Real-time
Analyzing 1.2 Million Network Packets per Second in Real-timeDataWorks Summit
 
dlux - Splunk Technical Overview
dlux - Splunk Technical Overviewdlux - Splunk Technical Overview
dlux - Splunk Technical OverviewDavid Lutz
 
The Event Mesh: real-time, event-driven, responsive APIs and beyond
The Event Mesh: real-time, event-driven, responsive APIs and beyondThe Event Mesh: real-time, event-driven, responsive APIs and beyond
The Event Mesh: real-time, event-driven, responsive APIs and beyondSolace
 
Continuous Integration for Dynamics 365/CRM
Continuous Integration for Dynamics 365/CRMContinuous Integration for Dynamics 365/CRM
Continuous Integration for Dynamics 365/CRMWael Hamze
 
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - Türkçe
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - TürkçeApache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - Türkçe
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - TürkçeEmre Akış
 
Getting Started with Splunk Enterprise - Demo
Getting Started with Splunk Enterprise - DemoGetting Started with Splunk Enterprise - Demo
Getting Started with Splunk Enterprise - DemoSplunk
 
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael MarchJIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael MarchAtlassian
 
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...Splunk
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming JobsDatabricks
 
Apache Camel v3, Camel K and Camel Quarkus
Apache Camel v3, Camel K and Camel QuarkusApache Camel v3, Camel K and Camel Quarkus
Apache Camel v3, Camel K and Camel QuarkusClaus Ibsen
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseSplunk
 
Software Quality Assurance(Sqa) automated software testing
Software Quality Assurance(Sqa) automated software testingSoftware Quality Assurance(Sqa) automated software testing
Software Quality Assurance(Sqa) automated software testingREHMAT ULLAH
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with PrometheusTobias Schmidt
 
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk
 
Feature store: Solving anti-patterns in ML-systems
Feature store: Solving anti-patterns in ML-systemsFeature store: Solving anti-patterns in ML-systems
Feature store: Solving anti-patterns in ML-systemsAndrzej Michałowski
 
Elegant and Scalable Code Querying with Code Property Graphs
Elegant and Scalable Code Querying with Code Property GraphsElegant and Scalable Code Querying with Code Property Graphs
Elegant and Scalable Code Querying with Code Property GraphsConnected Data World
 
Process Mining: Past, Present, and Future
Process Mining: Past, Present, and FutureProcess Mining: Past, Present, and Future
Process Mining: Past, Present, and FutureCelonis
 
Jmeter Performance Testing
Jmeter Performance TestingJmeter Performance Testing
Jmeter Performance TestingAtul Pant
 

Was ist angesagt? (20)

Analyzing 1.2 Million Network Packets per Second in Real-time
Analyzing 1.2 Million Network Packets per Second in Real-timeAnalyzing 1.2 Million Network Packets per Second in Real-time
Analyzing 1.2 Million Network Packets per Second in Real-time
 
dlux - Splunk Technical Overview
dlux - Splunk Technical Overviewdlux - Splunk Technical Overview
dlux - Splunk Technical Overview
 
The Event Mesh: real-time, event-driven, responsive APIs and beyond
The Event Mesh: real-time, event-driven, responsive APIs and beyondThe Event Mesh: real-time, event-driven, responsive APIs and beyond
The Event Mesh: real-time, event-driven, responsive APIs and beyond
 
Continuous Integration for Dynamics 365/CRM
Continuous Integration for Dynamics 365/CRMContinuous Integration for Dynamics 365/CRM
Continuous Integration for Dynamics 365/CRM
 
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - Türkçe
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - TürkçeApache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - Türkçe
Apache Kafka - Yüksek Performanslı Dağıtık Mesajlaşma Sistemi - Türkçe
 
Getting Started with Splunk Enterprise - Demo
Getting Started with Splunk Enterprise - DemoGetting Started with Splunk Enterprise - Demo
Getting Started with Splunk Enterprise - Demo
 
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael MarchJIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
 
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...
Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language ...
 
Productizing Structured Streaming Jobs
Productizing Structured Streaming JobsProductizing Structured Streaming Jobs
Productizing Structured Streaming Jobs
 
Apache Camel v3, Camel K and Camel Quarkus
Apache Camel v3, Camel K and Camel QuarkusApache Camel v3, Camel K and Camel Quarkus
Apache Camel v3, Camel K and Camel Quarkus
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
 
Software Quality Assurance(Sqa) automated software testing
Software Quality Assurance(Sqa) automated software testingSoftware Quality Assurance(Sqa) automated software testing
Software Quality Assurance(Sqa) automated software testing
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with Prometheus
 
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
 
Feature store: Solving anti-patterns in ML-systems
Feature store: Solving anti-patterns in ML-systemsFeature store: Solving anti-patterns in ML-systems
Feature store: Solving anti-patterns in ML-systems
 
Elegant and Scalable Code Querying with Code Property Graphs
Elegant and Scalable Code Querying with Code Property GraphsElegant and Scalable Code Querying with Code Property Graphs
Elegant and Scalable Code Querying with Code Property Graphs
 
Performance Engineering Basics
Performance Engineering BasicsPerformance Engineering Basics
Performance Engineering Basics
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
 
Process Mining: Past, Present, and Future
Process Mining: Past, Present, and FutureProcess Mining: Past, Present, and Future
Process Mining: Past, Present, and Future
 
Jmeter Performance Testing
Jmeter Performance TestingJmeter Performance Testing
Jmeter Performance Testing
 

Andere mochten auch

100 blue mix days technical training
100 blue mix days technical training100 blue mix days technical training
100 blue mix days technical trainingAjit Yohannan
 
Cloud Camp: Infrastructure as a service advance workloads
Cloud Camp: Infrastructure as a service advance workloadsCloud Camp: Infrastructure as a service advance workloads
Cloud Camp: Infrastructure as a service advance workloadsAsaf Nakash
 
De Persgroep Big Data Expo
De Persgroep Big Data ExpoDe Persgroep Big Data Expo
De Persgroep Big Data ExpoBigDataExpo
 
Delivering Quality Open Data by Chelsea Ursaner
Delivering Quality Open Data by Chelsea UrsanerDelivering Quality Open Data by Chelsea Ursaner
Delivering Quality Open Data by Chelsea UrsanerData Con LA
 
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...Lucidworks
 
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...SPS Paris
 
VMs All the Way Down (BSides Delaware 2016)
VMs All the Way Down (BSides Delaware 2016)VMs All the Way Down (BSides Delaware 2016)
VMs All the Way Down (BSides Delaware 2016)John Hubbard
 
Global Azure Bootcamp - Azure OMS
Global Azure Bootcamp - Azure OMSGlobal Azure Bootcamp - Azure OMS
Global Azure Bootcamp - Azure OMSBruno Lopes
 
(SEC320) Leveraging the Power of AWS to Automate Security & Compliance
(SEC320) Leveraging the Power of AWS to Automate Security & Compliance(SEC320) Leveraging the Power of AWS to Automate Security & Compliance
(SEC320) Leveraging the Power of AWS to Automate Security & ComplianceAmazon Web Services
 
KD2017_System Center in the "cloud first" era
KD2017_System Center in the "cloud first" eraKD2017_System Center in the "cloud first" era
KD2017_System Center in the "cloud first" eraTomica Kaniski
 
Big data for cio 2015
Big data for cio 2015Big data for cio 2015
Big data for cio 2015Zohar Elkayam
 
Bioocean1 :Introduction to Biological Oceanography
Bioocean1 :Introduction to Biological Oceanography Bioocean1 :Introduction to Biological Oceanography
Bioocean1 :Introduction to Biological Oceanography Gazi Abdullah
 
Generalized B2B Machine Learning by Andrew Waage
Generalized B2B Machine Learning by Andrew WaageGeneralized B2B Machine Learning by Andrew Waage
Generalized B2B Machine Learning by Andrew WaageData Con LA
 
SRE Study Notes - CH2,3,4
SRE Study Notes - CH2,3,4SRE Study Notes - CH2,3,4
SRE Study Notes - CH2,3,4Rick Hwang
 
Docker containerization cookbook
Docker containerization cookbookDocker containerization cookbook
Docker containerization cookbookPascal Louis
 
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VA
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VACleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VA
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VAClearedJobs.Net
 
Walmart Big Data Expo
Walmart Big Data ExpoWalmart Big Data Expo
Walmart Big Data ExpoBigDataExpo
 

Andere mochten auch (20)

Voetsporen 38
Voetsporen 38Voetsporen 38
Voetsporen 38
 
100 blue mix days technical training
100 blue mix days technical training100 blue mix days technical training
100 blue mix days technical training
 
Cloud Camp: Infrastructure as a service advance workloads
Cloud Camp: Infrastructure as a service advance workloadsCloud Camp: Infrastructure as a service advance workloads
Cloud Camp: Infrastructure as a service advance workloads
 
De Persgroep Big Data Expo
De Persgroep Big Data ExpoDe Persgroep Big Data Expo
De Persgroep Big Data Expo
 
Delivering Quality Open Data by Chelsea Ursaner
Delivering Quality Open Data by Chelsea UrsanerDelivering Quality Open Data by Chelsea Ursaner
Delivering Quality Open Data by Chelsea Ursaner
 
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...
Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...
 
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...
I1 - Securing Office 365 and Microsoft Azure like a rockstar (or like a group...
 
VMs All the Way Down (BSides Delaware 2016)
VMs All the Way Down (BSides Delaware 2016)VMs All the Way Down (BSides Delaware 2016)
VMs All the Way Down (BSides Delaware 2016)
 
Global Azure Bootcamp - Azure OMS
Global Azure Bootcamp - Azure OMSGlobal Azure Bootcamp - Azure OMS
Global Azure Bootcamp - Azure OMS
 
(SEC320) Leveraging the Power of AWS to Automate Security & Compliance
(SEC320) Leveraging the Power of AWS to Automate Security & Compliance(SEC320) Leveraging the Power of AWS to Automate Security & Compliance
(SEC320) Leveraging the Power of AWS to Automate Security & Compliance
 
KD2017_System Center in the "cloud first" era
KD2017_System Center in the "cloud first" eraKD2017_System Center in the "cloud first" era
KD2017_System Center in the "cloud first" era
 
Big data for cio 2015
Big data for cio 2015Big data for cio 2015
Big data for cio 2015
 
Bioocean1 :Introduction to Biological Oceanography
Bioocean1 :Introduction to Biological Oceanography Bioocean1 :Introduction to Biological Oceanography
Bioocean1 :Introduction to Biological Oceanography
 
Generalized B2B Machine Learning by Andrew Waage
Generalized B2B Machine Learning by Andrew WaageGeneralized B2B Machine Learning by Andrew Waage
Generalized B2B Machine Learning by Andrew Waage
 
SRE Study Notes - CH2,3,4
SRE Study Notes - CH2,3,4SRE Study Notes - CH2,3,4
SRE Study Notes - CH2,3,4
 
Docker containerization cookbook
Docker containerization cookbookDocker containerization cookbook
Docker containerization cookbook
 
Understanding big data
Understanding big dataUnderstanding big data
Understanding big data
 
Greach 2014 Sesamestreet Grails2 Workshop
Greach 2014 Sesamestreet Grails2 Workshop Greach 2014 Sesamestreet Grails2 Workshop
Greach 2014 Sesamestreet Grails2 Workshop
 
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VA
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VACleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VA
Cleared Job Fair Job Seeker Handbook June 15, 2017, Dulles, VA
 
Walmart Big Data Expo
Walmart Big Data ExpoWalmart Big Data Expo
Walmart Big Data Expo
 

Ähnlich wie Lambda Architecture and open source technology stack for real time big data

Lambda architecture for real time big data
Lambda architecture for real time big dataLambda architecture for real time big data
Lambda architecture for real time big dataTrieu Nguyen
 
Building Reactive Real-time Data Pipeline
Building Reactive Real-time Data PipelineBuilding Reactive Real-time Data Pipeline
Building Reactive Real-time Data PipelineTrieu Nguyen
 
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - TrivadisTechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - TrivadisTrivadis
 
Choosing the Right Database - Facebook DevC Malang Hackdays 2017
Choosing the Right Database - Facebook DevC Malang Hackdays 2017Choosing the Right Database - Facebook DevC Malang Hackdays 2017
Choosing the Right Database - Facebook DevC Malang Hackdays 2017Rendy Bambang Junior
 
Making Machine Learning Easy with H2O and WebFlux
Making Machine Learning Easy with H2O and WebFluxMaking Machine Learning Easy with H2O and WebFlux
Making Machine Learning Easy with H2O and WebFluxTrayan Iliev
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futuremarkgrover
 
Lyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesLyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesKarthik Murugesan
 
London atlassian meetup 31 jan 2016 jira metrics-extract slides
London atlassian meetup 31 jan 2016 jira metrics-extract slidesLondon atlassian meetup 31 jan 2016 jira metrics-extract slides
London atlassian meetup 31 jan 2016 jira metrics-extract slidesRudiger Wolf
 
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier DominguezMultiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier DominguezBig Data Spain
 
Keynote at-icpc-2020
Keynote at-icpc-2020Keynote at-icpc-2020
Keynote at-icpc-2020Ralf Laemmel
 
Is Spark the right choice for data analysis ?
Is Spark the right choice for data analysis ?Is Spark the right choice for data analysis ?
Is Spark the right choice for data analysis ?Ahmed Kamal
 
UX Analytics for Data-driven Product Development
UX Analytics for Data-driven Product DevelopmentUX Analytics for Data-driven Product Development
UX Analytics for Data-driven Product DevelopmentTrieu Nguyen
 
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...HostedbyConfluent
 
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Demi Ben-Ari
 
GraphLab Conference 2014 Keynote - Carlos Guestrin
GraphLab Conference 2014 Keynote - Carlos GuestrinGraphLab Conference 2014 Keynote - Carlos Guestrin
GraphLab Conference 2014 Keynote - Carlos GuestrinTuri, Inc.
 
Real time machine learning
Real time machine learningReal time machine learning
Real time machine learningVinoth Kannan
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Codemotion
 

Ähnlich wie Lambda Architecture and open source technology stack for real time big data (20)

Lambda architecture for real time big data
Lambda architecture for real time big dataLambda architecture for real time big data
Lambda architecture for real time big data
 
Building Reactive Real-time Data Pipeline
Building Reactive Real-time Data PipelineBuilding Reactive Real-time Data Pipeline
Building Reactive Real-time Data Pipeline
 
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - TrivadisTechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
 
Choosing the Right Database - Facebook DevC Malang Hackdays 2017
Choosing the Right Database - Facebook DevC Malang Hackdays 2017Choosing the Right Database - Facebook DevC Malang Hackdays 2017
Choosing the Right Database - Facebook DevC Malang Hackdays 2017
 
Making Machine Learning Easy with H2O and WebFlux
Making Machine Learning Easy with H2O and WebFluxMaking Machine Learning Easy with H2O and WebFlux
Making Machine Learning Easy with H2O and WebFlux
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the future
 
Lyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesLyft data Platform - 2019 slides
Lyft data Platform - 2019 slides
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
 
London atlassian meetup 31 jan 2016 jira metrics-extract slides
London atlassian meetup 31 jan 2016 jira metrics-extract slidesLondon atlassian meetup 31 jan 2016 jira metrics-extract slides
London atlassian meetup 31 jan 2016 jira metrics-extract slides
 
Multiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier DominguezMultiplatform Spark solution for Graph datasources by Javier Dominguez
Multiplatform Spark solution for Graph datasources by Javier Dominguez
 
Keynote at-icpc-2020
Keynote at-icpc-2020Keynote at-icpc-2020
Keynote at-icpc-2020
 
Is Spark the right choice for data analysis ?
Is Spark the right choice for data analysis ?Is Spark the right choice for data analysis ?
Is Spark the right choice for data analysis ?
 
UX Analytics for Data-driven Product Development
UX Analytics for Data-driven Product DevelopmentUX Analytics for Data-driven Product Development
UX Analytics for Data-driven Product Development
 
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...
Considerations for Abstracting Complexities of a Real-Time ML Platform, Zhenz...
 
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
 
GraphLab Conference 2014 Keynote - Carlos Guestrin
GraphLab Conference 2014 Keynote - Carlos GuestrinGraphLab Conference 2014 Keynote - Carlos Guestrin
GraphLab Conference 2014 Keynote - Carlos Guestrin
 
Real time machine learning
Real time machine learningReal time machine learning
Real time machine learning
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
 

Mehr von Trieu Nguyen

Building Your Customer Data Platform with LEO CDP in Travel Industry.pdf
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdfBuilding Your Customer Data Platform with LEO CDP in Travel Industry.pdf
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdfTrieu Nguyen
 
Building Your Customer Data Platform with LEO CDP - Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP - Spa and Hotel BusinessBuilding Your Customer Data Platform with LEO CDP - Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP - Spa and Hotel BusinessTrieu Nguyen
 
Building Your Customer Data Platform with LEO CDP
Building Your Customer Data Platform with LEO CDP Building Your Customer Data Platform with LEO CDP
Building Your Customer Data Platform with LEO CDP Trieu Nguyen
 
How to track and improve Customer Experience with LEO CDP
How to track and improve Customer Experience with LEO CDPHow to track and improve Customer Experience with LEO CDP
How to track and improve Customer Experience with LEO CDPTrieu Nguyen
 
[Notes] Customer 360 Analytics with LEO CDP
[Notes] Customer 360 Analytics with LEO CDP[Notes] Customer 360 Analytics with LEO CDP
[Notes] Customer 360 Analytics with LEO CDPTrieu Nguyen
 
Leo CDP - Pitch Deck
Leo CDP - Pitch DeckLeo CDP - Pitch Deck
Leo CDP - Pitch DeckTrieu Nguyen
 
LEO CDP - What's new in 2022
LEO CDP  - What's new in 2022LEO CDP  - What's new in 2022
LEO CDP - What's new in 2022Trieu Nguyen
 
Lộ trình triển khai LEO CDP cho ngành bất động sản
Lộ trình triển khai LEO CDP cho ngành bất động sảnLộ trình triển khai LEO CDP cho ngành bất động sản
Lộ trình triển khai LEO CDP cho ngành bất động sảnTrieu Nguyen
 
Why is LEO CDP important for digital business ?
Why is LEO CDP important for digital business ?Why is LEO CDP important for digital business ?
Why is LEO CDP important for digital business ?Trieu Nguyen
 
From Dataism to Customer Data Platform
From Dataism to Customer Data PlatformFrom Dataism to Customer Data Platform
From Dataism to Customer Data PlatformTrieu Nguyen
 
Data collection, processing & organization with USPA framework
Data collection, processing & organization with USPA frameworkData collection, processing & organization with USPA framework
Data collection, processing & organization with USPA frameworkTrieu Nguyen
 
Part 1: Introduction to digital marketing technology
Part 1: Introduction to digital marketing technologyPart 1: Introduction to digital marketing technology
Part 1: Introduction to digital marketing technologyTrieu Nguyen
 
Why is Customer Data Platform (CDP) ?
Why is Customer Data Platform (CDP) ?Why is Customer Data Platform (CDP) ?
Why is Customer Data Platform (CDP) ?Trieu Nguyen
 
How to build a Personalized News Recommendation Platform
How to build a Personalized News Recommendation PlatformHow to build a Personalized News Recommendation Platform
How to build a Personalized News Recommendation PlatformTrieu Nguyen
 
How to grow your business in the age of digital marketing 4.0
How to grow your business  in the age of digital marketing 4.0How to grow your business  in the age of digital marketing 4.0
How to grow your business in the age of digital marketing 4.0Trieu Nguyen
 
Video Ecosystem and some ideas about video big data
Video Ecosystem and some ideas about video big dataVideo Ecosystem and some ideas about video big data
Video Ecosystem and some ideas about video big dataTrieu Nguyen
 
Concepts, use cases and principles to build big data systems (1)
Concepts, use cases and principles to build big data systems (1)Concepts, use cases and principles to build big data systems (1)
Concepts, use cases and principles to build big data systems (1)Trieu Nguyen
 
Open OTT - Video Content Platform
Open OTT - Video Content PlatformOpen OTT - Video Content Platform
Open OTT - Video Content PlatformTrieu Nguyen
 
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Apache Hadoop and Spark: Introduction and Use Cases for Data AnalysisApache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Apache Hadoop and Spark: Introduction and Use Cases for Data AnalysisTrieu Nguyen
 
Introduction to Recommendation Systems (Vietnam Web Submit)
Introduction to Recommendation Systems (Vietnam Web Submit)Introduction to Recommendation Systems (Vietnam Web Submit)
Introduction to Recommendation Systems (Vietnam Web Submit)Trieu Nguyen
 

Mehr von Trieu Nguyen (20)

Building Your Customer Data Platform with LEO CDP in Travel Industry.pdf
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdfBuilding Your Customer Data Platform with LEO CDP in Travel Industry.pdf
Building Your Customer Data Platform with LEO CDP in Travel Industry.pdf
 
Building Your Customer Data Platform with LEO CDP - Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP - Spa and Hotel BusinessBuilding Your Customer Data Platform with LEO CDP - Spa and Hotel Business
Building Your Customer Data Platform with LEO CDP - Spa and Hotel Business
 
Building Your Customer Data Platform with LEO CDP
Building Your Customer Data Platform with LEO CDP Building Your Customer Data Platform with LEO CDP
Building Your Customer Data Platform with LEO CDP
 
How to track and improve Customer Experience with LEO CDP
How to track and improve Customer Experience with LEO CDPHow to track and improve Customer Experience with LEO CDP
How to track and improve Customer Experience with LEO CDP
 
[Notes] Customer 360 Analytics with LEO CDP
[Notes] Customer 360 Analytics with LEO CDP[Notes] Customer 360 Analytics with LEO CDP
[Notes] Customer 360 Analytics with LEO CDP
 
Leo CDP - Pitch Deck
Leo CDP - Pitch DeckLeo CDP - Pitch Deck
Leo CDP - Pitch Deck
 
LEO CDP - What's new in 2022
LEO CDP  - What's new in 2022LEO CDP  - What's new in 2022
LEO CDP - What's new in 2022
 
Lộ trình triển khai LEO CDP cho ngành bất động sản
Lộ trình triển khai LEO CDP cho ngành bất động sảnLộ trình triển khai LEO CDP cho ngành bất động sản
Lộ trình triển khai LEO CDP cho ngành bất động sản
 
Why is LEO CDP important for digital business ?
Why is LEO CDP important for digital business ?Why is LEO CDP important for digital business ?
Why is LEO CDP important for digital business ?
 
From Dataism to Customer Data Platform
From Dataism to Customer Data PlatformFrom Dataism to Customer Data Platform
From Dataism to Customer Data Platform
 
Data collection, processing & organization with USPA framework
Data collection, processing & organization with USPA frameworkData collection, processing & organization with USPA framework
Data collection, processing & organization with USPA framework
 
Part 1: Introduction to digital marketing technology
Part 1: Introduction to digital marketing technologyPart 1: Introduction to digital marketing technology
Part 1: Introduction to digital marketing technology
 
Why is Customer Data Platform (CDP) ?
Why is Customer Data Platform (CDP) ?Why is Customer Data Platform (CDP) ?
Why is Customer Data Platform (CDP) ?
 
How to build a Personalized News Recommendation Platform
How to build a Personalized News Recommendation PlatformHow to build a Personalized News Recommendation Platform
How to build a Personalized News Recommendation Platform
 
How to grow your business in the age of digital marketing 4.0
How to grow your business  in the age of digital marketing 4.0How to grow your business  in the age of digital marketing 4.0
How to grow your business in the age of digital marketing 4.0
 
Video Ecosystem and some ideas about video big data
Video Ecosystem and some ideas about video big dataVideo Ecosystem and some ideas about video big data
Video Ecosystem and some ideas about video big data
 
Concepts, use cases and principles to build big data systems (1)
Concepts, use cases and principles to build big data systems (1)Concepts, use cases and principles to build big data systems (1)
Concepts, use cases and principles to build big data systems (1)
 
Open OTT - Video Content Platform
Open OTT - Video Content PlatformOpen OTT - Video Content Platform
Open OTT - Video Content Platform
 
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Apache Hadoop and Spark: Introduction and Use Cases for Data AnalysisApache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
 
Introduction to Recommendation Systems (Vietnam Web Submit)
Introduction to Recommendation Systems (Vietnam Web Submit)Introduction to Recommendation Systems (Vietnam Web Submit)
Introduction to Recommendation Systems (Vietnam Web Submit)
 

Kürzlich hochgeladen

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????blackmambaettijean
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 

Kürzlich hochgeladen (20)

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 

Lambda Architecture and open source technology stack for real time big data

  • 1. Lambda Architecture and Open Source Tools for Real-time Big Data ● Concepts & Techniques “Thinking with Lambda” ● Case studies in Practice Trieu Nguyen - http://nguyentantrieu.info or @tantrieuf31 Principal Engineer at eClick Data Analytics team, FPT Online All contents and thoughts in this slide are my subjective ideas and compiled from Communities
  • 2. Just a little introduction ● 2008 Java Developer, developed Social Trading Network for a small startup (Yopco) ● 2011 worked at FPT Online, software engineer in Banbe Project, Restful API for VnExpress Mobile App ● 2012 joined Greengar Studios in 6 months, scaling backend API mobile games (iOS, Android) ● 2013 back to FPT Online, R&D about Big Data & Analytics, developing the new core Analytics Platform (on JVM Platform)
  • 3. Contents for this talk ● ● ● ● ● ● ● ● The lessons from history Problems In Practice What is the Lambda Architecture? Why lambda architecture for real-time big data ? Open Source Technology Stack Lambda in Practice (Mobile Data and Web Data) Lessons I have learned Questions & Answers
  • 4. History ? The best way to predict the future is looking at the past and now ?
  • 5. Big data is a buzzword for old problems
  • 9.
  • 10. Big Data + Old History http://www.youtube.com/watch?v=tp4y-_VoXdA
  • 11. This is Big DATA This is most valuable things!
  • 12.
  • 13. We can't solve problems by using the same kind of thinking we used when we created them. Albert Einstein Think more with Lambda and Reactive
  • 14.
  • 15.
  • 17.
  • 18. BBC Horizon 2013 The Age of Big Data http://www.youtube.com/watch?v=RE0ITQ7XQjM
  • 19.
  • 20.
  • 21.
  • 22.
  • 23. Google’s mission is to organize the world’s information and make it universally accessible and useful.
  • 24.
  • 26.
  • 27. How did Google scale their search engine ? How does Hadoop really work ?
  • 28.
  • 30. Trends of Now and the Future MapReduce Programming Reactive Programming Functional Programming Streaming Computation => All just the special cases of Lambda ● ● ● ●
  • 31.
  • 32. So what is the λ (Lambda) Architecture ?
  • 33.
  • 34.
  • 35. the Lambda Architecture: ● apply the (λ) Lambda philosophy in designing big data system ● equation “query = function(all data)” which is the basis of all data systems ● proposed by Nathan Marz (http://nathanmarz.com/), a software engineer from Twitter in his “Big Data” book. ● is based on three main design principles: ○ human fault-tolerance – the system is unsusceptible to data loss or data corruption because at scale it could be irreparable. (BUGS ?) ○ data immutability – store data in it’s rawest form immutable and for perpetuity. (INSERT/ SELECT/DELETE but no UPDATE !) ○ recomputation – with the two principles above it is always possible to (re)-compute results by running a function on the raw data.
  • 36. Lambda In Practice 2 case studies from my experiences
  • 37. Case Study 1: Mobile Data Monitor API Backend + System KPI
  • 38. Problem: Inside “mobile data”, What's the most valuable piece of information
  • 40. Web vs Mobile App Web Visitors Visits Pageviews Events Mobile App Users Sessions Events
  • 41. Metrics: Cause and Effect ● ● ● ● ● ● ● Screen Size => App Design, UI/UX, Usability App version => Deployment, Marketing Connectivity => Code, User Experience Location => Marketing, User Behaviour OS => Marketing, Cost, Development Memory => User Experience Feature Session => How to engage app users
  • 42. The data and the size, not too big for a small startup! Where is the lambda ? I used Groovy + GPars (Groovy Parallel Systems) + MongoDB for fast parallel computation (actor model) on statistical data http://gpars.codehaus.org/ The GPars framework offers Java developers intuitive and safe ways to handle Java or Groovy tasks concurrently. Support: ● ● ● ● ● ● ● ● Dataflow concurrency Actor programming model CSP Agent - an thread-safe reference to mutable state Concurrent collection processing Composable asynchronous functions Fork/Join STM (Software Transactional Memory)
  • 43. Mobile Apps => Backend APIs => Statistics => Find the Trends & Insights?
  • 44. Reactive Data Analytics for Mobile Apps It means real-time recommendation by: ➔ context (location, time) ➔ user profile (preferences, level, ...)
  • 45. Big Data on Small Devices: Data Science goes Mobile http://strataconf.com/strata2013/public/schedule/detail/27605
  • 46. Case Study 2: Web Data ● Real-time Data Analytics ● Monitoring Stream Data (Reactive) http://eclick.vn
  • 47. at eClick we must check campaigns in near-real-time (seconds) ! at eClick we have 30~40 GB Logs in Stream 10~20 GB Bandwidth just for tracking user actions (click, impression,...) in ONE day ! at eClick we have many types of log (video, web, mobile, system logs, ad-campaign, articles, … )
  • 48.
  • 49.
  • 51. Internet Netty Http Server TCP Connection Kafka Akka Workers Hadoop Tools Storm Redis Redis KPI Report the open-source lambda architecture at eClick
  • 52. The big-data technology stack ● Netty (http://netty.io/) a framework using reactive programming pattern for scaling HTTP system easier, by JBoss http://www.jboss.org ● Kafka (http://kafka.apache.org/) a publish-subscribe messaging rethought as a distributed commit log, open sourced by Linkedin ● Storm (http://storm-project.net/) the framework for distributed realtime computation system, by Twitter ● Redis (http://redis.io/) a advanced key-value in-memory NoSQL database, all fast statistical computations in here. ● Groovy for scripting layer on JVM, ad-hoc query on Redis ● Hadoop ecosystem: HDFS, Hive, HBase for batch processing ● RxJava https://github.com/Netflix/RxJava a library for composing asynchronous and event-based programs ● Hystrix https://github.com/Netflix/Hystrix : for Latency and Fault Tolerance for Distributed Systems
  • 53. My new ideas for the future Connecting the active functor pattern + reactive programming + stream computation + in-memory computing to make: ● real-time data analytics easier ● better recommendation system ● build more profitable in big data More Information: ● http://activefunctor.blogspot.com/ (a special case of Lambda that actively search best connections to form optimal topology) - from ideas when internship at DRD with my advisor. ● Can a function be persistent (stored as data), distributed in a cluster (cloud), reactive to right data (best value in network) ? ● http://www.reactivemanifesto.org/ (reactive pattern)
  • 54. Lessons What I have learned from Lambda and Big Data World
  • 55.
  • 56. What I have learned ● ● ● ● ● Study about lambda and read some books Ask questions=> analytics=> Profit & Value Collect any data you can, learn inside ! Implement it! Just right tools for right jobs. Turn your data into the things everyone can "look & feel"
  • 58. Study the “lambda” I studied Haskell in 2007 with Dr.Peter Gammie http://peteg.org/ when internship at DRD (a non-profit organization). ● Imperative programs will always be vulnerable to data races because they contain mutable variables. ● There are no data races in purely functional languages because they don't have mutable variables.
  • 60.
  • 61.
  • 62. Improve your business knowledge ! => read the Behavioral Economics Books http://www.goodreads.com/shelf/show/behavioral-economics
  • 64. Use your imagination is more than just knowledge you have
  • 65. Think more about Butterfly Effect!
  • 66. Z; om A to fr l get you you il “Logic w n will get in ginatio - Albert Einste ima .” ywhere ever Use you r with da imagination ta just log analytics, not ic Learn Data Visualization
  • 67. Questions & Answers The link of this slide is here: ● http://nguyentantrieu.info/blog/lambda-architecture-andopen-source-tools-for-real-time-big-data/ More useful resources: ● http://nguyentantrieu.info/blog ● http://www.mc2ads.com