SlideShare a Scribd company logo
1 of 89
Download to read offline
The hardest part of microservices: your data
Full slide deck here:
http://bit.ly/ceposta-hardest-part
Twitter: @christianposta
Blog: http://blog.christianposta.com
Email: christian@redhat.com
Christian Posta
Principal Architect – Red Hat
• Author “Microservices for Java Developers”
• Committer/contributor Apache Camel, Apache ActiveMQ,
Fabric8.io, Apache Kafka, Debezium.io, et. al.
• Worked with large Microservices, web-scale, unicorn
company
Free download @ http://developers.redhat.com
The hardest part of microservices: your data
The hardest part of microservices: your data
People try to copy Netflix, but they can only
copy what they see. They copy the
results, not the process.
Adrian Cockcroft, former Chief Cloud Architect, Netflix
“Microservices” is about optimizing… for speed.
• Maybe it doesn’t matter so much… What
we really care about is speed, reduced
time to value, and business outcomes.
• Maybe a data-driven approach is a better
way to answer this question...
Are you doing microservices?
• Number of features accepted
• % of features completed
• User satisfaction
• Feature Cycle time
• defects discovered after deployment
• customer lifetime value (future profit as a result of relationship with the
customer) https://en.wikipedia.org/wiki/Customer_lifetime_value
• revenue per feature
• mean time to recovery
• % improvement in SLA
• number of changes
• number of user complaints, recommendations, suggestions
• % favorable rating in surveys
• % of users using which features
• % reduction in error rates
• avg number of tx / user
• MANY MORE!
Are you doing microservices?
How does your company go fast?
Manage dependencies.
Data is a major dependency.
Wait. What is data?
What is one “thing”?
Book checkout / purchase Title Search
Recommendations
Weekly reporting
The hardest part of microservices: your data
Focus on domain models, not data models
• Break things into smaller,
understandable models
• Surround a model and its
“context” with a boundary
• Implement the model in code
or get a new model
• Explicitly map between
different contexts
• Model transactional
boundaries as aggregates
Aggregates
• Use the domain to lead you to invariant rules across your domain
model
• Model the invariants and their associated entities/value objects as
“aggregates”
• Aggregates focus on transactional boundaries (ie, transactional in
the “A” from ACID sense)
• Individual aggregates are transactionally consistent
• Aggregates use relaxed consistency models between aggregates
(ie, something like the Actor model?)
• Bounded Contexts use relaxed consistency models between
boundaries
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
Stick with these conveniences as long as you can.
Seriously.
But ...
• Load/size is too great to fit on one box
• Modules/use cases have different read/write
characteristics
• Queries/joins are getting too complex
• Security issues
• Lots of conflicting changes to the model/schema
• Need denormalized, optimized indexing engines
• We can live with eventual consistency (whatever that
really means)
From here on out, what we’re saying is
“thank you old reliable, awesome database…
we’ve got it from here”…
Kinda looks like a combinatorial mess….
“A microservice has its own database”
The hardest part of microservices: your data
How do we deal with data in this world?
We need to understand something about the data
inside our services and the data outside our services.
https://msdn.microsoft.com/en-us/library/ms954587.aspx
Data inside a service
Data inside a service
Data outside a service
Data outside a service
Data outside a service
We’re now building a full-fledged distributed system.
Some things to remember…
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
Plan for failures.
Build concepts of time, delay,
network, and failures into the
design as a first-class citizen.
How do you “read” data and how do you “update” data.
tx.begin()
c = retrieveCustomer()
c.addNewAddress(address)
tx.add(c)
tx.commit()
publishAddressChange(address, c.id)
tx.begin()
c = retrieveCustomer()
c.addNewAddress(address)
tx.add(c)
publishAddressChange(address, c.id)
tx.commit()
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
Separate reads and writes
(CQRS)
https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/
https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/
getBulkHats()
getBulkHatsForCatsExcept()
wellReallyIJustWantCertainHats()
justExecuteThisSqlForMe()
https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/
For our reads and writes, we need some “consistency”.
What is consistency?
The history of past operations we observe as
a reader of the data
The hardest part of microservices: your data
The hardest part of microservices: your data
We need reads and writes. But we expect failures. This
is starting to sound like a distributed-systems theorem
I’ve heard…
CAP tells us to pick 2: Consistency, Availability,
Partition Tolerance
CAP is a bad way to think about this.
The hardest part of microservices: your data
Linearizable (strict) consistency
CAP - C
Sequential consistency
Monotonic reads consistency
Eventual consistency
Consistency models…
https://en.wikipedia.org/wiki/Consistency_model
• Strict consistency (Linearizability)
• Sequential consistency
• Causal consistency
• Processor consistency
• PRAM consistency (FIFO)
• Bounded staleness consistency
• Monotonic read consistency
• Monotonic write consistency
• Read your writes consistency
• Eventual consistency
Can we really use relaxed consistency models?
Tradeoffs to make with read consistency and
performance
Replicated Data Consistency Explained through Baseball
(Doug Terry)
https://www.microsoft.com/en-us/research/publication/
replicated-data-consistency-explained-through-baseball/
• What consistency model do you need, depending on what
role you’re playing?
• What consistency model are you willing to pay for?
• Official score keeper? (Linearizability or RMW)
• Umpire? (Linearizability)
• Sports writer? (Bounded staleness, Eventual consistency)
• Radio updates? (Monotonic read, Bounded staleness)
• Statistician (Bounded staleness)
• Friends in the pub (Eventual consistency)
Replicated Data Consistency Explained through Baseball
(Doug Terry)
https://www.microsoft.com/en-us/research/publication/
replicated-data-consistency-explained-through-baseball/
Maybe we can use a relaxed consistency model for some
of those previously mentioned use cases…
Example relaxing consistency…
Internet companies created their own tools
for helping with this. (some opensource!!)
• Yelp – MySQL Streamer
https://github.com/Yelp/mysql_streamer
• LinkedIn – Databus
https://github.com/linkedin/databus
• Zendesk – Maxwell
https://github.com/zendesk/maxwell
Meet debezium.io
The hardest part of microservices: your data
Meet debezium.io
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
The hardest part of microservices: your data
WePay uses Debezium
https://wecode.wepay.com/posts/streaming-databases-in-realtime-with-mysql-debezium-kafka
Meet debezium.io
Twitter: @christianposta
Blog: http://blog.christianposta.com
Email: christian@redhat.com
Thanks for listening! Time for demo?

More Related Content

What's hot

APACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsAPACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsKetan Gote
 
Modeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQLModeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQLScyllaDB
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registryconfluent
 
Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium confluent
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
 
Deep Dive into Apache Kafka
Deep Dive into Apache KafkaDeep Dive into Apache Kafka
Deep Dive into Apache Kafkaconfluent
 
Cross Data Center Replication with Redis using Redis Enterprise
Cross Data Center Replication with Redis using Redis EnterpriseCross Data Center Replication with Redis using Redis Enterprise
Cross Data Center Replication with Redis using Redis EnterpriseCihan Biyikoglu
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingAraf Karsh Hamid
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controllerconfluent
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...LibbySchulze
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요Jo Hoon
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang
 
Introduction to the Disruptor
Introduction to the DisruptorIntroduction to the Disruptor
Introduction to the DisruptorTrisha Gee
 
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDPDockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDPThomas Graf
 
Building Bizweb Microservices with Docker
Building Bizweb Microservices with DockerBuilding Bizweb Microservices with Docker
Building Bizweb Microservices with DockerKhôi Nguyễn Minh
 
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedInBuilding a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedInDataWorks Summit
 
Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Brian Brazil
 

What's hot (20)

APACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka StreamsAPACHE KAFKA / Kafka Connect / Kafka Streams
APACHE KAFKA / Kafka Connect / Kafka Streams
 
Modeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQLModeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQL
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
 
Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
Deep Dive into Apache Kafka
Deep Dive into Apache KafkaDeep Dive into Apache Kafka
Deep Dive into Apache Kafka
 
Cross Data Center Replication with Redis using Redis Enterprise
Cross Data Center Replication with Redis using Redis EnterpriseCross Data Center Replication with Redis using Redis Enterprise
Cross Data Center Replication with Redis using Redis Enterprise
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb Sharding
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Kafka Security
Kafka SecurityKafka Security
Kafka Security
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
Introduction to the Disruptor
Introduction to the DisruptorIntroduction to the Disruptor
Introduction to the Disruptor
 
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDPDockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
 
Building Bizweb Microservices with Docker
Building Bizweb Microservices with DockerBuilding Bizweb Microservices with Docker
Building Bizweb Microservices with Docker
 
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedInBuilding a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
 
Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)
 

Similar to The hardest part of microservices: your data

A microservices journey - Round 2
A microservices journey - Round 2A microservices journey - Round 2
A microservices journey - Round 2Christian Posta
 
Microservices with Apache Camel, Docker and Fabric8 v2
Microservices with Apache Camel, Docker and Fabric8 v2Microservices with Apache Camel, Docker and Fabric8 v2
Microservices with Apache Camel, Docker and Fabric8 v2Christian Posta
 
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservicesExploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices💡 Tomasz Kogut
 
Advanced web application architecture Way2Web
Advanced web application architecture Way2WebAdvanced web application architecture Way2Web
Advanced web application architecture Way2WebMatthias Noback
 
Microservices Journey Summer 2017
Microservices Journey Summer 2017Microservices Journey Summer 2017
Microservices Journey Summer 2017Christian Posta
 
Advanced web application architecture - Talk
Advanced web application architecture - TalkAdvanced web application architecture - Talk
Advanced web application architecture - TalkMatthias Noback
 
The Hardest Part of Microservices: Calling Your Services
The Hardest Part of Microservices: Calling Your ServicesThe Hardest Part of Microservices: Calling Your Services
The Hardest Part of Microservices: Calling Your ServicesChristian Posta
 
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, Europe
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, EuropePatterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, Europe
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, EuropeFlip Kromer
 
PHX DevOps Days: Service Mesh Landscape
PHX DevOps Days: Service Mesh LandscapePHX DevOps Days: Service Mesh Landscape
PHX DevOps Days: Service Mesh LandscapeChristian Posta
 
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau
 
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.
 
DDD, CQRS and testing with ASP.Net MVC
DDD, CQRS and testing with ASP.Net MVCDDD, CQRS and testing with ASP.Net MVC
DDD, CQRS and testing with ASP.Net MVCAndy Butland
 
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...Rustem Feyzkhanov
 
Microservices for java architects it-symposium-2015-09-15
Microservices for java architects it-symposium-2015-09-15Microservices for java architects it-symposium-2015-09-15
Microservices for java architects it-symposium-2015-09-15Derek Ashmore
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disquszeeg
 
API World: The service-mesh landscape
API World: The service-mesh landscapeAPI World: The service-mesh landscape
API World: The service-mesh landscapeChristian Posta
 
Ratpack and Grails 3
Ratpack and Grails 3Ratpack and Grails 3
Ratpack and Grails 3Lari Hotari
 
Ratpack and Grails 3
 Ratpack and Grails 3 Ratpack and Grails 3
Ratpack and Grails 3GR8Conf
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataDataWorks Summit/Hadoop Summit
 
Serverless solution architecture in AWS
Serverless solution architecture in AWSServerless solution architecture in AWS
Serverless solution architecture in AWSRuncy Oommen
 

Similar to The hardest part of microservices: your data (20)

A microservices journey - Round 2
A microservices journey - Round 2A microservices journey - Round 2
A microservices journey - Round 2
 
Microservices with Apache Camel, Docker and Fabric8 v2
Microservices with Apache Camel, Docker and Fabric8 v2Microservices with Apache Camel, Docker and Fabric8 v2
Microservices with Apache Camel, Docker and Fabric8 v2
 
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservicesExploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
 
Advanced web application architecture Way2Web
Advanced web application architecture Way2WebAdvanced web application architecture Way2Web
Advanced web application architecture Way2Web
 
Microservices Journey Summer 2017
Microservices Journey Summer 2017Microservices Journey Summer 2017
Microservices Journey Summer 2017
 
Advanced web application architecture - Talk
Advanced web application architecture - TalkAdvanced web application architecture - Talk
Advanced web application architecture - Talk
 
The Hardest Part of Microservices: Calling Your Services
The Hardest Part of Microservices: Calling Your ServicesThe Hardest Part of Microservices: Calling Your Services
The Hardest Part of Microservices: Calling Your Services
 
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, Europe
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, EuropePatterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, Europe
Patterns of the Lambda Architecture -- 2015 April -- Hadoop Summit, Europe
 
PHX DevOps Days: Service Mesh Landscape
PHX DevOps Days: Service Mesh LandscapePHX DevOps Days: Service Mesh Landscape
PHX DevOps Days: Service Mesh Landscape
 
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
 
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudCase Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
 
DDD, CQRS and testing with ASP.Net MVC
DDD, CQRS and testing with ASP.Net MVCDDD, CQRS and testing with ASP.Net MVC
DDD, CQRS and testing with ASP.Net MVC
 
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
DataTalks.Club - Building Scalable End-to-End Deep Learning Pipelines in the ...
 
Microservices for java architects it-symposium-2015-09-15
Microservices for java architects it-symposium-2015-09-15Microservices for java architects it-symposium-2015-09-15
Microservices for java architects it-symposium-2015-09-15
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disqus
 
API World: The service-mesh landscape
API World: The service-mesh landscapeAPI World: The service-mesh landscape
API World: The service-mesh landscape
 
Ratpack and Grails 3
Ratpack and Grails 3Ratpack and Grails 3
Ratpack and Grails 3
 
Ratpack and Grails 3
 Ratpack and Grails 3 Ratpack and Grails 3
Ratpack and Grails 3
 
Apache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing dataApache Beam: A unified model for batch and stream processing data
Apache Beam: A unified model for batch and stream processing data
 
Serverless solution architecture in AWS
Serverless solution architecture in AWSServerless solution architecture in AWS
Serverless solution architecture in AWS
 

More from Christian Posta

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Understanding Wireguard, TLS and Workload Identity
Understanding Wireguard, TLS and Workload IdentityUnderstanding Wireguard, TLS and Workload Identity
Understanding Wireguard, TLS and Workload IdentityChristian Posta
 
Compliance and Zero Trust Ambient Mesh
Compliance and Zero Trust Ambient MeshCompliance and Zero Trust Ambient Mesh
Compliance and Zero Trust Ambient MeshChristian Posta
 
Cilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshCilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshChristian Posta
 
Multi-cluster service mesh with GlooMesh
Multi-cluster service mesh with GlooMeshMulti-cluster service mesh with GlooMesh
Multi-cluster service mesh with GlooMeshChristian Posta
 
Multicluster Kubernetes and Service Mesh Patterns
Multicluster Kubernetes and Service Mesh PatternsMulticluster Kubernetes and Service Mesh Patterns
Multicluster Kubernetes and Service Mesh PatternsChristian Posta
 
Cloud-Native Application Debugging with Envoy and Service Mesh
Cloud-Native Application Debugging with Envoy and Service MeshCloud-Native Application Debugging with Envoy and Service Mesh
Cloud-Native Application Debugging with Envoy and Service MeshChristian Posta
 
Kubernetes Ingress to Service Mesh (and beyond!)
Kubernetes Ingress to Service Mesh (and beyond!)Kubernetes Ingress to Service Mesh (and beyond!)
Kubernetes Ingress to Service Mesh (and beyond!)Christian Posta
 
The Truth About the Service Mesh Data Plane
The Truth About the Service Mesh Data PlaneThe Truth About the Service Mesh Data Plane
The Truth About the Service Mesh Data PlaneChristian Posta
 
Deep Dive: Building external auth plugins for Gloo Enterprise
Deep Dive: Building external auth plugins for Gloo EnterpriseDeep Dive: Building external auth plugins for Gloo Enterprise
Deep Dive: Building external auth plugins for Gloo EnterpriseChristian Posta
 
Role of edge gateways in relation to service mesh adoption
Role of edge gateways in relation to service mesh adoptionRole of edge gateways in relation to service mesh adoption
Role of edge gateways in relation to service mesh adoptionChristian Posta
 
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdNavigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdChristian Posta
 
Chaos Debugging for Microservices
Chaos Debugging for MicroservicesChaos Debugging for Microservices
Chaos Debugging for MicroservicesChristian Posta
 
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...Christian Posta
 
Service-mesh options with Linkerd, Consul, Istio and AWS AppMesh
Service-mesh options with Linkerd, Consul, Istio and AWS AppMeshService-mesh options with Linkerd, Consul, Istio and AWS AppMesh
Service-mesh options with Linkerd, Consul, Istio and AWS AppMeshChristian Posta
 
Intro Istio and what's new Istio 1.1
Intro Istio and what's new Istio 1.1Intro Istio and what's new Istio 1.1
Intro Istio and what's new Istio 1.1Christian Posta
 
API Gateways are going through an identity crisis
API Gateways are going through an identity crisisAPI Gateways are going through an identity crisis
API Gateways are going through an identity crisisChristian Posta
 
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...Christian Posta
 
Making sense of microservices, service mesh, and serverless
Making sense of microservices, service mesh, and serverlessMaking sense of microservices, service mesh, and serverless
Making sense of microservices, service mesh, and serverlessChristian Posta
 

More from Christian Posta (20)

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Understanding Wireguard, TLS and Workload Identity
Understanding Wireguard, TLS and Workload IdentityUnderstanding Wireguard, TLS and Workload Identity
Understanding Wireguard, TLS and Workload Identity
 
Compliance and Zero Trust Ambient Mesh
Compliance and Zero Trust Ambient MeshCompliance and Zero Trust Ambient Mesh
Compliance and Zero Trust Ambient Mesh
 
Cilium + Istio with Gloo Mesh
Cilium + Istio with Gloo MeshCilium + Istio with Gloo Mesh
Cilium + Istio with Gloo Mesh
 
Multi-cluster service mesh with GlooMesh
Multi-cluster service mesh with GlooMeshMulti-cluster service mesh with GlooMesh
Multi-cluster service mesh with GlooMesh
 
Multicluster Kubernetes and Service Mesh Patterns
Multicluster Kubernetes and Service Mesh PatternsMulticluster Kubernetes and Service Mesh Patterns
Multicluster Kubernetes and Service Mesh Patterns
 
Cloud-Native Application Debugging with Envoy and Service Mesh
Cloud-Native Application Debugging with Envoy and Service MeshCloud-Native Application Debugging with Envoy and Service Mesh
Cloud-Native Application Debugging with Envoy and Service Mesh
 
Kubernetes Ingress to Service Mesh (and beyond!)
Kubernetes Ingress to Service Mesh (and beyond!)Kubernetes Ingress to Service Mesh (and beyond!)
Kubernetes Ingress to Service Mesh (and beyond!)
 
The Truth About the Service Mesh Data Plane
The Truth About the Service Mesh Data PlaneThe Truth About the Service Mesh Data Plane
The Truth About the Service Mesh Data Plane
 
Deep Dive: Building external auth plugins for Gloo Enterprise
Deep Dive: Building external auth plugins for Gloo EnterpriseDeep Dive: Building external auth plugins for Gloo Enterprise
Deep Dive: Building external auth plugins for Gloo Enterprise
 
Role of edge gateways in relation to service mesh adoption
Role of edge gateways in relation to service mesh adoptionRole of edge gateways in relation to service mesh adoption
Role of edge gateways in relation to service mesh adoption
 
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdNavigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
 
Chaos Debugging for Microservices
Chaos Debugging for MicroservicesChaos Debugging for Microservices
Chaos Debugging for Microservices
 
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...
Leveraging Envoy Proxy and GraphQL to Lower the Risk of Monolith to Microserv...
 
Service-mesh options with Linkerd, Consul, Istio and AWS AppMesh
Service-mesh options with Linkerd, Consul, Istio and AWS AppMeshService-mesh options with Linkerd, Consul, Istio and AWS AppMesh
Service-mesh options with Linkerd, Consul, Istio and AWS AppMesh
 
Intro Istio and what's new Istio 1.1
Intro Istio and what's new Istio 1.1Intro Istio and what's new Istio 1.1
Intro Istio and what's new Istio 1.1
 
API Gateways are going through an identity crisis
API Gateways are going through an identity crisisAPI Gateways are going through an identity crisis
API Gateways are going through an identity crisis
 
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...
KubeCon NA 2018: Evolution of Integration and Microservices with Service Mesh...
 
Intro to Knative
Intro to KnativeIntro to Knative
Intro to Knative
 
Making sense of microservices, service mesh, and serverless
Making sense of microservices, service mesh, and serverlessMaking sense of microservices, service mesh, and serverless
Making sense of microservices, service mesh, and serverless
 

Recently uploaded

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 

Recently uploaded (20)

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 

The hardest part of microservices: your data

Editor's Notes

  1. Speed!!!.... As in performance? Or scale? What is this speed thing all about? This is a very different way of thinking about IT. Typically IT is optimized for Cost. Many parts of the business are. We’re not product companies anymore…. IT was traditionally used to transform otherwise paper processes or manual processes. And to support things like CRM, Accounting, Procurement, etc. Internally supporting. But now companies are using IT to deliver value through services. In fact, startups, are finding out to deliver value through digital channels and are quickly disrupting old guard enterprise corporations. We are service companies. Services require bi-direction/omni-directional interactions, communication with our customers. Creating value is done with customers. The faster you can get things to market the faster you can see what works and what doesn’t. We don’t know what will work up front. We don’t know what will deliver business value up front. We need to discover it. What we want is to build an organization that’s able to experiment, fail fast, and iterate on what does work. We basically want IT to drive outcomes that deliver business value. And we want to go fast.
  2. The discovery of what’s important, and the experimentation process leads us to want to find business value. We want to quickly find out the things that don’t work and minimize the cost it takes to do these experiments. This transformation is a process, not something that happens over night, and not something you can copy. You’ll even note that each organization is different in how it can go about this process; each needs to balance speed, safety, and business value for itself.
  3. Get back to first principles. Focus on principles, patterns, methodologies. Tools will help, but you cannot start with tools.
  4. Autonomy….
  5. What is it? Who defines it? Who owns that definition? Who owns the instances of it. How do I get it? How do I not miss something? And if I do solve these questions what does the architecture look like? Bunch of point to point connections? Lots of big up front design? Lots of contracts and governance? These things tend to break autonomy. Let’s explore this a bit and see what problems we run into with data in a microservices world.
  6. Now, understanding the domain, understanding the data model, an understanding where the boundaries is complex stuff. It cannot be solved with technology alone. Let me give you a simple example…. This seems like a simple, even absurd question. It’s really not. This one simple question can illustrate how ambiguous contradictory our language is with respect to understanding “real life” We cannot understand how to store a representation of a perceived “real life” unless we can describe it in plain language without ambiguity. What is a book and how do we represent it? We need to first understand what “one book is.” If an author has written two books we may expect to see two “book” entries represented in some kind of editor database or bibliographic database as “two records”. If they’ve written two editions of the one book, does it appear multiple times? Or do we model that as a revision record? Or maybe each edition gets its own record? If a library or bookstore has 5 copies each of two books, do they record that as books? Is a book really just a copy of a book? Or do we call it a copy? But maybe a library inventory system may just refer to copies as books as well for the purposes of counting the total number of physical books. So we could come up with “copies” and “books” but they can be used interchangeably depending on who’s asking. Or maybe for some systems a Book is something with a hard cover because they want to exclude periodicals, magazines, ebooks? So a “manual” may be classified as a book in some contexts, but not others. Or maybe a book is just a bounded physical unit? But some novels are so long they are actually broken down into two physical elements. Maybe labeled Volume I and Volume II. So then are those separate books or one Book? Or the opposite; maybe multiple novel compositions are bounded together into a single physical unit; but really they are individual works. So we could have a system where the author has written one book, it’s broken into two phsycial volumes, also known as books, and each volume has 5 copies each for a total of ten books. So what is one book? It gets incredibly confusing. So now just try to wrap your head around a Customer, or Patient, or Account, etc. The same polysemes exist there, only far more convoluted and ambiguous. And now when we talk about microservices we talk about the big ball of mud and how we cannot change part of it without re-deploying others. That is the easy part. Reconciling all of the different implicit usages of domain models across multiple contexts slammed into a single application is the hard part.
  7. A is the book checkout system -- book is a physical copy (second edition, volume I, II, etc all individual books) B is the book search system – book can be individual works where a composition may be multiple books and volumes I and II are all the same book C is the checkout reporting engine – a book is what A thinks is a book D is a recommendation engine – a book isn’t even a book, it’s an “interest” which has a mapping between books Book recommendations (D) D also wants to consume messages from A. But the things we need to do in our service is sufficiently different that we want to change the language. A and B have a translation and are coordinating, D is not coordinating and will build an AC that will do the translation. And it’s nobody else’s business how this works. In this case, maybe we have a Book recommendation engine that also reads what gets checked out and whom. Maybe D has some more complicated models it uses for describing recommendations. It wants to use A’s data, but it doesn’t want to conform to A’s domain model. It builds an Anti-Corruption layer to keep its model pure and that can do the translation between its and A’s models.
  8. An order. A customer. An account. A return. A claim. A discount?
  9. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  10. We store our data inside this thing…
  11. We store our data inside this thing… Do we really, as developers, understand how to properly use this thing?
  12. Put it all into one big database… No, seriously.. Just do this for your applications. You’ll save yourself a lot of trouble. Focus on
  13. Traditional Databases have tremendous flexibiluty, safety, etc.
  14. Traditional Databases have tremendous flexibiluty, safety, etc.
  15. Traditional Databases have tremendous flexibiluty, safety, etc.
  16. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  17. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  18. Business Agility!!! Journey … Understand them Test them Change them Different pace – rate of change is key !!! Agile business… Before going to far, we should have a definition about what microservices are. When we talk about microservices, we talk about breaking up complicated, potentially really large systems, whatever they may be, into smaller components. We break them into smaller components so we can understand them individually, test them individually, scale them and ultimately change them at a different pace than the rest of the system. You can imagine having to please every master in a monoithic environment can slow/bogg things down and inhibit change. Which as we discussed in the beginning is the key here. We need to be able to work on systems that can change with the rest of the business as it’s getting even more competitive and disruptive. One of the keys to this flexibility and ability to change is to focus on autonomy. Systems should be designed to be more autonomous so that changes don’t affect other downstream systems, faults don’t ripple across into cascading failures etc. The more dependencies we have (on other systems, protocols, shared libraries, databases, etc) the harder it can be to make changes. So we talk about services having and owning their own data, chosing the right technology for their function, and conciously enforcing modularity through APIs and contracts. Autonomy is key here But autonomoy of systems includes autonomoy of teams as well. Microservices can be a means to an end for a company serious about investing into their digital experiecne they provide to customers. It’s not in and of itself the end goal. It’s part of a digital transformation that encompasses all parts of the organization.
  19. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  20. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  21. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  22. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  23. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  24. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  25. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  26. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  27. One large database! We should focus on how we design our data models so that they can be sharded and distributed…. Focus on transactions, etc not 2PC
  28. This concept of defining language, developing models to describe a domain, implementing those models, enforcing assertions, etc all happen within a certain context, and that context is vitally important in software. In common language, we are smart enough to resolve these types of language conflicts within a sentence because of its context. The computer doesn’t have this context. We have to make it explicit. And any context needs to have explicit boundaries. This model needs to be “useful” ie, it should be able to be implemented. Try to establish a model that’s both useful for discussion with the domain experts and is implementable. There are infinite ways to model/think about something. Balance both masters with the model you choose. Large complex domains may need multiple models. And really the only way to understand a language and model is within a certain context. That context should have boundaries so it doesn’t bleed or force others to bleed definitions and semantics. Bounded context: within this space, this is the context of the language. This is what it means and it’s not ambiguous. Central thing about a model is the language you create to express the prblem and solution very crisply. Need clear language and need boundaries. Anti corruption layers are translations between the different models that may exist in multiple bounded contexts. They keep an internal model consistent and pure without bleeding across the boundaries. Bounded contexts tend to be “self contained systems” themselves with a complete vertical stack of the software including UI, business logic, data models, and database. They tend to not share databases across multiple models.
  29. In this scenario we may have established our boundaries… our customer profile service has taken an update to customer preferences. A customer and its profile/preferences may be modeled in other services like our recommendation engine, our master customer SOR, our social alerting engine, etc. And we need to update some important information... So the systems that are interested in this data must first be defined and implemented in code ahead of time. Adding a new system requires changes. Additionally, these downstream systems are not transactional... So if there are errors somewhere, then it’s up to the application try and decide what action to take... And while deciding that action, the application could fail.. And no state is stored about where it left off.. And now we’re in an inconsistent state.
  30. You could try adding compensation logic and stateful tracking of this locally.. And it’s also great practice to implement idempotent consumers.. The problem with is there could be “read uncommitted” issues like dirty reads or dirty writes that happen downstream because of this, and a compensation now gets much more complicated.
  31. We could just try emmitting events and say “whenever this happens over here, we just update a message queue”… now we have to try get consensus between the two systems. This can be expensive, as consensus tends to be, and you also suffer from availability issues..2PC is an anti-availability protocol.
  32. 2PC is perfectly fine when consensus is required, though have to consider the drawbacks. 2PC requires operational overhead to manage the TX Log of the tx manager. Also, you can run into issues with deadlocks when holding locks too long. You can also end up in heuristic situations where one side unilaterally rollsback. Now you need human intervention and reconcilliation logic. People poopoo 2PC, but it may be appropriate in some situations..
  33. Another situation that will tend to come up is identifying boundaries around IO and read/write patterns. How do we get the writes over to the read database? Do we do 2PC from the application? Do we use a message queue?
  34. What about the so called N+1 problem? Where we interact with downstream services, or maybe we take on events and need to enrich them with additional metadata. For example, we may want to group and sort a set of customers that fall into a certain criteria for specific recommendations, and we need to enrich the customer with additional preferences.. So we query for the customer list and then we loop through and enrich each customer.. Can downstream systems sustain this kind of rapid invocation? If they can, are you exposed at all to udnerlying storage incnsistenecies and concurrency issues? Do they just try create bulk APIs? And those APIs are inconsistent across providers (pagination, missed processing of singular elements, etc)
  35. So maybe we set up a cache in front of the service to alleviate the penalties of calling downstream services rapidly… and now what sort of stale data can you deal with? Bounded staleness? How do you handle cache eviction?
  36. We expect some levels of consistency and we need to be able to withstand faults because we know faults occur… But brewer says we can only pick 2 out of consistency, availability, partition tolerance...
  37. Consistency models… the set of allowable histories of operations We say that we read what we wrote
  38. Now, a process is allowed to read the most recently written value from any process, not just itself. The register becomes a place of coordination between two processes; they share state. We relax our model and say when we read, we read the value at the time of the read and take into account other processes writes…
  39. Now, a process is allowed to read the most recently written value from any process, not just itself. The register becomes a place of coordination between two processes; they share state. We relax our model and say when we read, we read the value at the time of the read and take into account other processes writes…
  40. Somewhere (or some node… a database, a service, a set of databases in a cluster) where there is an appearance of an order that is immediately visible to everyone viewing it. Moreover, linearizability’s time bounds guarantee that those changes will be visible to other participants after the operation completes. Hence, linearizability prohibits stale reads
  41. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  42. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  43. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  44. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  45. Score keeper – needs to read most up to date version of the score… cannot do an eventually consistent read (bounded staleness, consistent prefix, monotonic read).. BUT could do a read my writes read Umpire needs to do a strict consistent read to determine at the 9th inning or any afterward whether he can end the game
  46. Show a diagram of consistency related to performance..
  47. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  48. Show a diagram of consistency related to performance..
  49. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  50. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  51. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  52. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.
  53. Different users will see my message at different times–but each user will see my operations in order. Once seen, a post shouldn’t disappear.