SlideShare ist ein Scribd-Unternehmen logo
1 von 48
Downloaden Sie, um offline zu lesen
Design Patterns for Distributed
  Non-Relational Databases
                  aka
 Just Enough Distributed Systems To Be
               Dangerous
            (in 40 minutes)


             Todd Lipcon
              (@tlipcon)
                 Cloudera


            June 11, 2009
Introduction
Common Underlying Assumptions
Design Patterns
   Consistent Hashing
   Consistency Models
   Data Models
   Storage Layouts
   Log-Structured Merge Trees
Cluster Management
   Omniscient Master
   Gossip
Questions to Ask Presenters
Why We’re All Here


     Scaling up doesn’t work
     Scaling out with traditional RDBMSs isn’t so
     hot either
         Sharding scales, but you lose all the features that
         make RDBMSs useful!
         Sharding is operationally obnoxious.
     If we don’t need relational features, we want a
     distributed NRDBMS.
Closed-source NRDBMSs
“The Inspiration”




           Google BigTable
                    Applications: webtable, Reader, Maps, Blogger,
                    etc.
           Amazon Dynamo
                    Shopping Cart, ?
           Yahoo! PNUTS
                    Applications: ?
Data Interfaces
“This is the NOSQL meetup, right?”




          Every row has a key (PK)
          Key/value get/put
          multiget/multiput
          Range scan? With predicate pushdown?
          MapReduce?
          SQL?
Underlying Assumptions
Assumptions - Data Size


      The data does not fit on one node.
      The data may not fit on one rack.
      SANs are too expensive.

  Conclusion:
  The system must partition its data across many
  nodes.
Assumptions - Reliability
      The system must be highly available to serve
      web (and other) applications.
      Since the system runs on many nodes, nodes
      will crash during normal operation.
      Data must be safe even though disks and
      nodes will fail.

  Conclusion:
  The system must replicate each row to multiple
  nodes and remain available despite certain node and
  disk failure.
Assumptions - Performance
...and price thereof


            All systems we’re talking about today are
            meant for real-time use.
            95th or 99th percentile is more important than
            average latency
            Commodity hardware and slow disks.

     Conclusion:
     The system needs to perform well on commodity
     hardware, and maintain low latency even during
     recovery operations.
Design Patterns
Partitioning Schemes
“Where does a key live?”




          Given a key, we need to determine which
          node(s) it belongs on.
          If that node is down, we need to find another
          copy elsewhere.
          Difficulties:
                 Unbounded number of keys.
                 Dynamic cluster membership.
                 Node failures.
Consistent Hashing
Maintaining hashing in a dynamic cluster
Consistent Hashing
Key Placement
Consistency Models

     A consistency model determines rules for
     visibility and apparent order of updates.
     Example:
         Row X is replicated on nodes M and N
         Client A writes row X to node N
         Some period of time t elapses.
         Client B reads row X from node M
         Does client B see the write from client A?
     Consistency is a continuum with tradeoffs
Strict Consistency

     All read operations must return the data from
     the latest completed write operation, regardless
     of which replica the operations went to
     Implies either:
         All operations for a given row go to the same node
         (replication for availability)
         or nodes employ some kind of distributed
         transaction protocol (eg 2 Phase Commit or Paxos)
     CAP Theorem: Strict Consistency can’t be
     achieved at the same time as availability and
     partition-tolerance.
Eventual Consistency

     As t → ∞, readers will see writes.
     In a steady state, the system is guaranteed to
     eventually return the last written value
     For example: DNS, or MySQL Slave
     Replication (log shipping)
     Special cases of eventual consistency:
         Read-your-own-writes consistency (“sent mail”
         box)
         Causal consistency (if you write Y after reading X,
         anyone who reads Y sees X)
         gmail has RYOW but not causal!
Timestamps and Vector Clocks
Determining a history of a row



           Eventual consistency relies on deciding what
           value a row will eventually converge to
           In the case of two writers writing at “the
           same” time, this is difficult
           Timestamps are one solution, but rely on
           synchronized clocks and don’t capture causality
           Vector clocks are an alternative method of
           capturing order in a distributed system
Vector Clocks

     Definition:
         A vector clock is a tuple {t1 , t2 , ..., tn } of clock
         values from each node
         v1 < v2 if:
               For all i, v1i ≤ v2i
               For at least one i, v1i < v2i
         v1 < v2 implies global time ordering of events
     When data is written from node i, it sets ti to
     its clock value.
     This allows eventual consistency to resolve
     consistency between writes on multiple replicas.
Data Models
What’s in a row?




          Primary Key → Value
          Value could be:
                   Blob
                   Structured (set of columns)
                   Semi-structured (set of column families with
                   arbitrary columns, eg linkto:<url> in webtable)
                   Each has advantages and disadvantages
          Secondary Indexes? Tables/namespaces?
Multi-Version Storage
Using Timestamps for a 3rd dimension



          Each table cell has a timestamp
          Timestamps don’t necessarily need to
          correspond to real life
          Multiple versions (and tombstones) can exist
          concurrently for a given row
          Reads may return “most recent”, “most recent
          before T”, etc. (free snapshots)
          System may provide optimistic concurrency
          control with compare-and-swap on timestamps
Storage Layouts
How do we lay out rows and columns on disk?




          Determines performance of different access
          patterns
          Storage layout maps directly to disk access
          patterns
          Fast writes? Fast reads? Fast scans?
          Whole-row access or subsets of columns?
Row-based Storage




     Pros:
         Good locality of access (on disk and in cache) of
         different columns
         Read/write of a single row is a single IO operation.
     Cons:
         But if you want to scan only one column, you still
         read all.
Columnar Storage




     Pros:
         Data for a given column is stored sequentially
         Scanning a single column (eg aggregate queries) is
         fast
     Cons:
         Reading a single row may seek once per column.
Columnar Storage with Locality Groups




     Columns are organized into families (“locality
     groups”)
     Benefits of row-based layout within a group.
     Benefits of column-based - don’t have to read
     groups you don’t care about.
Log Structured Merge Trees
aka “The BigTable model”

           Random IO for writes is bad (and impossible in
           some DFSs)
           LSM Trees convert random writes to sequential
           writes
           Writes go to a commit log and in-memory
           storage (Memtable)
           The Memtable is occasionally flushed to disk
           (SSTable)
           The disk stores are periodically compacted
    P. E. O’Neil, E. Cheng, D. Gawlick, and E. J. O’Neil. The log-structured merge-tree

    (LSM-tree). Acta Informatica. 1996.
LSM Data Layout
LSM Write Path
LSM Read Path
LSM Read Path + Bloom Filters
LSM Memtable Flush
LSM Compaction
Cluster Management

     Clients need to know where to find data
     (consistent hashing tokens, etc)
     Internal nodes may need to find each other as
     well
     Since nodes may fail and recover, a
     configuration file doesn’t really suffice
     We need a way of keeping some kind of
     consistent view of the cluster state
Omniscient Master

     When nodes join/leave or change state, they
     talk to a master
     That master holds the authoritative view of the
     world
     Pros: simplicity, single consistent view of the
     cluster
     Cons: potential SPOF unless master is made
     highly available. Not partition-tolerant.
Gossip
     Gossip is one method to propagate a view of
     cluster status.
     Every t seconds, on each node:
         The node selects some other node to chat with.
         The node reconciles its view of the cluster with its
         gossip buddy.
         Each node maintains a “timestamp” for itself and
         for the most recent information it has from every
         other node.
     Information about cluster state spreads in
     O(lgn) rounds (eventual consistency)
     Scalable and no SPOF, but state is only
     eventually consistent
Gossip - Initial State
Gossip - Round 1
Gossip - Round 2
Gossip - Round 3
Gossip - Round 4
Questions to Ask Presenters
Scalability and Reliability

      What are the scaling bottlenecks? How does it
      react when overloaded?
      Are there any single points of failure?
      When nodes fail, does the system maintain
      availability of all data?
      Does the system automatically re-replicate
      when replicas are lost?
      When new nodes are added, does the system
      automatically rebalance data?
Performance

     What’s the goal? Batch throughput or request
     latency?
     How many seeks for reads? For writes? How
     many net RTTs?
     What 99th percentile latencies have been
     measured in practice?
     How do failures impact serving latencies?
     What throughput has been measured in
     practice for bulk loads?
Consistency

     What consistency model does the system
     provide?
     What situations would cause a lapse of
     consistency, if any?
     Can consistency semantics be tweaked by
     configuration settings?
     Is there a way to do compare-and-swap on row
     contents for optimistic locking? Multirow?
Cluster Management and Topology

     Does the system have a single master? Does it
     use gossip to spread cluster management data?
     Can it withstand network partitions and still
     provide some level of service?
     Can it be deployed across multiple datacenters
     for disaster recovery?
     Can nodes be commissioned/decomissioned
     automatically without downtime?
     Operational hooks for monitoring and metrics?
Data Model and Storage

     What data model and storage system does the
     system provide?
     Is it pluggable?
     What IO patterns does the system cause under
     different workloads?
     Is the system best at random or sequential
     access? For read-mostly or write-mostly?
     Are there practical limits on key, value, or row
     sizes?
     Is compression available?
Data Access Methods


     What methods exist for accessing data? Can I
     access it from language X?
     Is there a way to perform filtering or selection
     at the server side?
     Are there bulk load tools to get data in/out
     efficiently?
     Is there a provision for data backup/restore?
Real Life Considerations
(I was talking about fake life in the first 45 slides)

            Who uses this system? How big are the
            clusters it’s deployed on, and what kind of load
            do they handle?
            Who develops this system? Is this a community
            project or run by a single organization? Are
            outside contributions regularly accepted?
            Who supports this system? Is there an active
            community who will help me deploy it and
            debug issues? Docs?
            What is the open source license?
            What is the development roadmap?
Questions?
http://cloudera-todd.s3.amazonaws.com/nosql.pdf

Weitere ähnliche Inhalte

Was ist angesagt?

An Introduction to Big Data, NoSQL and MongoDB
An Introduction to Big Data, NoSQL and MongoDBAn Introduction to Big Data, NoSQL and MongoDB
An Introduction to Big Data, NoSQL and MongoDB
William LaForest
 
An overview of Neo4j Internals
An overview of Neo4j InternalsAn overview of Neo4j Internals
An overview of Neo4j Internals
Tobias Lindaaker
 
Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
Chandler Huang
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 

Was ist angesagt? (20)

Presto: SQL-on-anything
Presto: SQL-on-anythingPresto: SQL-on-anything
Presto: SQL-on-anything
 
[EPPG] Oracle to PostgreSQL, Challenges to Opportunity
[EPPG] Oracle to PostgreSQL, Challenges to Opportunity[EPPG] Oracle to PostgreSQL, Challenges to Opportunity
[EPPG] Oracle to PostgreSQL, Challenges to Opportunity
 
An Introduction to Big Data, NoSQL and MongoDB
An Introduction to Big Data, NoSQL and MongoDBAn Introduction to Big Data, NoSQL and MongoDB
An Introduction to Big Data, NoSQL and MongoDB
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
 
Understanding and Improving Code Generation
Understanding and Improving Code GenerationUnderstanding and Improving Code Generation
Understanding and Improving Code Generation
 
Hive: Loading Data
Hive: Loading DataHive: Loading Data
Hive: Loading Data
 
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsSpark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka Streams
 
7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth
 
An overview of Neo4j Internals
An overview of Neo4j InternalsAn overview of Neo4j Internals
An overview of Neo4j Internals
 
Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
 
Introduction to MongoDB.pptx
Introduction to MongoDB.pptxIntroduction to MongoDB.pptx
Introduction to MongoDB.pptx
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL Database
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
 
A Technical Introduction to WiredTiger
A Technical Introduction to WiredTigerA Technical Introduction to WiredTiger
A Technical Introduction to WiredTiger
 
RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Resource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache StormResource Aware Scheduling in Apache Storm
Resource Aware Scheduling in Apache Storm
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 
Presto
PrestoPresto
Presto
 
NoSQL databases - An introduction
NoSQL databases - An introductionNoSQL databases - An introduction
NoSQL databases - An introduction
 

Andere mochten auch

Modelando aplicação em documento - MongoDB
Modelando aplicação em documento - MongoDBModelando aplicação em documento - MongoDB
Modelando aplicação em documento - MongoDB
Thiago Avelino
 
Messaging With Apache ActiveMQ
Messaging With Apache ActiveMQMessaging With Apache ActiveMQ
Messaging With Apache ActiveMQ
Bruce Snyder
 
ActiveMQ In Action
ActiveMQ In ActionActiveMQ In Action
ActiveMQ In Action
Bruce Snyder
 
Apache ActiveMQ - Enterprise messaging in action
Apache ActiveMQ - Enterprise messaging in actionApache ActiveMQ - Enterprise messaging in action
Apache ActiveMQ - Enterprise messaging in action
dejanb
 
VoltDB and Erlang - Tech planet 2012
VoltDB and Erlang - Tech planet 2012VoltDB and Erlang - Tech planet 2012
VoltDB and Erlang - Tech planet 2012
Eonblast
 
LOAD BALANCING ALGORITHMS
LOAD BALANCING ALGORITHMSLOAD BALANCING ALGORITHMS
LOAD BALANCING ALGORITHMS
tanmayshah95
 
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
Alexandre Morgaut
 

Andere mochten auch (20)

Scalable Databases - From Relational Databases To Polyglot Persistence
Scalable Databases - From Relational Databases To Polyglot PersistenceScalable Databases - From Relational Databases To Polyglot Persistence
Scalable Databases - From Relational Databases To Polyglot Persistence
 
Why Projects Fail: Obstacles and Solutions
Why Projects Fail: Obstacles and SolutionsWhy Projects Fail: Obstacles and Solutions
Why Projects Fail: Obstacles and Solutions
 
Scaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approachScaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approach
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
 
Server load balancer ppt
Server load balancer pptServer load balancer ppt
Server load balancer ppt
 
Developing web apps using Erlang-Web
Developing web apps using Erlang-WebDeveloping web apps using Erlang-Web
Developing web apps using Erlang-Web
 
Node.js and NoSQL
Node.js and NoSQLNode.js and NoSQL
Node.js and NoSQL
 
Elasticsearch - Zero to Hero
Elasticsearch - Zero to HeroElasticsearch - Zero to Hero
Elasticsearch - Zero to Hero
 
Non relational databases-no sql
Non relational databases-no sqlNon relational databases-no sql
Non relational databases-no sql
 
Modelando aplicação em documento - MongoDB
Modelando aplicação em documento - MongoDBModelando aplicação em documento - MongoDB
Modelando aplicação em documento - MongoDB
 
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems GOOGLE: Designs, Lessons and Advice from Building Large   Distributed Systems
GOOGLE: Designs, Lessons and Advice from Building Large Distributed Systems
 
Messaging With Apache ActiveMQ
Messaging With Apache ActiveMQMessaging With Apache ActiveMQ
Messaging With Apache ActiveMQ
 
ActiveMQ In Action
ActiveMQ In ActionActiveMQ In Action
ActiveMQ In Action
 
Apache ActiveMQ - Enterprise messaging in action
Apache ActiveMQ - Enterprise messaging in actionApache ActiveMQ - Enterprise messaging in action
Apache ActiveMQ - Enterprise messaging in action
 
Relational vs. Non-Relational
Relational vs. Non-RelationalRelational vs. Non-Relational
Relational vs. Non-Relational
 
VoltDB and Erlang - Tech planet 2012
VoltDB and Erlang - Tech planet 2012VoltDB and Erlang - Tech planet 2012
VoltDB and Erlang - Tech planet 2012
 
LOAD BALANCING ALGORITHMS
LOAD BALANCING ALGORITHMSLOAD BALANCING ALGORITHMS
LOAD BALANCING ALGORITHMS
 
Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)
Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)
Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)
 
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
Wakanda: NoSQL for Model-Driven Web applications - NoSQL matters 2012
 
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...
The Missing Manual for Leveled Compaction Strategy (Wei Deng & Ryan Svihla, D...
 

Ähnlich wie Design Patterns for Distributed Non-Relational Databases

Design Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databasesDesign Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databases
lovingprince58
 
Lecture-04-Principles of data management.pdf
Lecture-04-Principles of data management.pdfLecture-04-Principles of data management.pdf
Lecture-04-Principles of data management.pdf
manimozhi98
 
Distributed Systems: scalability and high availability
Distributed Systems: scalability and high availabilityDistributed Systems: scalability and high availability
Distributed Systems: scalability and high availability
Renato Lucindo
 
Spinnaker VLDB 2011
Spinnaker VLDB 2011Spinnaker VLDB 2011
Spinnaker VLDB 2011
sandeep_tata
 
Distributed Algorithms
Distributed AlgorithmsDistributed Algorithms
Distributed Algorithms
913245857
 
Handling Data in Mega Scale Web Systems
Handling Data in Mega Scale Web SystemsHandling Data in Mega Scale Web Systems
Handling Data in Mega Scale Web Systems
Vineet Gupta
 

Ähnlich wie Design Patterns for Distributed Non-Relational Databases (20)

Design Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databasesDesign Patterns For Distributed NO-reational databases
Design Patterns For Distributed NO-reational databases
 
Basics of Distributed Systems - Distributed Storage
Basics of Distributed Systems - Distributed StorageBasics of Distributed Systems - Distributed Storage
Basics of Distributed Systems - Distributed Storage
 
Cassandra
CassandraCassandra
Cassandra
 
Lecture-04-Principles of data management.pdf
Lecture-04-Principles of data management.pdfLecture-04-Principles of data management.pdf
Lecture-04-Principles of data management.pdf
 
Schemaless Databases
Schemaless DatabasesSchemaless Databases
Schemaless Databases
 
NoSql Database
NoSql DatabaseNoSql Database
NoSql Database
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for Sysadmins
 
Cassandra & Python - Springfield MO User Group
Cassandra & Python - Springfield MO User GroupCassandra & Python - Springfield MO User Group
Cassandra & Python - Springfield MO User Group
 
Distributed Systems: scalability and high availability
Distributed Systems: scalability and high availabilityDistributed Systems: scalability and high availability
Distributed Systems: scalability and high availability
 
Software architecture for data applications
Software architecture for data applicationsSoftware architecture for data applications
Software architecture for data applications
 
Cassandra
CassandraCassandra
Cassandra
 
Data Engineering for Data Scientists
Data Engineering for Data Scientists Data Engineering for Data Scientists
Data Engineering for Data Scientists
 
MySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspectiveMySQL 5.7 clustering: The developer perspective
MySQL 5.7 clustering: The developer perspective
 
Spinnaker VLDB 2011
Spinnaker VLDB 2011Spinnaker VLDB 2011
Spinnaker VLDB 2011
 
Distributed systems and scalability rules
Distributed systems and scalability rulesDistributed systems and scalability rules
Distributed systems and scalability rules
 
Distributed Algorithms
Distributed AlgorithmsDistributed Algorithms
Distributed Algorithms
 
Everything you always wanted to know about Distributed databases, at devoxx l...
Everything you always wanted to know about Distributed databases, at devoxx l...Everything you always wanted to know about Distributed databases, at devoxx l...
Everything you always wanted to know about Distributed databases, at devoxx l...
 
Handling Data in Mega Scale Web Systems
Handling Data in Mega Scale Web SystemsHandling Data in Mega Scale Web Systems
Handling Data in Mega Scale Web Systems
 
No sql (not only sql)
No sql                 (not only sql)No sql                 (not only sql)
No sql (not only sql)
 
Highly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowakiHighly available distributed databases, how they work, javier ramirez at teowaki
Highly available distributed databases, how they work, javier ramirez at teowaki
 

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 

Design Patterns for Distributed Non-Relational Databases

  • 1. Design Patterns for Distributed Non-Relational Databases aka Just Enough Distributed Systems To Be Dangerous (in 40 minutes) Todd Lipcon (@tlipcon) Cloudera June 11, 2009
  • 2. Introduction Common Underlying Assumptions Design Patterns Consistent Hashing Consistency Models Data Models Storage Layouts Log-Structured Merge Trees Cluster Management Omniscient Master Gossip Questions to Ask Presenters
  • 3. Why We’re All Here Scaling up doesn’t work Scaling out with traditional RDBMSs isn’t so hot either Sharding scales, but you lose all the features that make RDBMSs useful! Sharding is operationally obnoxious. If we don’t need relational features, we want a distributed NRDBMS.
  • 4. Closed-source NRDBMSs “The Inspiration” Google BigTable Applications: webtable, Reader, Maps, Blogger, etc. Amazon Dynamo Shopping Cart, ? Yahoo! PNUTS Applications: ?
  • 5. Data Interfaces “This is the NOSQL meetup, right?” Every row has a key (PK) Key/value get/put multiget/multiput Range scan? With predicate pushdown? MapReduce? SQL?
  • 7. Assumptions - Data Size The data does not fit on one node. The data may not fit on one rack. SANs are too expensive. Conclusion: The system must partition its data across many nodes.
  • 8. Assumptions - Reliability The system must be highly available to serve web (and other) applications. Since the system runs on many nodes, nodes will crash during normal operation. Data must be safe even though disks and nodes will fail. Conclusion: The system must replicate each row to multiple nodes and remain available despite certain node and disk failure.
  • 9. Assumptions - Performance ...and price thereof All systems we’re talking about today are meant for real-time use. 95th or 99th percentile is more important than average latency Commodity hardware and slow disks. Conclusion: The system needs to perform well on commodity hardware, and maintain low latency even during recovery operations.
  • 11. Partitioning Schemes “Where does a key live?” Given a key, we need to determine which node(s) it belongs on. If that node is down, we need to find another copy elsewhere. Difficulties: Unbounded number of keys. Dynamic cluster membership. Node failures.
  • 14. Consistency Models A consistency model determines rules for visibility and apparent order of updates. Example: Row X is replicated on nodes M and N Client A writes row X to node N Some period of time t elapses. Client B reads row X from node M Does client B see the write from client A? Consistency is a continuum with tradeoffs
  • 15. Strict Consistency All read operations must return the data from the latest completed write operation, regardless of which replica the operations went to Implies either: All operations for a given row go to the same node (replication for availability) or nodes employ some kind of distributed transaction protocol (eg 2 Phase Commit or Paxos) CAP Theorem: Strict Consistency can’t be achieved at the same time as availability and partition-tolerance.
  • 16. Eventual Consistency As t → ∞, readers will see writes. In a steady state, the system is guaranteed to eventually return the last written value For example: DNS, or MySQL Slave Replication (log shipping) Special cases of eventual consistency: Read-your-own-writes consistency (“sent mail” box) Causal consistency (if you write Y after reading X, anyone who reads Y sees X) gmail has RYOW but not causal!
  • 17. Timestamps and Vector Clocks Determining a history of a row Eventual consistency relies on deciding what value a row will eventually converge to In the case of two writers writing at “the same” time, this is difficult Timestamps are one solution, but rely on synchronized clocks and don’t capture causality Vector clocks are an alternative method of capturing order in a distributed system
  • 18. Vector Clocks Definition: A vector clock is a tuple {t1 , t2 , ..., tn } of clock values from each node v1 < v2 if: For all i, v1i ≤ v2i For at least one i, v1i < v2i v1 < v2 implies global time ordering of events When data is written from node i, it sets ti to its clock value. This allows eventual consistency to resolve consistency between writes on multiple replicas.
  • 19. Data Models What’s in a row? Primary Key → Value Value could be: Blob Structured (set of columns) Semi-structured (set of column families with arbitrary columns, eg linkto:<url> in webtable) Each has advantages and disadvantages Secondary Indexes? Tables/namespaces?
  • 20. Multi-Version Storage Using Timestamps for a 3rd dimension Each table cell has a timestamp Timestamps don’t necessarily need to correspond to real life Multiple versions (and tombstones) can exist concurrently for a given row Reads may return “most recent”, “most recent before T”, etc. (free snapshots) System may provide optimistic concurrency control with compare-and-swap on timestamps
  • 21. Storage Layouts How do we lay out rows and columns on disk? Determines performance of different access patterns Storage layout maps directly to disk access patterns Fast writes? Fast reads? Fast scans? Whole-row access or subsets of columns?
  • 22. Row-based Storage Pros: Good locality of access (on disk and in cache) of different columns Read/write of a single row is a single IO operation. Cons: But if you want to scan only one column, you still read all.
  • 23. Columnar Storage Pros: Data for a given column is stored sequentially Scanning a single column (eg aggregate queries) is fast Cons: Reading a single row may seek once per column.
  • 24. Columnar Storage with Locality Groups Columns are organized into families (“locality groups”) Benefits of row-based layout within a group. Benefits of column-based - don’t have to read groups you don’t care about.
  • 25. Log Structured Merge Trees aka “The BigTable model” Random IO for writes is bad (and impossible in some DFSs) LSM Trees convert random writes to sequential writes Writes go to a commit log and in-memory storage (Memtable) The Memtable is occasionally flushed to disk (SSTable) The disk stores are periodically compacted P. E. O’Neil, E. Cheng, D. Gawlick, and E. J. O’Neil. The log-structured merge-tree (LSM-tree). Acta Informatica. 1996.
  • 29. LSM Read Path + Bloom Filters
  • 32. Cluster Management Clients need to know where to find data (consistent hashing tokens, etc) Internal nodes may need to find each other as well Since nodes may fail and recover, a configuration file doesn’t really suffice We need a way of keeping some kind of consistent view of the cluster state
  • 33. Omniscient Master When nodes join/leave or change state, they talk to a master That master holds the authoritative view of the world Pros: simplicity, single consistent view of the cluster Cons: potential SPOF unless master is made highly available. Not partition-tolerant.
  • 34. Gossip Gossip is one method to propagate a view of cluster status. Every t seconds, on each node: The node selects some other node to chat with. The node reconciles its view of the cluster with its gossip buddy. Each node maintains a “timestamp” for itself and for the most recent information it has from every other node. Information about cluster state spreads in O(lgn) rounds (eventual consistency) Scalable and no SPOF, but state is only eventually consistent
  • 40. Questions to Ask Presenters
  • 41. Scalability and Reliability What are the scaling bottlenecks? How does it react when overloaded? Are there any single points of failure? When nodes fail, does the system maintain availability of all data? Does the system automatically re-replicate when replicas are lost? When new nodes are added, does the system automatically rebalance data?
  • 42. Performance What’s the goal? Batch throughput or request latency? How many seeks for reads? For writes? How many net RTTs? What 99th percentile latencies have been measured in practice? How do failures impact serving latencies? What throughput has been measured in practice for bulk loads?
  • 43. Consistency What consistency model does the system provide? What situations would cause a lapse of consistency, if any? Can consistency semantics be tweaked by configuration settings? Is there a way to do compare-and-swap on row contents for optimistic locking? Multirow?
  • 44. Cluster Management and Topology Does the system have a single master? Does it use gossip to spread cluster management data? Can it withstand network partitions and still provide some level of service? Can it be deployed across multiple datacenters for disaster recovery? Can nodes be commissioned/decomissioned automatically without downtime? Operational hooks for monitoring and metrics?
  • 45. Data Model and Storage What data model and storage system does the system provide? Is it pluggable? What IO patterns does the system cause under different workloads? Is the system best at random or sequential access? For read-mostly or write-mostly? Are there practical limits on key, value, or row sizes? Is compression available?
  • 46. Data Access Methods What methods exist for accessing data? Can I access it from language X? Is there a way to perform filtering or selection at the server side? Are there bulk load tools to get data in/out efficiently? Is there a provision for data backup/restore?
  • 47. Real Life Considerations (I was talking about fake life in the first 45 slides) Who uses this system? How big are the clusters it’s deployed on, and what kind of load do they handle? Who develops this system? Is this a community project or run by a single organization? Are outside contributions regularly accepted? Who supports this system? Is there an active community who will help me deploy it and debug issues? Docs? What is the open source license? What is the development roadmap?