SlideShare ist ein Scribd-Unternehmen logo
1 von 56
Cassandra Performance and
    Scalability on AWS
           August 8th, 2012
           Adrian Cockcroft
      @adrianco #netflixcloud #cassandra12
   http://www.linkedin.com/in/adriancockcroft
Things we don’t do
Things we do do. Run benchmarks.
              Now.
YOLO
Screenshots from Live Demo

 Backup slides from pre-runs of the
 demo with some updates to show
     what actually happened
Asgard
cass_perf apps, with no instances running
Jenkins
Jenkins perf_test jobs
Jmeter Setup
Build parameters
Jmeter Setup
Build parameters
Jmeter Setup
Build parameters
Asgard
Iinitial set of cass instances up and running
Back to Presentation

While the load gets going….
Cassandra Performance and Scalability on AWS
Scalability from 48 to 288 nodes on AWS
 http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html


              Client Writes/s by node count – Replication Factor = 3
1200000
                                                                    1099837
1000000

 800000
                                                         Used 288 of m1.xlarge
 600000                                                  4 CPU, 15 GB RAM, 8 ECU
                                          537172         Cassandra 0.86
 400000                                                  Benchmark config only
                                366828                   existed for about 1hr
 200000               174373

      0
          0         50         100       150       200   250       300        350
Blah Blah                           Blah

 (I’m skipping all the cloud intro etc. Netflix
 runs in the cloud, if you hadn’t figured that
 out already you aren’t paying attention and
    should go read slideshare.net/netflix)
“Some people skate to the puck,
I skate to where the puck is going to be”
             Wayne Gretzky
Cassandra on AWS
The Past                     The Future
• Instance: m2.4xlarge       • Instance: hi1.4xlarge
• Storage: 2 drives, 1.7TB   • Storage: 2 SSD volumes, 2TB
• CPU: 8 Cores, 26 ECU       • CPU: 8 HT cores, 35 ECU
• RAM: 68GB                  • RAM: 64GB
• Network: 1Gbit             • Network: 10Gbit
• IOPS: ~500                 • IOPS: ~100,000
• Throughput: ~100Mbyte/s    • Throughput: ~1Gbyte/s
• Cost: $1.80/hr             • Cost: $3.10/hr
Cassandra Disk vs. SSD Benchmark
     Same Throughput, Lower Latency, Half Cost
Live Demo Workload
• Jenkins automation
   – Jmeter load driver
   – Asgard provisioning
   – Priam instance management
• Traffic
   – Reading/writing whole 100 column rows
   – Randomly selected from 25M row keys
   – Run for 10minutes, then double ring size
The Netflix Streaming Service
Major Front End Services
• Non-member Web Site www.netflix.com
    – Marketing driven, sign up flow, SOX/PCI scope

• Member Web Site movies.netflix.com
    – Personalization driven

• CDNs for delivering bulk video/audio
    – Netflix CDN: openconnect.netflix.com

• API for external and device user interfaces
    – Mostly private APIs, public API docs at developer.netflix.com

• API for controlling video playback
    – DRM, QoS management, Bookmarks
Netflix Deployed on AWS
 2009          2009           2010          2010        2010            2011

Content        Logs           Play          WWW          API              CS
  Content          S3                                                   International
                                 DRM         Sign-Up     Metadata
 Management     Terabytes                                                 CS lookup


    EC2                                                                 Diagnostics &
                  EMR         CDN routing     Search    Device Config
  Encoding                                                                 Actions


     S3                                       Movie      TV Movie       Customer Call
                Hive & Pig    Bookmarks
  Petabytes                                  Choosing    Choosing            Log


                Business                                   Social
                                Logging      Ratings                    CS Analytics
               Intelligence                              Facebook
  CDNs
   ISPs
 Terabits
Customers
Cassandra Instance Architecture

Linux Base AMI (CentOS)
     Priam
   Cassandra
   Manager
      Token
                    Java7
Management, Bac
kups, Autoscaling
 Tomcat/Java7       AppDynamics


 Monitoring
                      appagent
                     monitoring     Cassandra 1.09
 Log rotation
AppDynamics         GC and thread
machineagent        dump logging
     Etc.
Priam – Cassandra Automation
           Available at http://github.com/netflix

•   Netflix Platform Tomcat Code
•   Zero touch auto-configuration
•   State management for Cassandra JVM
•   Token allocation and assignment
•   Broken node auto-replacement
•   Full and incremental backup to S3
•   Restore sequencing from S3
•   Grow/Shrink Cassandra “ring”
Astyanax
               Available at http://github.com/netflix

• Features
  –   Complete abstraction of connection pool from RPC protocol
  –   Fluent Style API
  –   Operation retry with backoff
  –   Token aware
• Recipes
  –   Distribute row lock (without zookeeper)
  –   Multi-DC row lock
  –   Uniqueness constraint
  –   Multi-row uniqueness constraint
  –   Large file storage
Scale Up

Return to live demo to watch new
      nodes coming online
Kiklos
                Clusters growing from 12 to 24
   in-service, bootstrapping, garbage-collecting, cass-down


• http://explorers.us-east-
  1.dyntest.netflix.net:7001/jr/cassandradashbo
  ard
Kiklos
             Clusters growing from 12 to 24
in-service, bootstrapping, garbage-collecting, cass-down
Asgard

Showed 24 nodes per cluster, but
    didn’t get a screen shot
Back to Presentation

While jenkins/jmeter collects graphs
   and shuts down the systems
Cassandra on AWS

A highly available and durable
     deployment pattern
High Availability
• Cassandra stores 3 local copies, 1 per zone
  – Synchronous access, durable, highly available
  – Read/Write One fastest, use for fire and forget
  – Read/Write Quorum 2 of 3, use for read-after-write


• AWS Availability Zones
  – Separate buildings
  – Separate power etc.
  – Fairly close together
“Traditional” Cassandra Write Data Flows
        Single Region, Multiple Availability Zone, Not Token Aware

                                         Cassandra
                                         •Disks
                                        2•Zone A       2
                                             4 2
1. Client Writes to any   Cassandra 3                      3Cassandra   If a node goes
   Cassandra Node         •Disks5                           •Disks 5    offline, hinted handoff
2. Coordinator Node       •Zone C            1              •Zone B     completes the write
   replicates to nodes                                                  when the node comes
                                        Non Token
   and Zones                                                            back up.
3. Nodes return ack to                   Aware
   coordinator                           Clients                        Requests can choose to
4. Coordinator returns    Cassandra                         Cassandra   wait for one node, a
   ack to client          •Disks                            •Disks      quorum, or all nodes to
5. Data written to        •Zone B                           •Zone C     ack the write
   internal commit log                             3
   disk (no more than                    Cassandra                      SSTable disk writes and
   10 seconds later)                     •Disks    5                    compactions occur
                                         •Zone A
                                                                        asynchronously
Astyanax - Cassandra Write Data Flows
          Single Region, Multiple Availability Zone, Token Aware

                                       Cassandra
                                       •Disks
                                       •Zone A

1. Client Writes to      Cassandra 2                 2Cassandra   If a node goes
   nodes and Zones       •Disks3                      •Disks 3    offline, hinted handoff
2. Nodes return ack to   •Zone C           1          •Zone B     completes the write
   client                              Token                      when the node comes
3. Data written to                                                back up.
   internal commit log                 Aware
   disks (no more than                 Clients                    Requests can choose to
   10 seconds later)     Cassandra                    Cassandra   wait for one node, a
                         •Disks                       •Disks      quorum, or all nodes to
                         •Zone B                      •Zone C     ack the write
                                          2
                                       Cassandra                  SSTable disk writes and
                                       •Disks    3                compactions occur
                                       •Zone A
                                                                  asynchronously
Data Flows for Multi-Region Writes
          Token Aware, Consistency Level = Local Quorum

1. Client writes to local replicas                   If a node or region goes offline, hinted handoff
2. Local write acks returned to                      completes the write when the node comes back up.
   Client which continues when                       Nightly global compare and repair jobs ensure
   2 of 3 local nodes are                            everything stays consistent.
   committed
3. Local coordinator writes to
   remote coordinator.                                                       100+ms latency
                                                      Cassandra                                       Cassandra
4. When data arrives, remote                          • Disks
                                                      • Zone A
                                                                                                      • Disks
                                                                                                      • Zone A

   coordinator node acks and         Cassandra   2                2
                                                                  Cassandra              Cassandra                4Cassandra
                                          6
                                     • Disks                      • Disks 6 3           5• Disks6                 4 Disks6
   copies to other remote zones      • Zone C
                                                           1
                                                                  • Zone B               • Zone C
                                                                                                                   •
                                                                                                                   • Zone B

                                                                                                                        4
5. Remote nodes ack to local                           US                                              EU
   coordinator                                       Clients                                         Clients
                                     Cassandra                        2
                                                                  Cassandra              Cassandra                 Cassandra
6. Data flushed to internal          • Disks
                                     • Zone B
                                                                  • Disks
                                                                  • Zone C
                                                                          6              • Disks
                                                                                         • Zone B
                                                                                                                   • Disks
                                                                                                                   • Zone C

   commit log disks (no more                          Cassandra                                              5
                                                                                                     6Cassandra
                                                      • Disks
   than 10 seconds later)                             • Zone A
                                                                                                      • Disks
                                                                                                      • Zone A
Extending to Multi-Region
        Added production UK/Ireland support with no downtime
       Minimize impact on original cluster using bulk backup move

1.   Create cluster in EU                     Take a Boeing 737 on a domestic flight, upgrade it to a
                                              747 by adding more engines, fuel and bigger wings
2.   Backup US cluster to S3
                                              and fly it to Europe without landing it on the way…
3.   Restore backup in EU
4.   Local repair EU cluster
5.   Global repair/join
                                               Cassandra
                                                                      100+ms latency          Cassandra   1
                                               • Disks                                        • Disks
                                               • Zone A                                       • Zone A


                               Cassandra                   Cassandra              Cassandra               Cassandra
                               • Disks                     • Disks                • Disks                 • Disks
                               • Zone C                    • Zone B               • Zone C                • Zone B


                                                US                         5                    EU
                                              Clients                                         Clients
                               Cassandra                   Cassandra              Cassandra               Cassandra
                               • Disks                     • Disks                • Disks                 • Disks
                               • Zone B                    • Zone C               • Zone B                • Zone C


                                               Cassandra                                      Cassandra
                                               • Disks                                        • Disks
                                               • Zone A
                                                                                         3    • Zone A
                                                                                                              4
                                          2
                                                  S3
Cassandra Backup
• Full Backup                                              Cassandra

                                           Cassandra                       Cassandra

  – Time based snapshot
  – SSTable compress -> S3     Cassandra                                               Cassandra




• Incremental                                                S3
                                                           Backup
                             Cassandra                                                   Cassandra

  – SSTable write triggers
    compressed copy to S3
                                    Cassandra                                    Cassandra


• Archive                                          Cassandra       Cassandra


  – Copy cross region
                                  A
ETL for Cassandra
•   Data is de-normalized over many clusters!
•   Too many to restore from backups for ETL
•   Solution – read backup files using Hadoop
•   Aegisthus
    – http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html

    – High throughput raw SSTable processing
    – Re-normalizes many clusters to a consistent view
    – Extract, Transform, then Load into Teradata
Netflix Open Source Strategy
• Release PaaS Components git-by-git
  – Source at github.com/netflix – we build from it…
  – Intros and techniques at techblog.netflix.com
  – Blog post or new code every few weeks


• Motivations
  – Give back to Apache licensed OSS community
  – Motivate, retain, hire top engineers
  – “Peer pressure” code cleanup, external contributions
Open Source Projects and Posts
     Legend
 Github / Techblog             Priam                  Exhibitor          Servo and Autoscaling
                       Cassandra as a Service   Zookeeper as a Service          Scripts
Apache Contributions
                             Astyanax                                            Honu
                                                       Curator
   Techblog Post        Cassandra client for                               Log4j streaming to
                                                 Zookeeper Patterns
                               Java                                             Hadoop
   Coming Soon
                                                      EVCache
                            CassJMeter                                       Circuit Breaker
                                                  Memcached as a
                        Cassandra test suite                             Robust service pattern
                                                     Service

                            Cassandra                                            Asgard
                                                  Eureka / Discovery
                         Multi-region EC2                                AutoScaleGroup based
                                                  Service Directory
                         datastore support                                   AWS console

                             Aegisthus                Archaius
                                                                             Chaos Monkey
                          Hadoop ETL for         Dynamics Properties
                                                                         Robustness verification
                            Cassandra                 Service
Chaos Monkey
http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

• Computers (Datacenter or AWS) randomly die
   – Fact of life, but too infrequent to test resiliency
• Test to make sure systems are resilient
   – Allow any instance to fail without customer impact
• Chaos Monkey hours
   – Monday-Friday 9am-3pm random instance kill
• Application configuration option
   – Apps now have to opt-out from Chaos Monkey
Asgard
http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html


• Replacement for AWS Console at Scale
    – Groovy/Grails/JVM based
    – Supports all AWS regions on a global basis
    – Specific to AWS feature set

• Hides the AWS credentials
    – Use AWS IAM to issue restricted keys for Asgard
    – Each Asgard instance manages one account
    – One install each for test, prod, audit accounts
Roadmap for 2012
•   More resiliency and improved availability
•   More automation, orchestration
•   “Hardening” the platform, code clean-up
•   Lower latency for web services and devices
•   IPv6 – running now, see techblog for details
•   More open sourced components
•   Las Vegas in November - AWS Re:Invent
Back to Live Demo
Disclaimers
• We didn’t have time to tune the demo
• These are the plots from the live demo run
• Run’s need to be longer to get to steady state
• Data size only reached around 5GB per node
• Plenty of “I wonder why it did that” remains
• It’s a fair comparison, but not the best absolute
  performance possible for this workload and
  configuration
• When you remove the IO bottleneck, the next
  few bottlenecks appear…
Activity during the talk 10:30-11:30
 Custom AppDynamics dashboard showing CPU and IOPS per node
Jmeter Plots
• Plots are the output of the Jenkins build

• Each instance has its own set of plots

• Each availability zone has its own summary plots

• One of the three zone summary plots is compared for
  each metric

• Plot collection is currently duplicated as we are
  transitioning from “Epic” to “Atlas”
Jenkins
Collected results and graphs after job has completed
The past
m2.4xlarge



 Instances
 per zone




The future
hi1.4xlarge
The past
 m2.4xlarge

Transactions
per zone, same
as total client
transactions




 The future
 hi1.4xlarge
The past
m2.4xlarge




The future
hi1.4xlarge
The past
m2.4xlarge




Thousands of
Microseconds




The future
hi1.4xlarge
The past
m2.4xlarge




Microseconds




The future
hi1.4xlarge
The past
m2.4xlarge




The future
hi1.4xlarge
The past
m2.4xlarge




The future
hi1.4xlarge
Takeaway

  Netflix has built and deployed a scalable global platform based on
                           Cassandra and AWS.

Key components of the Netflix PaaS are being released as Open Source
          projects so you can build your own custom PaaS.

           If you like lots of SSD’s come and work for us….

                         http://github.com/Netflix
                        http://techblog.netflix.com
                        http://slideshare.net/Netflix

                 http://www.linkedin.com/in/adriancockcroft
                    @adrianco #netflixcloud #cassandra12

Weitere ähnliche Inhalte

Was ist angesagt?

Snowflake essentials
Snowflake essentialsSnowflake essentials
Snowflake essentialsqureshihamid
 
(CMP201) All You Need To Know About Auto Scaling
(CMP201) All You Need To Know About Auto Scaling(CMP201) All You Need To Know About Auto Scaling
(CMP201) All You Need To Know About Auto ScalingAmazon Web Services
 
Azure Database Services for MySQL PostgreSQL and MariaDB
Azure Database Services for MySQL PostgreSQL and MariaDBAzure Database Services for MySQL PostgreSQL and MariaDB
Azure Database Services for MySQL PostgreSQL and MariaDBNicholas Vossburg
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaBest Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaKai Wähner
 
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...Edureka!
 
Azure Fundamentals Part 1
Azure Fundamentals Part 1Azure Fundamentals Part 1
Azure Fundamentals Part 1CCG
 
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial ManagerAmazon Web Services Korea
 
Enterprise Cloud Operating Model Design
Enterprise Cloud Operating Model DesignEnterprise Cloud Operating Model Design
Enterprise Cloud Operating Model DesignJoseph Schwartz
 
Well Architected Framework - Data
Well Architected Framework - Data Well Architected Framework - Data
Well Architected Framework - Data Craig Milroy
 
Data Migration Using AWS Snowball, Snowball Edge & Snowmobile
Data Migration Using AWS Snowball, Snowball Edge & SnowmobileData Migration Using AWS Snowball, Snowball Edge & Snowmobile
Data Migration Using AWS Snowball, Snowball Edge & SnowmobileAmazon Web Services
 
Designing and Building Next Generation Data Pipelines at Scale with Structure...
Designing and Building Next Generation Data Pipelines at Scale with Structure...Designing and Building Next Generation Data Pipelines at Scale with Structure...
Designing and Building Next Generation Data Pipelines at Scale with Structure...Databricks
 
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나Amazon Web Services Korea
 
Legacy application modernization with microsoft azure
Legacy application modernization with microsoft azureLegacy application modernization with microsoft azure
Legacy application modernization with microsoft azureOptiSol Business Solutions
 
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...Amazon Web Services Korea
 
Cloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GoogleCloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GooglePatrick Pierson
 
Introduction to azure cosmos db
Introduction to azure cosmos dbIntroduction to azure cosmos db
Introduction to azure cosmos dbRatan Parai
 

Was ist angesagt? (20)

Snowflake essentials
Snowflake essentialsSnowflake essentials
Snowflake essentials
 
(CMP201) All You Need To Know About Auto Scaling
(CMP201) All You Need To Know About Auto Scaling(CMP201) All You Need To Know About Auto Scaling
(CMP201) All You Need To Know About Auto Scaling
 
Azure Database Services for MySQL PostgreSQL and MariaDB
Azure Database Services for MySQL PostgreSQL and MariaDBAzure Database Services for MySQL PostgreSQL and MariaDB
Azure Database Services for MySQL PostgreSQL and MariaDB
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaBest Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
 
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...
Amazon Redshift Tutorial | AWS Tutorial for Beginners | AWS Certification Tra...
 
App Modernization with Microsoft Azure
App Modernization with Microsoft AzureApp Modernization with Microsoft Azure
App Modernization with Microsoft Azure
 
Azure Fundamentals Part 1
Azure Fundamentals Part 1Azure Fundamentals Part 1
Azure Fundamentals Part 1
 
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager
[AWS Builders 온라인 시리즈] AWS, 최적의 비용 효율화 방법은? - 조효원, AWS Cloud Financial Manager
 
Enterprise Cloud Operating Model Design
Enterprise Cloud Operating Model DesignEnterprise Cloud Operating Model Design
Enterprise Cloud Operating Model Design
 
Amazon Aurora: Under the Hood
Amazon Aurora: Under the HoodAmazon Aurora: Under the Hood
Amazon Aurora: Under the Hood
 
Well Architected Framework - Data
Well Architected Framework - Data Well Architected Framework - Data
Well Architected Framework - Data
 
SAP on Azure - Deck
SAP on Azure - DeckSAP on Azure - Deck
SAP on Azure - Deck
 
Databases on AWS Workshop.pdf
Databases on AWS Workshop.pdfDatabases on AWS Workshop.pdf
Databases on AWS Workshop.pdf
 
Data Migration Using AWS Snowball, Snowball Edge & Snowmobile
Data Migration Using AWS Snowball, Snowball Edge & SnowmobileData Migration Using AWS Snowball, Snowball Edge & Snowmobile
Data Migration Using AWS Snowball, Snowball Edge & Snowmobile
 
Designing and Building Next Generation Data Pipelines at Scale with Structure...
Designing and Building Next Generation Data Pipelines at Scale with Structure...Designing and Building Next Generation Data Pipelines at Scale with Structure...
Designing and Building Next Generation Data Pipelines at Scale with Structure...
 
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나
AWS Lake Formation을 통한 손쉬운 데이터 레이크 구성 및 관리 - 윤석찬 :: AWS Unboxing 온라인 세미나
 
Legacy application modernization with microsoft azure
Legacy application modernization with microsoft azureLegacy application modernization with microsoft azure
Legacy application modernization with microsoft azure
 
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...
대용량 데이터베이스의 클라우드 네이티브 DB로 전환 시 확인해야 하는 체크 포인트-김지훈, AWS Database Specialist SA...
 
Cloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GoogleCloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs Google
 
Introduction to azure cosmos db
Introduction to azure cosmos dbIntroduction to azure cosmos db
Introduction to azure cosmos db
 

Andere mochten auch

Architectures for High Availability - QConSF
Architectures for High Availability - QConSFArchitectures for High Availability - QConSF
Architectures for High Availability - QConSFAdrian Cockcroft
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesAdrian Cockcroft
 
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...Adrian Cockcroft
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconAdrian Cockcroft
 
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Adrian Cockcroft
 
Netflix Global Cloud Architecture
Netflix Global Cloud ArchitectureNetflix Global Cloud Architecture
Netflix Global Cloud ArchitectureAdrian Cockcroft
 
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial IntroductionGluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial IntroductionAdrian Cockcroft
 
Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Adrian Cockcroft
 
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)Adrian Cockcroft
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformAdrian Cockcroft
 
Performance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationPerformance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationRamkumar Nottath
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAdrian Cockcroft
 
Microservices Workshop All Topics Deck 2016
Microservices Workshop All Topics Deck 2016Microservices Workshop All Topics Deck 2016
Microservices Workshop All Topics Deck 2016Adrian Cockcroft
 
Cassandra and Solid State Drives
Cassandra and Solid State DrivesCassandra and Solid State Drives
Cassandra and Solid State DrivesRick Branson
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connectAdrian Cockcroft
 

Andere mochten auch (20)

Architectures for High Availability - QConSF
Architectures for High Availability - QConSFArchitectures for High Availability - QConSF
Architectures for High Availability - QConSF
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
 
Gluecon keynote
Gluecon keynoteGluecon keynote
Gluecon keynote
 
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at Gluecon
 
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
 
Netflix Global Cloud Architecture
Netflix Global Cloud ArchitectureNetflix Global Cloud Architecture
Netflix Global Cloud Architecture
 
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial IntroductionGluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction
 
Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013
 
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
Gluecon 2013 - Netflix Cloud Native Tutorial Details (part 2)
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source Platform
 
NetflixOSS Meetup
NetflixOSS MeetupNetflixOSS Meetup
NetflixOSS Meetup
 
Netflix and Open Source
Netflix and Open SourceNetflix and Open Source
Netflix and Open Source
 
Dystopia as a Service
Dystopia as a ServiceDystopia as a Service
Dystopia as a Service
 
Performance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationPerformance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migration
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
 
Microservices Workshop All Topics Deck 2016
Microservices Workshop All Topics Deck 2016Microservices Workshop All Topics Deck 2016
Microservices Workshop All Topics Deck 2016
 
Cassandra and Solid State Drives
Cassandra and Solid State DrivesCassandra and Solid State Drives
Cassandra and Solid State Drives
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connect
 
Netflix in the Cloud
Netflix in the CloudNetflix in the Cloud
Netflix in the Cloud
 

Ähnlich wie Cassandra Performance and Scalability on AWS

Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts
Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source EffortsCassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts
Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source EffortsAcunu
 
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...DataStax Academy
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons
 
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...Ontico
 
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...Amazon Web Services
 
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012Amazon Web Services
 
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...Lviv Startup Club
 
Cassandra Operations at Netflix
Cassandra Operations at NetflixCassandra Operations at Netflix
Cassandra Operations at Netflixgreggulrich
 
VDI storage and storage virtualization
VDI storage and storage virtualizationVDI storage and storage virtualization
VDI storage and storage virtualizationSisimon Soman
 
Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...DataWorks Summit
 
High order bits from cassandra & hadoop
High order bits from cassandra & hadoopHigh order bits from cassandra & hadoop
High order bits from cassandra & hadoopsrisatish ambati
 
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...Lviv Startup Club
 
Cassandra for mission critical data
Cassandra for mission critical dataCassandra for mission critical data
Cassandra for mission critical dataOleksandr Semenov
 
High order bits from cassandra & hadoop
High order bits from cassandra & hadoopHigh order bits from cassandra & hadoop
High order bits from cassandra & hadoopsrisatish ambati
 
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Chris Fregly
 
Servers fail, who cares?
Servers fail, who cares? Servers fail, who cares?
Servers fail, who cares? greggulrich
 
Cluster Computing with Dryad
Cluster Computing with DryadCluster Computing with Dryad
Cluster Computing with Dryadbutest
 

Ähnlich wie Cassandra Performance and Scalability on AWS (20)

Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts
Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source EffortsCassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts
Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts
 
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...
C* Summit 2013: Netflix Open Source Tools and Benchmarks for Cassandra by Adr...
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
 
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...
Cистема распределенного, масштабируемого и высоконадежного хранения данных дл...
 
Svc 202-netflix-open-source
Svc 202-netflix-open-sourceSvc 202-netflix-open-source
Svc 202-netflix-open-source
 
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
How Netflix’s Tools Can Help Accelerate Your Start-up (SVC202) | AWS re:Inven...
 
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
 
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...
Yaroslav Nedashkovsky - "Data Engineering in Information Security: how to col...
 
Cassandra Operations at Netflix
Cassandra Operations at NetflixCassandra Operations at Netflix
Cassandra Operations at Netflix
 
Cassandra at no_sql
Cassandra at no_sqlCassandra at no_sql
Cassandra at no_sql
 
VDI storage and storage virtualization
VDI storage and storage virtualizationVDI storage and storage virtualization
VDI storage and storage virtualization
 
Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...
 
High order bits from cassandra & hadoop
High order bits from cassandra & hadoopHigh order bits from cassandra & hadoop
High order bits from cassandra & hadoop
 
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...
Vitalii Bondarenko - “Azure real-time analytics and kappa architecture with K...
 
Cassandra for mission critical data
Cassandra for mission critical dataCassandra for mission critical data
Cassandra for mission critical data
 
High order bits from cassandra & hadoop
High order bits from cassandra & hadoopHigh order bits from cassandra & hadoop
High order bits from cassandra & hadoop
 
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
 
Data Stores @ Netflix
Data Stores @ NetflixData Stores @ Netflix
Data Stores @ Netflix
 
Servers fail, who cares?
Servers fail, who cares? Servers fail, who cares?
Servers fail, who cares?
 
Cluster Computing with Dryad
Cluster Computing with DryadCluster Computing with Dryad
Cluster Computing with Dryad
 

Mehr von Adrian Cockcroft

Netflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search RoadshowNetflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search RoadshowAdrian Cockcroft
 
Netflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumNetflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumAdrian Cockcroft
 
Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Adrian Cockcroft
 
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)Adrian Cockcroft
 
Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)Adrian Cockcroft
 
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...Adrian Cockcroft
 
Migrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global CassandraMigrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global CassandraAdrian Cockcroft
 
Netflix Velocity Conference 2011
Netflix Velocity Conference 2011Netflix Velocity Conference 2011
Netflix Velocity Conference 2011Adrian Cockcroft
 
Cmg06 utilization is useless
Cmg06 utilization is uselessCmg06 utilization is useless
Cmg06 utilization is uselessAdrian Cockcroft
 
Netflix on Cloud - combined slides for Dev and Ops
Netflix on Cloud - combined slides for Dev and OpsNetflix on Cloud - combined slides for Dev and Ops
Netflix on Cloud - combined slides for Dev and OpsAdrian Cockcroft
 

Mehr von Adrian Cockcroft (14)

Netflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search RoadshowNetflix Global Applications - NoSQL Search Roadshow
Netflix Global Applications - NoSQL Search Roadshow
 
Netflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumNetflix in the Cloud at SV Forum
Netflix in the Cloud at SV Forum
 
Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3)
 
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
 
Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)
 
Global Netflix Platform
Global Netflix PlatformGlobal Netflix Platform
Global Netflix Platform
 
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
 
Migrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global CassandraMigrating Netflix from Datacenter Oracle to Global Cassandra
Migrating Netflix from Datacenter Oracle to Global Cassandra
 
Netflix Velocity Conference 2011
Netflix Velocity Conference 2011Netflix Velocity Conference 2011
Netflix Velocity Conference 2011
 
Migrating to Public Cloud
Migrating to Public CloudMigrating to Public Cloud
Migrating to Public Cloud
 
Netflix in the cloud 2011
Netflix in the cloud 2011Netflix in the cloud 2011
Netflix in the cloud 2011
 
Cmg06 utilization is useless
Cmg06 utilization is uselessCmg06 utilization is useless
Cmg06 utilization is useless
 
Netflix on Cloud - combined slides for Dev and Ops
Netflix on Cloud - combined slides for Dev and OpsNetflix on Cloud - combined slides for Dev and Ops
Netflix on Cloud - combined slides for Dev and Ops
 
NoSQL for Netflix
NoSQL for NetflixNoSQL for Netflix
NoSQL for Netflix
 

Kürzlich hochgeladen

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Kürzlich hochgeladen (20)

Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Cassandra Performance and Scalability on AWS

  • 1. Cassandra Performance and Scalability on AWS August 8th, 2012 Adrian Cockcroft @adrianco #netflixcloud #cassandra12 http://www.linkedin.com/in/adriancockcroft
  • 3. Things we do do. Run benchmarks. Now.
  • 5. Screenshots from Live Demo Backup slides from pre-runs of the demo with some updates to show what actually happened
  • 6. Asgard cass_perf apps, with no instances running
  • 11. Asgard Iinitial set of cass instances up and running
  • 12. Back to Presentation While the load gets going….
  • 14. Scalability from 48 to 288 nodes on AWS http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html Client Writes/s by node count – Replication Factor = 3 1200000 1099837 1000000 800000 Used 288 of m1.xlarge 600000 4 CPU, 15 GB RAM, 8 ECU 537172 Cassandra 0.86 400000 Benchmark config only 366828 existed for about 1hr 200000 174373 0 0 50 100 150 200 250 300 350
  • 15. Blah Blah Blah (I’m skipping all the cloud intro etc. Netflix runs in the cloud, if you hadn’t figured that out already you aren’t paying attention and should go read slideshare.net/netflix)
  • 16. “Some people skate to the puck, I skate to where the puck is going to be” Wayne Gretzky
  • 17. Cassandra on AWS The Past The Future • Instance: m2.4xlarge • Instance: hi1.4xlarge • Storage: 2 drives, 1.7TB • Storage: 2 SSD volumes, 2TB • CPU: 8 Cores, 26 ECU • CPU: 8 HT cores, 35 ECU • RAM: 68GB • RAM: 64GB • Network: 1Gbit • Network: 10Gbit • IOPS: ~500 • IOPS: ~100,000 • Throughput: ~100Mbyte/s • Throughput: ~1Gbyte/s • Cost: $1.80/hr • Cost: $3.10/hr
  • 18. Cassandra Disk vs. SSD Benchmark Same Throughput, Lower Latency, Half Cost
  • 19. Live Demo Workload • Jenkins automation – Jmeter load driver – Asgard provisioning – Priam instance management • Traffic – Reading/writing whole 100 column rows – Randomly selected from 25M row keys – Run for 10minutes, then double ring size
  • 21. Major Front End Services • Non-member Web Site www.netflix.com – Marketing driven, sign up flow, SOX/PCI scope • Member Web Site movies.netflix.com – Personalization driven • CDNs for delivering bulk video/audio – Netflix CDN: openconnect.netflix.com • API for external and device user interfaces – Mostly private APIs, public API docs at developer.netflix.com • API for controlling video playback – DRM, QoS management, Bookmarks
  • 22. Netflix Deployed on AWS 2009 2009 2010 2010 2010 2011 Content Logs Play WWW API CS Content S3 International DRM Sign-Up Metadata Management Terabytes CS lookup EC2 Diagnostics & EMR CDN routing Search Device Config Encoding Actions S3 Movie TV Movie Customer Call Hive & Pig Bookmarks Petabytes Choosing Choosing Log Business Social Logging Ratings CS Analytics Intelligence Facebook CDNs ISPs Terabits Customers
  • 23. Cassandra Instance Architecture Linux Base AMI (CentOS) Priam Cassandra Manager Token Java7 Management, Bac kups, Autoscaling Tomcat/Java7 AppDynamics Monitoring appagent monitoring Cassandra 1.09 Log rotation AppDynamics GC and thread machineagent dump logging Etc.
  • 24. Priam – Cassandra Automation Available at http://github.com/netflix • Netflix Platform Tomcat Code • Zero touch auto-configuration • State management for Cassandra JVM • Token allocation and assignment • Broken node auto-replacement • Full and incremental backup to S3 • Restore sequencing from S3 • Grow/Shrink Cassandra “ring”
  • 25. Astyanax Available at http://github.com/netflix • Features – Complete abstraction of connection pool from RPC protocol – Fluent Style API – Operation retry with backoff – Token aware • Recipes – Distribute row lock (without zookeeper) – Multi-DC row lock – Uniqueness constraint – Multi-row uniqueness constraint – Large file storage
  • 26. Scale Up Return to live demo to watch new nodes coming online
  • 27. Kiklos Clusters growing from 12 to 24 in-service, bootstrapping, garbage-collecting, cass-down • http://explorers.us-east- 1.dyntest.netflix.net:7001/jr/cassandradashbo ard
  • 28. Kiklos Clusters growing from 12 to 24 in-service, bootstrapping, garbage-collecting, cass-down
  • 29. Asgard Showed 24 nodes per cluster, but didn’t get a screen shot
  • 30. Back to Presentation While jenkins/jmeter collects graphs and shuts down the systems
  • 31. Cassandra on AWS A highly available and durable deployment pattern
  • 32. High Availability • Cassandra stores 3 local copies, 1 per zone – Synchronous access, durable, highly available – Read/Write One fastest, use for fire and forget – Read/Write Quorum 2 of 3, use for read-after-write • AWS Availability Zones – Separate buildings – Separate power etc. – Fairly close together
  • 33. “Traditional” Cassandra Write Data Flows Single Region, Multiple Availability Zone, Not Token Aware Cassandra •Disks 2•Zone A 2 4 2 1. Client Writes to any Cassandra 3 3Cassandra If a node goes Cassandra Node •Disks5 •Disks 5 offline, hinted handoff 2. Coordinator Node •Zone C 1 •Zone B completes the write replicates to nodes when the node comes Non Token and Zones back up. 3. Nodes return ack to Aware coordinator Clients Requests can choose to 4. Coordinator returns Cassandra Cassandra wait for one node, a ack to client •Disks •Disks quorum, or all nodes to 5. Data written to •Zone B •Zone C ack the write internal commit log 3 disk (no more than Cassandra SSTable disk writes and 10 seconds later) •Disks 5 compactions occur •Zone A asynchronously
  • 34. Astyanax - Cassandra Write Data Flows Single Region, Multiple Availability Zone, Token Aware Cassandra •Disks •Zone A 1. Client Writes to Cassandra 2 2Cassandra If a node goes nodes and Zones •Disks3 •Disks 3 offline, hinted handoff 2. Nodes return ack to •Zone C 1 •Zone B completes the write client Token when the node comes 3. Data written to back up. internal commit log Aware disks (no more than Clients Requests can choose to 10 seconds later) Cassandra Cassandra wait for one node, a •Disks •Disks quorum, or all nodes to •Zone B •Zone C ack the write 2 Cassandra SSTable disk writes and •Disks 3 compactions occur •Zone A asynchronously
  • 35. Data Flows for Multi-Region Writes Token Aware, Consistency Level = Local Quorum 1. Client writes to local replicas If a node or region goes offline, hinted handoff 2. Local write acks returned to completes the write when the node comes back up. Client which continues when Nightly global compare and repair jobs ensure 2 of 3 local nodes are everything stays consistent. committed 3. Local coordinator writes to remote coordinator. 100+ms latency Cassandra Cassandra 4. When data arrives, remote • Disks • Zone A • Disks • Zone A coordinator node acks and Cassandra 2 2 Cassandra Cassandra 4Cassandra 6 • Disks • Disks 6 3 5• Disks6 4 Disks6 copies to other remote zones • Zone C 1 • Zone B • Zone C • • Zone B 4 5. Remote nodes ack to local US EU coordinator Clients Clients Cassandra 2 Cassandra Cassandra Cassandra 6. Data flushed to internal • Disks • Zone B • Disks • Zone C 6 • Disks • Zone B • Disks • Zone C commit log disks (no more Cassandra 5 6Cassandra • Disks than 10 seconds later) • Zone A • Disks • Zone A
  • 36. Extending to Multi-Region Added production UK/Ireland support with no downtime Minimize impact on original cluster using bulk backup move 1. Create cluster in EU Take a Boeing 737 on a domestic flight, upgrade it to a 747 by adding more engines, fuel and bigger wings 2. Backup US cluster to S3 and fly it to Europe without landing it on the way… 3. Restore backup in EU 4. Local repair EU cluster 5. Global repair/join Cassandra 100+ms latency Cassandra 1 • Disks • Disks • Zone A • Zone A Cassandra Cassandra Cassandra Cassandra • Disks • Disks • Disks • Disks • Zone C • Zone B • Zone C • Zone B US 5 EU Clients Clients Cassandra Cassandra Cassandra Cassandra • Disks • Disks • Disks • Disks • Zone B • Zone C • Zone B • Zone C Cassandra Cassandra • Disks • Disks • Zone A 3 • Zone A 4 2 S3
  • 37. Cassandra Backup • Full Backup Cassandra Cassandra Cassandra – Time based snapshot – SSTable compress -> S3 Cassandra Cassandra • Incremental S3 Backup Cassandra Cassandra – SSTable write triggers compressed copy to S3 Cassandra Cassandra • Archive Cassandra Cassandra – Copy cross region A
  • 38. ETL for Cassandra • Data is de-normalized over many clusters! • Too many to restore from backups for ETL • Solution – read backup files using Hadoop • Aegisthus – http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html – High throughput raw SSTable processing – Re-normalizes many clusters to a consistent view – Extract, Transform, then Load into Teradata
  • 39. Netflix Open Source Strategy • Release PaaS Components git-by-git – Source at github.com/netflix – we build from it… – Intros and techniques at techblog.netflix.com – Blog post or new code every few weeks • Motivations – Give back to Apache licensed OSS community – Motivate, retain, hire top engineers – “Peer pressure” code cleanup, external contributions
  • 40. Open Source Projects and Posts Legend Github / Techblog Priam Exhibitor Servo and Autoscaling Cassandra as a Service Zookeeper as a Service Scripts Apache Contributions Astyanax Honu Curator Techblog Post Cassandra client for Log4j streaming to Zookeeper Patterns Java Hadoop Coming Soon EVCache CassJMeter Circuit Breaker Memcached as a Cassandra test suite Robust service pattern Service Cassandra Asgard Eureka / Discovery Multi-region EC2 AutoScaleGroup based Service Directory datastore support AWS console Aegisthus Archaius Chaos Monkey Hadoop ETL for Dynamics Properties Robustness verification Cassandra Service
  • 41. Chaos Monkey http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html • Computers (Datacenter or AWS) randomly die – Fact of life, but too infrequent to test resiliency • Test to make sure systems are resilient – Allow any instance to fail without customer impact • Chaos Monkey hours – Monday-Friday 9am-3pm random instance kill • Application configuration option – Apps now have to opt-out from Chaos Monkey
  • 42. Asgard http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html • Replacement for AWS Console at Scale – Groovy/Grails/JVM based – Supports all AWS regions on a global basis – Specific to AWS feature set • Hides the AWS credentials – Use AWS IAM to issue restricted keys for Asgard – Each Asgard instance manages one account – One install each for test, prod, audit accounts
  • 43. Roadmap for 2012 • More resiliency and improved availability • More automation, orchestration • “Hardening” the platform, code clean-up • Lower latency for web services and devices • IPv6 – running now, see techblog for details • More open sourced components • Las Vegas in November - AWS Re:Invent
  • 44. Back to Live Demo
  • 45. Disclaimers • We didn’t have time to tune the demo • These are the plots from the live demo run • Run’s need to be longer to get to steady state • Data size only reached around 5GB per node • Plenty of “I wonder why it did that” remains • It’s a fair comparison, but not the best absolute performance possible for this workload and configuration • When you remove the IO bottleneck, the next few bottlenecks appear…
  • 46. Activity during the talk 10:30-11:30 Custom AppDynamics dashboard showing CPU and IOPS per node
  • 47. Jmeter Plots • Plots are the output of the Jenkins build • Each instance has its own set of plots • Each availability zone has its own summary plots • One of the three zone summary plots is compared for each metric • Plot collection is currently duplicated as we are transitioning from “Epic” to “Atlas”
  • 48. Jenkins Collected results and graphs after job has completed
  • 49. The past m2.4xlarge Instances per zone The future hi1.4xlarge
  • 50. The past m2.4xlarge Transactions per zone, same as total client transactions The future hi1.4xlarge
  • 56. Takeaway Netflix has built and deployed a scalable global platform based on Cassandra and AWS. Key components of the Netflix PaaS are being released as Open Source projects so you can build your own custom PaaS. If you like lots of SSD’s come and work for us…. http://github.com/Netflix http://techblog.netflix.com http://slideshare.net/Netflix http://www.linkedin.com/in/adriancockcroft @adrianco #netflixcloud #cassandra12

Hinweis der Redaktion

  1. Complete connection pool abstractionQueries and mutations wrapped in objects created by the Keyspace implementation making it possible to retry failed operations.  This varies from other connection pool implementations on which the operation is created on a specific connection and must be completely redone if it fails.Simplified serialization via method overloading.  The low level thrift library only understands data that is serialized to a byte array.  Hector requires serializers to be specified for nearly every call.  Astyanax minimizes the places where serializers are specified by using predefined ColumnFamiliy and ColumnPath definitions which specify the serializers.  The API also overloads set and get operation for common data types.The internal library does not log anything.  All internal events are instead ... calls to a ConnectionPoolMonitor interface.  This allows customization of log levels and filtering of repeating events outside of the scope of the connection poolSuper columns will soon be replaced by Composite column names. As such it is recommended to not use super columns at all and to use Composite column names instead. There is some support for super columns in Astyanax but those methods have been deprecated and will eventually be removed.