SlideShare ist ein Scribd-Unternehmen logo
1 von 62
Downloaden Sie, um offline zu lesen
Attila Szegedi, Software Engineer
@asz




                                1
Everything I ever
learned about JVM
performance tuning
     @twitter

        2
Everything More
than I ever wanted
 to learned about
JVM performance
       tuning
      @twitter
        3
• Memory tuning
• CPU usage tuning
• Lock contention tuning
• I/O tuning


                   4
Twitter’s biggest enemy




           5
Twitter’s biggest enemy
       Latency




           5
Twitter’s biggest enemy
       Latency




           5
Twitter’s biggest enemy
                         Latency




CC licensed image from http://www.flickr.com/photos/dunechaser/
213255210/                          5
Latency contributors


• By far the biggest contributor is garbage collector
• others are, in no particular order:
   • in-process locking and thread scheduling,
   • I/O,
   • application algorithmic inefficiencies.
                             6
Areas of performance
           tuning

• Memory tuning
• Lock contention tuning
• CPU usage tuning
• I/O tuning
                           7
Areas of memory
      performance tuning


• Memory footprint tuning
• Allocation rate tuning
• Garbage collection tuning

                              8
Memory footprint tuning


• So you got an OutOfMemoryError…
   • Maybe you just have too much data!
   • Maybe your data representation is fat!
   • You can also have a genuine memory leak…
                          9
Too much data

• Run with -verbosegc
• Observe numbers in “Full GC” messages secs]
  [Full GC $before->$after($total), $time

• Can you give the JVM more memory?
• Do you need all that data in memory? Consider
  using:
   • a LRU cache, or…
   • soft references*   10
Fat data

• Can be a problem when you want to do wacky
  things, like
   • load the full Twitter social graph in a single
     JVM
   • load all user metadata in a single JVM
• Slimming internal data representation works at
  these economies of scale
                       11
Fat data: object header


 • JVM object header is normally two machine
     words.
 • That’s 16 bytes, or 128 bits on a 64-bit JVM!
 • new java.lang.Object() takes 16 bytes.
 •   new byte[0]   takes 24 bytes.


                         12
Fat data: padding

              class A {
                  byte x;
              }
              class B extends A {
                  byte y;
              }



•   new A()   takes 24 bytes.

•   new B()   takes 32 bytes.
                        13
Fat data: no inline structs

         class C {
           Object obj = new Object();
         }




  • new C() takes 40 bytes.
  • similarly, no inline array elements.
                         14
Slimming taken to
      extreme
• A research project had to load the full follower
  graph in memory
• Each vertex’s edges ended up being represented
  as int arrays
• If it grows further, we can consider variable-
  length differential encoding in a byte array



                      15
Compressed object
     pointers


• Pointers become 4 bytes long
• Usable below 32 GB of max heap size
• Automatically used below 30 GB of max heap

                    16
Compressed object
           pointers
                   Uncompressed Compressed            32-bit

      Pointer           8               4               4

  Object header         16             12*              8

   Array header         24              16             12

  Superclass pad        8               4               4

* Object can have 4 bytes of fields and still only take up 16 bytes
                                17
Avoid instances of
      primitive wrappers

• Hard won experience with Scala 2.7.7:
 • a Seq[Int] stores java.lang.Integer
 • an Array[Int] stores int
 • first needs (24 + 32 * length) bytes
 • second needs (24 + 4 * length) bytes
                        18
Avoid instances of
     primitive wrappers

• This was fixed in Scala 2.8, but it shows that:
 • you often don’t know the performance
   characteristics of your libraries,
 • and won’t ever know them until you run your
   application under a profiler.


                          19
Map footprints


•   Guava MapMaker.makeMap() takes 2272 bytes!

•   MapMaker.concurrencyLevel(1).makeMap()
    takes 352 bytes!
• ConcurrentMap with level 1 makes sense
    sometimes (i.e. you don’t want a
    ConcurrentModificationException)

                       20
Thrift can be heavy



• Thrift generated classes are used to encapsulate a
  wire tranfer format.
• Using them as your domain objects: almost never
  a good idea.



                         21
Thrift can be heavy


• Every Thrift class with a primitive field has a
  java.util.BitSet __isset_bit_vector       field.

• It adds between 52 and 72 bytes of overhead per
  object.



                            22
Thrift can be heavy




         23
Thrift can be heavy


• Thrift does not support 32-bit floats.
• Coupling domain model with transport:
 • resistance to change domain model
• You also miss oportunities for interning and N-to-1
  normalization.

                          24
class Location {
   public String city;
   public String region;
   public String countryCode;
   public int metro;
   public List<String> placeIds;
   public double lat;
   public double lon;
   public double confidence;




                        25
class SharedLocation {
   public String city;
   public String region;
   public String countryCode;
   public int metro;
   public List<String> placeIds;
class UniqueLocation {
   private SharedLocation sharedLocation;
   public double lat;
   public double lon;
   public double confidence;




                        26
Careful with thread locals

• Thread locals stick around.
• Particularly problematic in thread pools with m⨯n
  resource association.
 • 200 pooled threads using 50 connections: you end
   up with 10 000 connection buffers.
• Consider using synchronized objects, or
• just create new objects all the time.
                          27
Part II:
fighting latency



       28
Performance tradeoff

               Memory




                 Time



  Convenient, but oversimplified view.
                  29
Performance triangle

          Memory footprint




 Throughput                  Latency




                 30
Performance triangle

                     Compactness




       Throughput                   Responsiveness
                     C ⨯T ⨯ R = a
• Tuning: vary C, T, R for fixed a
• Optimization: increase a  31
Performance triangle

• Compactness: inverse of memory footprint
• Responsiveness: longest pause the application will
  experience
• Throughput: amount of useful application CPU work
  over time
• Can trade one for the other, within limits.
• If you have spare CPU, can be pure win.
                           32
Responsiveness vs.
   throughput




        33
Biggest threat to
responsiveness in the JVM
  is the garbage collector


            34
Memory pools

Eden       Survivor             Old



                         Code
       Permanent
                        cache




  This is entirely HotSpot specific!
                   35
How does young gen
           work?
          Eden          S1        S2      Old

• All new allocation happens in eden.
 • It only costs a pointer bump.
• When eden fills up, stop-the-world copy-collection
  into the survivor space.
 • Dead objects cost zero to collect.
• Aftr several collections, survivors get tenured into
  old generation.
                             36
Ideal young gen operation


• Big enough to hold more than one set of all
  concurrent request-response cycle objects.
• Each survivor space big enough to hold active
  request objects + tenuring ones.
• Tenuring threshold such that long-lived objects
  tenure fast.

                           37
Old generation collectors

• Throughput collectors
 •   -XX:+UseSerialGC

 •   -XX:+UseParallelGC

 •   -XX:+UseParallelOldGC

• Low-pause collectors
 •   -XX:+UseConcMarkSweepGC

 •   -XX:+UseG1GC   (can’t discuss it here)

                             38
Adaptive sizing policy

• Throughput collectors can automatically tune
  themselves:
 •   -XX:+UseAdaptiveSizePolicy

 •   -XX:MaxGCPauseMillis=…      (i.e. 100)
 •   -XX:GCTimeRatio=…   (i.e. 19)



                            39
Adaptive sizing policy at
         work




            40
Choose a collector


• Bulk service: throughput collector, no adaptive sizing
  policy.
• Everything else: try throughput collector with
  adaptive sizing policy. If it didn’t work, use
  concurrent mark-and-sweep (CMS).



                           41
Always start with tuning
 the young generation
• Enable -XX:+PrintGCDetails, -XX:+PrintHeapAtGC,
  and -XX:+PrintTenuringDistribution.

• Watch survivor sizes! You’ll need to determine
  “desired survivor size”.
• There’s no such thing as a “desired eden size”, mind
  you. The bigger, the better, with some
  responsiveness caveats.
• Watch the tenuring threshold; might need to tune it
  to tenure long lived objects faster.
                          42
-XX:+PrintHeapAtGC


Heap after GC invocations=7000 (full 87):
  par new generation    total 4608000K, used 398455K
   eden space 4096000K,    0% used
   from space 512000K, 77% used
   to   space 512000K,    0% used
  concurrent mark-sweep generation total 3072000K, used 1565157K
  concurrent-mark-sweep perm gen total 53256K, used 31889K
}




                                43
-XX:+PrintTenuringDistribution

Desired   survivor size   262144000 bytes, new threshold 4 (max 4)
- age     1: 137474336    bytes, 137474336 total
- age     2:   37725496   bytes, 175199832 total
- age     3:   23551752   bytes, 198751584 total
- age     4:   14772272   bytes, 213523856 total



 • Things of interest:
  • Number of ages
  • Size distribution in ages
    • You want strongly declining.
                                   44
Tuning the CMS

• Give your app as much memory as possible.
 • CMS is speculative. More memory, less punitive
   miscalculations.
• Try using CMS without tuning. Use -verbosegc and
  -XX:+PrintGCDetails.

 • Didn’t get any “Full GC” messages? You’re done!
• Otherwise, tune the young generation first.
                         45
Tuning the old generation


 • Goals:
  • Keep the fragmentation low.
  • Avoid full GC stops.
 • Fortunately, the two goals are not conflicting.

                          46
Tuning the old generation


 • Find the minimum and maximum working set size
   (observe “Full GC” numbers under stable state and
   under load).
 • Overprovision the numbers by 25-33%.
  • This gives CMS a cushion to concurrently clean
    memory as it’s used.

                           47
Tuning the old generation


 • Set -XX:InitiatingOccupancyFraction to
   between 80-75, respectively.
  • corresponds to overprovisioned heap ratio.
 • You can lower initiating occupancy fraction to 0 if
   you have CPU to spare.


                         48
Responsiveness still not
     good enough?
• Too many live objects during young gen GC:
 • Reduce NewSize, reduce survivor spaces, reduce
   tenuring threshold.
• Too many threads:
 • Find the minimal concurrency level, or
 • split the service into several JVMs.
                          49
Responsiveness still not
    good enough?
• Does the CMS abortable preclean phase, well,
  abort?
 • It is sensitive to number of objects in the new
   generation, so
   • go for smaller new generation
   • try to reduce the amount of short-lived garbage
     your app creates.

                         50
Part III:
let’s take a break from GC



            51
Thread coordination
        optimization

• You don’t have to always go for synchronized.
• Synchronization is a read barrier on entry; write
  barrier on exit.
• Sometimes you only need a half-barrier; i.e. in a
  producer-observer pattern.
• Volatiles can be used as half-barriers.
                            52
Thread coordination
        optimization

• For atomic update of a single value, you only need
  Atomic{Integer|Long}.compareAndSet().

• You can use AtomicReference.compareAndSet() for
  atomic update of composite values represented by
  immutable objects.


                          53
Fight CMS fragmentation
   with slab allocators

• CMS doesn’t compact, so it’s prone to fragmentation,
  which will lead to a stop-the-world pause.
• Apache Cassandra uses a slab allocator internally.


                           54
Cassandra slab allocator


 • 2MB slab sizes
 • copy byte[] into them using compare-and-set
 • GC before: 30-60 seconds every hour
 • GC after: 5 seconds once in 3 days and 10 hours

                       55
Slab allocator constraints
• Works for limited usage:
 • Buffers are written to linearly, flushed to disk and
   recycled when they fill up.
 • The objects need to be converted to binary
   representation anyway.
• If you need random freeing and compaction, you’re
  heading down the wrong direction.
• If you find yourself writing a full memory manager
  on top of byte buffers, stop!
                          56
Soft references revisited

• Soft reference clearing is based on the amount of
  free memory available when GC encounters the
  reference.
• By definition, throughput collectors always clear
  them.
• Can use them with CMS, but they increase memory
  pressure and make the behavior less predictable.
• Need two GC cycles to get rid of referenced objects.
                           57
Everything More
than I ever wanted
 to learned about
JVM performance
       tuning
      @twitter
   Questions?
        58
Attila Szegedi, Software Engineer
@asz




                               59

Weitere ähnliche Inhalte

Was ist angesagt?

The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesDatabricks
 
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLBuilding a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLDatabricks
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsBrendan Gregg
 
Best Practices for Using Apache Spark on AWS
Best Practices for Using Apache Spark on AWSBest Practices for Using Apache Spark on AWS
Best Practices for Using Apache Spark on AWSAmazon Web Services
 
Dynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDatabricks
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
Memory access tracing [poug17]
Memory access tracing [poug17]Memory access tracing [poug17]
Memory access tracing [poug17]Mahmoud Hatem
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkKazuaki Ishizaki
 
[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on KubernetesBruno Borges
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama
 
Memory Management in Apache Spark
Memory Management in Apache SparkMemory Management in Apache Spark
Memory Management in Apache SparkDatabricks
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc.
 
Cosco: An Efficient Facebook-Scale Shuffle Service
Cosco: An Efficient Facebook-Scale Shuffle ServiceCosco: An Efficient Facebook-Scale Shuffle Service
Cosco: An Efficient Facebook-Scale Shuffle ServiceDatabricks
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introductioncolorant
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesNishith Agarwal
 

Was ist angesagt? (20)

The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
 
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQLBuilding a SIMD Supported Vectorized Native Engine for Spark SQL
Building a SIMD Supported Vectorized Native Engine for Spark SQL
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
 
Best Practices for Using Apache Spark on AWS
Best Practices for Using Apache Spark on AWSBest Practices for Using Apache Spark on AWS
Best Practices for Using Apache Spark on AWS
 
Dynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache SparkDynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache Spark
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
Memory access tracing [poug17]
Memory access tracing [poug17]Memory access tracing [poug17]
Memory access tracing [poug17]
 
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkEvening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in Flink
 
Tuning the g1gc
Tuning the g1gcTuning the g1gc
Tuning the g1gc
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
 
[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
 
Memory Management in Apache Spark
Memory Management in Apache SparkMemory Management in Apache Spark
Memory Management in Apache Spark
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
Cosco: An Efficient Facebook-Scale Shuffle Service
Cosco: An Efficient Facebook-Scale Shuffle ServiceCosco: An Efficient Facebook-Scale Shuffle Service
Cosco: An Efficient Facebook-Scale Shuffle Service
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introduction
 
Kafka slideshare
Kafka   slideshareKafka   slideshare
Kafka slideshare
 
HBase Storage Internals
HBase Storage InternalsHBase Storage Internals
HBase Storage Internals
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilities
 

Andere mochten auch

Java performance tuning
Java performance tuningJava performance tuning
Java performance tuningJerry Kurian
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainAttila Szegedi
 
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil TeneUnderstanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil Tenejaxconf
 
Java Performance Monitoring & Tuning
Java Performance Monitoring & TuningJava Performance Monitoring & Tuning
Java Performance Monitoring & TuningMuhammed Shakir
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applicationspkoza
 
Pimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPierre Laporte
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesMatt Harrison
 
F5 link controller
F5  link controllerF5  link controller
F5 link controllerJimmy Saigon
 
Regions Of Pennsylvania
Regions Of PennsylvaniaRegions Of Pennsylvania
Regions Of Pennsylvaniaiheart4th
 
Infrastructureless Wireless networks
Infrastructureless Wireless networksInfrastructureless Wireless networks
Infrastructureless Wireless networksGwendal Simon
 
Tropical seasonal forests
Tropical seasonal forestsTropical seasonal forests
Tropical seasonal forestsVinaya Shah
 
radioligand binding studies
radioligand binding studiesradioligand binding studies
radioligand binding studiesankit
 
Feature Story - Sample
Feature Story - SampleFeature Story - Sample
Feature Story - SampleCourtney Dunn
 
Guide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesGuide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesSarah Fox
 
Relationship marketing concept, process and importance
Relationship marketing concept, process and importanceRelationship marketing concept, process and importance
Relationship marketing concept, process and importancegaurav jain
 
Citing Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatCiting Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatkhornberger
 
Difference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsDifference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsInteractive Bees
 
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowHow Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowAmie Weller
 
Layouting Your School Paper
Layouting Your School PaperLayouting Your School Paper
Layouting Your School PaperJerry Noveno
 
Pharmaceutical packaging
Pharmaceutical packagingPharmaceutical packaging
Pharmaceutical packagingceutics1315
 

Andere mochten auch (20)

Java performance tuning
Java performance tuningJava performance tuning
Java performance tuning
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages Toolchain
 
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil TeneUnderstanding Java Garbage Collection and What You Can Do About It: Gil Tene
Understanding Java Garbage Collection and What You Can Do About It: Gil Tene
 
Java Performance Monitoring & Tuning
Java Performance Monitoring & TuningJava Performance Monitoring & Tuning
Java Performance Monitoring & Tuning
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
 
Pimp my gc - Supersonic Scala
Pimp my gc - Supersonic ScalaPimp my gc - Supersonic Scala
Pimp my gc - Supersonic Scala
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 Minutes
 
F5 link controller
F5  link controllerF5  link controller
F5 link controller
 
Regions Of Pennsylvania
Regions Of PennsylvaniaRegions Of Pennsylvania
Regions Of Pennsylvania
 
Infrastructureless Wireless networks
Infrastructureless Wireless networksInfrastructureless Wireless networks
Infrastructureless Wireless networks
 
Tropical seasonal forests
Tropical seasonal forestsTropical seasonal forests
Tropical seasonal forests
 
radioligand binding studies
radioligand binding studiesradioligand binding studies
radioligand binding studies
 
Feature Story - Sample
Feature Story - SampleFeature Story - Sample
Feature Story - Sample
 
Guide to Construction Procurement Strategies
Guide to Construction Procurement StrategiesGuide to Construction Procurement Strategies
Guide to Construction Procurement Strategies
 
Relationship marketing concept, process and importance
Relationship marketing concept, process and importanceRelationship marketing concept, process and importance
Relationship marketing concept, process and importance
 
Citing Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA formatCiting Yourself (citing your previous work) in MLA or APA format
Citing Yourself (citing your previous work) in MLA or APA format
 
Difference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leafletsDifference between flyers, brochures, posters & leaflets
Difference between flyers, brochures, posters & leaflets
 
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't knowHow Brands Grow : A summary of Byron Sharp's book on what marketers don't know
How Brands Grow : A summary of Byron Sharp's book on what marketers don't know
 
Layouting Your School Paper
Layouting Your School PaperLayouting Your School Paper
Layouting Your School Paper
 
Pharmaceutical packaging
Pharmaceutical packagingPharmaceutical packaging
Pharmaceutical packaging
 

Ähnlich wie Everything I Ever Learned About JVM Performance Tuning @Twitter

Decima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnDecima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnGuerrilla
 
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_SummaryHiram Fleitas León
 
Memory Management & Garbage Collection
Memory Management & Garbage CollectionMemory Management & Garbage Collection
Memory Management & Garbage CollectionAbhishek Sur
 
Performance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersPerformance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersNLJUG
 
Spil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLSpil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLThijs Terlouw
 
NYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentNYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentSpeedment, Inc.
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...Chester Chen
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterLucidworks
 
NickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleNickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleKostas Mavridis
 
An Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAn Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAbhishek Asthana
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]Speedment, Inc.
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]Malin Weiss
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Christopher Curtin
 
Large scale computing with mapreduce
Large scale computing with mapreduceLarge scale computing with mapreduce
Large scale computing with mapreducehansen3032
 
Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Saltmarch Media
 
7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth7. Key-Value Databases: In Depth
7. Key-Value Databases: In DepthFabio Fumarola
 
High Performance With Java
High Performance With JavaHigh Performance With Java
High Performance With Javamalduarte
 
In-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteIn-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteDataWorks Summit
 

Ähnlich wie Everything I Ever Learned About JVM Performance Tuning @Twitter (20)

Decima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero DawnDecima Engine: Visibility in Horizon Zero Dawn
Decima Engine: Visibility in Horizon Zero Dawn
 
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
[DBA]_HiramFleitas_SQL_PASS_Summit_2017_Summary
 
Memory Management & Garbage Collection
Memory Management & Garbage CollectionMemory Management & Garbage Collection
Memory Management & Garbage Collection
 
Performance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen BorgersPerformance van Java 8 en verder - Jeroen Borgers
Performance van Java 8 en verder - Jeroen Borgers
 
Spil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NLSpil Storage Platform (Erlang) @ EUG-NL
Spil Storage Platform (Erlang) @ EUG-NL
 
Pulsar
PulsarPulsar
Pulsar
 
NYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ SpeedmentNYJavaSIG - Big Data Microservices w/ Speedment
NYJavaSIG - Big Data Microservices w/ Speedment
 
Loom promises: be there!
Loom promises: be there!Loom promises: be there!
Loom promises: be there!
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, Twitter
 
NickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScaleNickKallen_DataArchitectureAtTwitterScale
NickKallen_DataArchitectureAtTwitterScale
 
An Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in JavaAn Introduction to JVM Internals and Garbage Collection in Java
An Introduction to JVM Internals and Garbage Collection in Java
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
 
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
 
Large scale computing with mapreduce
Large scale computing with mapreduceLarge scale computing with mapreduce
Large scale computing with mapreduce
 
Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?Is NoSQL The Future of Data Storage?
Is NoSQL The Future of Data Storage?
 
7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth7. Key-Value Databases: In Depth
7. Key-Value Databases: In Depth
 
High Performance With Java
High Performance With JavaHigh Performance With Java
High Performance With Java
 
In-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great TasteIn-memory Caching in HDFS: Lower Latency, Same Great Taste
In-memory Caching in HDFS: Lower Latency, Same Great Taste
 

Kürzlich hochgeladen

activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 

Kürzlich hochgeladen (20)

activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 

Everything I Ever Learned About JVM Performance Tuning @Twitter

  • 1. Attila Szegedi, Software Engineer @asz 1
  • 2. Everything I ever learned about JVM performance tuning @twitter 2
  • 3. Everything More than I ever wanted to learned about JVM performance tuning @twitter 3
  • 4. • Memory tuning • CPU usage tuning • Lock contention tuning • I/O tuning 4
  • 8. Twitter’s biggest enemy Latency CC licensed image from http://www.flickr.com/photos/dunechaser/ 213255210/ 5
  • 9. Latency contributors • By far the biggest contributor is garbage collector • others are, in no particular order: • in-process locking and thread scheduling, • I/O, • application algorithmic inefficiencies. 6
  • 10. Areas of performance tuning • Memory tuning • Lock contention tuning • CPU usage tuning • I/O tuning 7
  • 11. Areas of memory performance tuning • Memory footprint tuning • Allocation rate tuning • Garbage collection tuning 8
  • 12. Memory footprint tuning • So you got an OutOfMemoryError… • Maybe you just have too much data! • Maybe your data representation is fat! • You can also have a genuine memory leak… 9
  • 13. Too much data • Run with -verbosegc • Observe numbers in “Full GC” messages secs] [Full GC $before->$after($total), $time • Can you give the JVM more memory? • Do you need all that data in memory? Consider using: • a LRU cache, or… • soft references* 10
  • 14. Fat data • Can be a problem when you want to do wacky things, like • load the full Twitter social graph in a single JVM • load all user metadata in a single JVM • Slimming internal data representation works at these economies of scale 11
  • 15. Fat data: object header • JVM object header is normally two machine words. • That’s 16 bytes, or 128 bits on a 64-bit JVM! • new java.lang.Object() takes 16 bytes. • new byte[0] takes 24 bytes. 12
  • 16. Fat data: padding class A { byte x; } class B extends A { byte y; } • new A() takes 24 bytes. • new B() takes 32 bytes. 13
  • 17. Fat data: no inline structs class C { Object obj = new Object(); } • new C() takes 40 bytes. • similarly, no inline array elements. 14
  • 18. Slimming taken to extreme • A research project had to load the full follower graph in memory • Each vertex’s edges ended up being represented as int arrays • If it grows further, we can consider variable- length differential encoding in a byte array 15
  • 19. Compressed object pointers • Pointers become 4 bytes long • Usable below 32 GB of max heap size • Automatically used below 30 GB of max heap 16
  • 20. Compressed object pointers Uncompressed Compressed 32-bit Pointer 8 4 4 Object header 16 12* 8 Array header 24 16 12 Superclass pad 8 4 4 * Object can have 4 bytes of fields and still only take up 16 bytes 17
  • 21. Avoid instances of primitive wrappers • Hard won experience with Scala 2.7.7: • a Seq[Int] stores java.lang.Integer • an Array[Int] stores int • first needs (24 + 32 * length) bytes • second needs (24 + 4 * length) bytes 18
  • 22. Avoid instances of primitive wrappers • This was fixed in Scala 2.8, but it shows that: • you often don’t know the performance characteristics of your libraries, • and won’t ever know them until you run your application under a profiler. 19
  • 23. Map footprints • Guava MapMaker.makeMap() takes 2272 bytes! • MapMaker.concurrencyLevel(1).makeMap() takes 352 bytes! • ConcurrentMap with level 1 makes sense sometimes (i.e. you don’t want a ConcurrentModificationException) 20
  • 24. Thrift can be heavy • Thrift generated classes are used to encapsulate a wire tranfer format. • Using them as your domain objects: almost never a good idea. 21
  • 25. Thrift can be heavy • Every Thrift class with a primitive field has a java.util.BitSet __isset_bit_vector field. • It adds between 52 and 72 bytes of overhead per object. 22
  • 26. Thrift can be heavy 23
  • 27. Thrift can be heavy • Thrift does not support 32-bit floats. • Coupling domain model with transport: • resistance to change domain model • You also miss oportunities for interning and N-to-1 normalization. 24
  • 28. class Location { public String city; public String region; public String countryCode; public int metro; public List<String> placeIds; public double lat; public double lon; public double confidence; 25
  • 29. class SharedLocation { public String city; public String region; public String countryCode; public int metro; public List<String> placeIds; class UniqueLocation { private SharedLocation sharedLocation; public double lat; public double lon; public double confidence; 26
  • 30. Careful with thread locals • Thread locals stick around. • Particularly problematic in thread pools with m⨯n resource association. • 200 pooled threads using 50 connections: you end up with 10 000 connection buffers. • Consider using synchronized objects, or • just create new objects all the time. 27
  • 32. Performance tradeoff Memory Time Convenient, but oversimplified view. 29
  • 33. Performance triangle Memory footprint Throughput Latency 30
  • 34. Performance triangle Compactness Throughput Responsiveness C ⨯T ⨯ R = a • Tuning: vary C, T, R for fixed a • Optimization: increase a 31
  • 35. Performance triangle • Compactness: inverse of memory footprint • Responsiveness: longest pause the application will experience • Throughput: amount of useful application CPU work over time • Can trade one for the other, within limits. • If you have spare CPU, can be pure win. 32
  • 36. Responsiveness vs. throughput 33
  • 37. Biggest threat to responsiveness in the JVM is the garbage collector 34
  • 38. Memory pools Eden Survivor Old Code Permanent cache This is entirely HotSpot specific! 35
  • 39. How does young gen work? Eden S1 S2 Old • All new allocation happens in eden. • It only costs a pointer bump. • When eden fills up, stop-the-world copy-collection into the survivor space. • Dead objects cost zero to collect. • Aftr several collections, survivors get tenured into old generation. 36
  • 40. Ideal young gen operation • Big enough to hold more than one set of all concurrent request-response cycle objects. • Each survivor space big enough to hold active request objects + tenuring ones. • Tenuring threshold such that long-lived objects tenure fast. 37
  • 41. Old generation collectors • Throughput collectors • -XX:+UseSerialGC • -XX:+UseParallelGC • -XX:+UseParallelOldGC • Low-pause collectors • -XX:+UseConcMarkSweepGC • -XX:+UseG1GC (can’t discuss it here) 38
  • 42. Adaptive sizing policy • Throughput collectors can automatically tune themselves: • -XX:+UseAdaptiveSizePolicy • -XX:MaxGCPauseMillis=… (i.e. 100) • -XX:GCTimeRatio=… (i.e. 19) 39
  • 44. Choose a collector • Bulk service: throughput collector, no adaptive sizing policy. • Everything else: try throughput collector with adaptive sizing policy. If it didn’t work, use concurrent mark-and-sweep (CMS). 41
  • 45. Always start with tuning the young generation • Enable -XX:+PrintGCDetails, -XX:+PrintHeapAtGC, and -XX:+PrintTenuringDistribution. • Watch survivor sizes! You’ll need to determine “desired survivor size”. • There’s no such thing as a “desired eden size”, mind you. The bigger, the better, with some responsiveness caveats. • Watch the tenuring threshold; might need to tune it to tenure long lived objects faster. 42
  • 46. -XX:+PrintHeapAtGC Heap after GC invocations=7000 (full 87): par new generation total 4608000K, used 398455K eden space 4096000K, 0% used from space 512000K, 77% used to space 512000K, 0% used concurrent mark-sweep generation total 3072000K, used 1565157K concurrent-mark-sweep perm gen total 53256K, used 31889K } 43
  • 47. -XX:+PrintTenuringDistribution Desired survivor size 262144000 bytes, new threshold 4 (max 4) - age 1: 137474336 bytes, 137474336 total - age 2: 37725496 bytes, 175199832 total - age 3: 23551752 bytes, 198751584 total - age 4: 14772272 bytes, 213523856 total • Things of interest: • Number of ages • Size distribution in ages • You want strongly declining. 44
  • 48. Tuning the CMS • Give your app as much memory as possible. • CMS is speculative. More memory, less punitive miscalculations. • Try using CMS without tuning. Use -verbosegc and -XX:+PrintGCDetails. • Didn’t get any “Full GC” messages? You’re done! • Otherwise, tune the young generation first. 45
  • 49. Tuning the old generation • Goals: • Keep the fragmentation low. • Avoid full GC stops. • Fortunately, the two goals are not conflicting. 46
  • 50. Tuning the old generation • Find the minimum and maximum working set size (observe “Full GC” numbers under stable state and under load). • Overprovision the numbers by 25-33%. • This gives CMS a cushion to concurrently clean memory as it’s used. 47
  • 51. Tuning the old generation • Set -XX:InitiatingOccupancyFraction to between 80-75, respectively. • corresponds to overprovisioned heap ratio. • You can lower initiating occupancy fraction to 0 if you have CPU to spare. 48
  • 52. Responsiveness still not good enough? • Too many live objects during young gen GC: • Reduce NewSize, reduce survivor spaces, reduce tenuring threshold. • Too many threads: • Find the minimal concurrency level, or • split the service into several JVMs. 49
  • 53. Responsiveness still not good enough? • Does the CMS abortable preclean phase, well, abort? • It is sensitive to number of objects in the new generation, so • go for smaller new generation • try to reduce the amount of short-lived garbage your app creates. 50
  • 54. Part III: let’s take a break from GC 51
  • 55. Thread coordination optimization • You don’t have to always go for synchronized. • Synchronization is a read barrier on entry; write barrier on exit. • Sometimes you only need a half-barrier; i.e. in a producer-observer pattern. • Volatiles can be used as half-barriers. 52
  • 56. Thread coordination optimization • For atomic update of a single value, you only need Atomic{Integer|Long}.compareAndSet(). • You can use AtomicReference.compareAndSet() for atomic update of composite values represented by immutable objects. 53
  • 57. Fight CMS fragmentation with slab allocators • CMS doesn’t compact, so it’s prone to fragmentation, which will lead to a stop-the-world pause. • Apache Cassandra uses a slab allocator internally. 54
  • 58. Cassandra slab allocator • 2MB slab sizes • copy byte[] into them using compare-and-set • GC before: 30-60 seconds every hour • GC after: 5 seconds once in 3 days and 10 hours 55
  • 59. Slab allocator constraints • Works for limited usage: • Buffers are written to linearly, flushed to disk and recycled when they fill up. • The objects need to be converted to binary representation anyway. • If you need random freeing and compaction, you’re heading down the wrong direction. • If you find yourself writing a full memory manager on top of byte buffers, stop! 56
  • 60. Soft references revisited • Soft reference clearing is based on the amount of free memory available when GC encounters the reference. • By definition, throughput collectors always clear them. • Can use them with CMS, but they increase memory pressure and make the behavior less predictable. • Need two GC cycles to get rid of referenced objects. 57
  • 61. Everything More than I ever wanted to learned about JVM performance tuning @twitter Questions? 58
  • 62. Attila Szegedi, Software Engineer @asz 59

Hinweis der Redaktion

  1. Here it is all together\n
  2. Here it is all together\n
  3. Here it is all together\n
  4. Here it is all together\n
  5. Here it is all together\n
  6. Here it is all together\n
  7. Here it is all together\n
  8. Here it is all together\n
  9. Here it is all together\n
  10. Here it is all together\n
  11. Here it is all together\n
  12. Here it is all together\n
  13. Here it is all together\n
  14. Here it is all together\n
  15. Here it is all together\n
  16. Here it is all together\n
  17. Here it is all together\n
  18. Here it is all together\n
  19. Here it is all together\n
  20. Here it is all together\n
  21. Here it is all together\n
  22. Here it is all together\n
  23. Here it is all together\n
  24. Here it is all together\n
  25. Here it is all together\n
  26. Here it is all together\n
  27. \n
  28. \n
  29. Here it is all together\n
  30. Here it is all together\n
  31. Here it is all together\n
  32. Here it is all together\n
  33. Here it is all together\n
  34. Here it is all together\n
  35. Here it is all together\n
  36. Here it is all together\n
  37. Here it is all together\n
  38. Here it is all together\n
  39. Here it is all together\n
  40. Here it is all together\n
  41. Here it is all together\n
  42. Here it is all together\n
  43. Here it is all together\n
  44. Here it is all together\n
  45. Here it is all together\n
  46. Here it is all together\n
  47. Here it is all together\n
  48. Here it is all together\n
  49. Here it is all together\n
  50. Here it is all together\n
  51. Here it is all together\n
  52. Here it is all together\n
  53. Here it is all together\n
  54. Here it is all together\n
  55. Here it is all together\n
  56. Here it is all together\n
  57. Here it is all together\n
  58. Here it is all together\n
  59. Here it is all together\n
  60. Here it is all together\n
  61. Here it is all together\n