SlideShare ist ein Scribd-Unternehmen logo
1 von 43
Downloaden Sie, um offline zu lesen
Konstantine Krutiy
Principal Engineer, Crew Lead
PATH TO EXTRA PERFORMANCE
Eliminate unneeded work
§ Choose data types wisely
Eliminate unneeded waits
§ Reduce number of locks
Make system operate in more efficient way
§ Optimize BIOS settings
§ Stay in same “technology slice”
§ Make sure you have enough RAM
ART OF CHOOSING DATATYPES
WHY DATA TYPE MATTERS ?
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set
starts from
100 billion records
WHY DATA TYPE MATTERS ?
Fastest CPU today is 3.7 GHz
It takes
1 / 3,700,000,000 of a second
to do single operation
“BIG DATA” record set
starts from
100 billion records
Processing time
1 / 3,700,000,000 sec X 100,000,000,000 = 27 sec
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money
Data type: MONEY
Internal data type: NUMERIC(18,4)
Storage: Store as numeric
Data type: NUMERIC
Internal data type: NUMERIC(37,15)
Storage: Store as integer
Data type: INT
Internal data type: INT
DO YOU NEED TO STORE DATA SAME
WAY IT IS PRESENTED ?
Presentation: $395.17
Data: 395.17
Storage: Store as Money
Data type: MONEY
Internal data type: NUMERIC(18,4)
Storage: Store as numeric
Data type: NUMERIC
Internal data type: NUMERIC(37,15)
Storage: Store as integer
Data type: INT
Internal data type: INT
DATA TYPE BENCHMARK DATA
DATA TYPE BENCHMARK AVERAGES IN SEC
27.2
29.7
37
0
5
10
15
20
25
30
35
40
INT NUMERIC(18,5) NUMERIC(37,15)
MAKING RIGHT CHOICES
• If you can store data as INTEGER
• Choose INTEGER
• If your data fits into 18 digits of PRECISION
• Choose NUMERIC(18)
• If your data larger then 18 digits of PRECISION
• Choose NUMERIC(your-desired-precision)
Vertica default for NUMERIC is NUMERIC(37,15)
ELIMINATING UNNECESSARY LOCKING
LOCKING BEHAVIOR
AUTOCOMMIT = ON (jdbc driver default)
§ Each statement treated as complete transaction
§ When statement completes changes automatically
committed to database
AUTOCOMMIT = OFF
§ Transaction continue until manually run COMMIT or
ROLLBACK
§ Locks kept on objects for transaction duration
CONTROLLING AUTOCOMMIT STATE
JAVA:
conn = DriverManager.getConnection("jdbc:vertica://DBHost:5433/MyDB", myProperties);
// get the state of the auto commit parameter
System.out.println("Autocommit state: " + conn.getAutoCommit());
// Change the auto commit state to false
conn.setAutoCommit(false);
SQL:
IMPACT ON LOCK COUNTS BY CHANGING
AUTOCOMMIT SETTING TO OFF
HOW TO DISABLE – OBVIOUS METHOD
HOW TO DISABLE – BETTER METHOD
BIOS SETTINGS OPTIMIZATIONS
WHAT IS TUNABLE IN BIOS?
HOW TO TUNE ?
http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf
DOES IT REALLY MATTER ?
0
100
200
300
400
500
600
700
800
900
1000
DSS BIOS settings with 1x
DRAM refresh rate
DSS BIOS settings with 4x
DRAM refresh rate
HPC BIOS settings with 4x
DRAM refresh rate
HPC + HyperThreading BIOS
settings with 4x DRAM
refresh rate
HPC - NO TurboBoost BIOS
settings with 4x DRAM
refresh rate
Sec
DSS	
  BIOS	
  se)ngs	
  with	
  1x	
  DRAM	
  refresh	
  rate	
   738.949439	
  
DSS	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   745.111176	
  
HPC	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   552.148285	
  
HPC	
  +	
  HyperThreading	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   877.838469	
  
HPC	
  -­‐	
  NO	
  TurboBoost	
  BIOS	
  se)ngs	
  with	
  4x	
  DRAM	
  refresh	
  rate	
   561.260084	
  
Performance
increase potential
about 40%
WHAT TUNING DOC SAYS ?
STAYING IN THE SAME “TECHNOLOGY SLICE”
WHAT I WILL BE SLICING THROUGH ???
CPU and chipset
Hardware
Operating System (OS)
Database Management System (DBMS)
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3 CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
WHAT IS “TECHNOLOGY SLICE” ANYWAY ???
CPU Gen3 CPU Gen4
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
OS v. 37
DBMS v. 7
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3 CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
COMMON “TECHNOLOGY SLICE” TRAP
CPU Gen3 CPU Gen4
✔
Server Gen-B
OS v. 36
DBMS v. 6
Server Gen-C
✔
OS v. 37
✔
DBMS v. 7
✔
CPU Gen5
Server
Gen-D
CPU Gen6 CPU Gen7
Server Gen-E
Srv
Gen
F
OS v. 38
Server
Gen-A
OS v. 35OS v. 34
DBMS v. 5DBMS v. 4DBMS v. 3
?
?
SYMPTOMS OF “TECHNOLOGY SLICE” ISSUES
System AVG: 57.90
Nice AVG: 46.56
System AVG > Nice AVG
System AVG / Nice AVG = 1.24
System AVG: 11.19
Nice AVG: 57.38
System AVG < Nice AVG
System AVG / Nice AVG = 0.19
“TECHNOLOGY SLICE” PERFORMANCE IMPACT
0
20
40
60
80
100
120
140
different “TECHNOLOGY SLICE” kernel proper “TECHNOLOGY SLICE” kernel
Sec
SUFFICIENT RAM CALCULATIONS
DO I REALLY NEED MORE RAM ?
select event_type, count(1) from query_events group by event_type order by 2 desc;
Spilled events are very good
indication of queries not fitting in
RAM
HOW I CAN QUANTIFY IMPACT ?
select 'event_timestamp' as timestamp_type,
min(event_timestamp) as min_timestamp,
max(event_timestamp) as max_timestamp from query_events
union
select 'query_timestamp' as timestamp_type,
min(start_timestamp) as min_timestamp,
max(start_timestamp) as max_timestamp from query_requests;
System tables in Vertica have
individual rolling window. Make
sure you understand relation of
histories available.
HOW I CAN QUANTIFY IMPACT ? CONT.
select spilled_queries, total_qieries, round( spilled_queries / total_qieries * 100 , 2 ) as spilled_queries_percent
from
(select count(1) as total_qieries from query_requests
where request_type = 'QUERY' and start_timestamp > (select min(event_timestamp) from query_events)) query_data,
(select count(1) as spilled_queries
from (select session_id, transaction_id, statement_id from query_events
where event_type ilike '%SPILLED%' group by session_id, transaction_id, statement_id) spill_data) spill_data2;
Amount of spilled queries in
relation to entire query volume.
CAN MY SPILLED DATA FIT IN TO RAM ?
select min(counter_value) as min_bytes_spilled,
max(counter_value) as max_bytes_spilled,
avg(counter_value) as avg_bytes_spilled
from execution_engine_profiles
where counter_name = 'bytes spilled' and counter_value > 0;
Understanding size of
spillage to disk.
WHO CAUSING SPILLS ?
select user_name, count(1) as spill_event_count
from query_events where event_type ilike '%SPILLED%' group by user_name order by 2 desc;
In Vertica RAM allocated to queries
through resource pools. Resource
pools connected to users. Knowing
user will point us to resource pool,
which needs tuning.
WHAT I SHOULD TUNE ?
select distinct resource_pool from users where user_name in ('peter', 'john');
Identified resource pool with
spilled queries. Now we know
what to tune.
The resource pool parameters of
MEMORYSIZE and
PLANNEDCONCURRENCY provide the
options that let you tune the target
memory allocated to queries.
WHAT I SHOULD CHANGE ?
HP Vertica Analytics Platform Version 7.1.x Documentation
Administrator's Guide
Managing the Database
Managing Workloads
Resource Pool Architecture
Target Memory Determination for Queries in Concurrent Environments
Q & A

Weitere ähnliche Inhalte

Was ist angesagt?

Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writerKyle Hailey
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
 
OOUG: Oracle transaction locking
OOUG: Oracle transaction lockingOOUG: Oracle transaction locking
OOUG: Oracle transaction lockingKyle Hailey
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the DataHao Chen
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streamingRamūnas Urbonas
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksKyle Hailey
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOAltinity Ltd
 
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...Altinity Ltd
 
How to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraHow to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraSveta Smirnova
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouseAltinity Ltd
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL-Consulting
 
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Altinity Ltd
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...InfluxData
 
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEODangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEOAltinity Ltd
 
Oracle: Binding versus caging
Oracle: Binding versus cagingOracle: Binding versus caging
Oracle: Binding versus cagingBertrandDrouvot
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXzznate
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisSveta Smirnova
 

Was ist angesagt? (20)

Profiling the logwriter and database writer
Profiling the logwriter and database writerProfiling the logwriter and database writer
Profiling the logwriter and database writer
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
 
OOUG: Oracle transaction locking
OOUG: Oracle transaction lockingOOUG: Oracle transaction locking
OOUG: Oracle transaction locking
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the Data
 
PostgreSQL and RAM usage
PostgreSQL and RAM usagePostgreSQL and RAM usage
PostgreSQL and RAM usage
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streaming
 
Rac 12c optimization
Rac 12c optimizationRac 12c optimization
Rac 12c optimization
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEOClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
ClickHouse tips and tricks. Webinar slides. By Robert Hodges, Altinity CEO
 
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
ClickHouse Unleashed 2020: Our Favorite New Features for Your Analytical Appl...
 
How to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with GaleraHow to Avoid Pitfalls in Schema Upgrade with Galera
How to Avoid Pitfalls in Schema Upgrade with Galera
 
Your first ClickHouse data warehouse
Your first ClickHouse data warehouseYour first ClickHouse data warehouse
Your first ClickHouse data warehouse
 
PostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQPostgreSQL Meetup Berlin at Zalando HQ
PostgreSQL Meetup Berlin at Zalando HQ
 
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...InfluxDB IOx Tech Talks: The Impossible Dream:  Easy-to-Use, Super Fast Softw...
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
 
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEODangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
 
Oracle: Binding versus caging
Oracle: Binding versus cagingOracle: Binding versus caging
Oracle: Binding versus caging
 
Advanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMXAdvanced Apache Cassandra Operations with JMX
Advanced Apache Cassandra Operations with JMX
 
Using Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data AnalysisUsing Apache Spark and MySQL for Data Analysis
Using Apache Spark and MySQL for Data Analysis
 

Ähnlich wie Extra performance out of thin air

The hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaThe hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaAlluxio, Inc.
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and ArchitectureSidney Chen
 
Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Intel Software Brasil
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New FeaturesAmazon Web Services
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceAltinity Ltd
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxAltinity Ltd
 
Barga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteBarga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteRoger Barga
 
Cassandra Tuning - above and beyond
Cassandra Tuning - above and beyondCassandra Tuning - above and beyond
Cassandra Tuning - above and beyondMatija Gobec
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...DataStax
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamStewart Needham
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterpriseInfluxData
 
Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Matt Warren
 
Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Chris Adkin
 
CMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesCMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesAmazon Web Services
 
Migrating To PostgreSQL
Migrating To PostgreSQLMigrating To PostgreSQL
Migrating To PostgreSQLGrant Fritchey
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent MonitoringIntelie
 
Patterns in the cloud
Patterns in the cloudPatterns in the cloud
Patterns in the cloudDavid Manning
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafkaNitin Kumar
 
Scalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseScalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseChristina Lin
 

Ähnlich wie Extra performance out of thin air (20)

The hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at HelixaThe hidden engineering behind machine learning products at Helixa
The hidden engineering behind machine learning products at Helixa
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 
Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™ Entenda de onde vem toda a potência do Intel® Xeon Phi™
Entenda de onde vem toda a potência do Intel® Xeon Phi™
 
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
(DAT402) Amazon RDS PostgreSQL:Lessons Learned & New Features
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open SourceBuilding an Analytic Extension to MySQL with ClickHouse and Open Source
Building an Analytic Extension to MySQL with ClickHouse and Open Source
 
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptxBuilding an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx
 
Barga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 KeynoteBarga IC2E & IoTDI'16 Keynote
Barga IC2E & IoTDI'16 Keynote
 
Cassandra Tuning - above and beyond
Cassandra Tuning - above and beyondCassandra Tuning - above and beyond
Cassandra Tuning - above and beyond
 
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
Cassandra Tuning - Above and Beyond (Matija Gobec, SmartCat) | Cassandra Summ...
 
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart NeedhamThe post release technologies of Crysis 3 (Slides Only) - Stewart Needham
The post release technologies of Crysis 3 (Slides Only) - Stewart Needham
 
Monitoring InfluxEnterprise
Monitoring InfluxEnterpriseMonitoring InfluxEnterprise
Monitoring InfluxEnterprise
 
Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016Performance and how to measure it - ProgSCon London 2016
Performance and how to measure it - ProgSCon London 2016
 
Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)Column store indexes and batch processing mode (nx power lite)
Column store indexes and batch processing mode (nx power lite)
 
Osol Pgsql
Osol PgsqlOsol Pgsql
Osol Pgsql
 
CMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 InstancesCMP301_Deep Dive on Amazon EC2 Instances
CMP301_Deep Dive on Amazon EC2 Instances
 
Migrating To PostgreSQL
Migrating To PostgreSQLMigrating To PostgreSQL
Migrating To PostgreSQL
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
 
Patterns in the cloud
Patterns in the cloudPatterns in the cloud
Patterns in the cloud
 
Deep learning with kafka
Deep learning with kafkaDeep learning with kafka
Deep learning with kafka
 
Scalable Integration with JBoss Fuse
Scalable Integration with JBoss FuseScalable Integration with JBoss Fuse
Scalable Integration with JBoss Fuse
 

Kürzlich hochgeladen

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 

Kürzlich hochgeladen (20)

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 

Extra performance out of thin air

  • 2. PATH TO EXTRA PERFORMANCE Eliminate unneeded work § Choose data types wisely Eliminate unneeded waits § Reduce number of locks Make system operate in more efficient way § Optimize BIOS settings § Stay in same “technology slice” § Make sure you have enough RAM
  • 3. ART OF CHOOSING DATATYPES
  • 4. WHY DATA TYPE MATTERS ?
  • 5. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation
  • 6. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation “BIG DATA” record set starts from 100 billion records
  • 7. WHY DATA TYPE MATTERS ? Fastest CPU today is 3.7 GHz It takes 1 / 3,700,000,000 of a second to do single operation “BIG DATA” record set starts from 100 billion records Processing time 1 / 3,700,000,000 sec X 100,000,000,000 = 27 sec
  • 8. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ?
  • 9. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17
  • 10. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17
  • 11. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17 Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4) Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15) Storage: Store as integer Data type: INT Internal data type: INT
  • 12. DO YOU NEED TO STORE DATA SAME WAY IT IS PRESENTED ? Presentation: $395.17 Data: 395.17 Storage: Store as Money Data type: MONEY Internal data type: NUMERIC(18,4) Storage: Store as numeric Data type: NUMERIC Internal data type: NUMERIC(37,15) Storage: Store as integer Data type: INT Internal data type: INT
  • 14. DATA TYPE BENCHMARK AVERAGES IN SEC 27.2 29.7 37 0 5 10 15 20 25 30 35 40 INT NUMERIC(18,5) NUMERIC(37,15)
  • 15. MAKING RIGHT CHOICES • If you can store data as INTEGER • Choose INTEGER • If your data fits into 18 digits of PRECISION • Choose NUMERIC(18) • If your data larger then 18 digits of PRECISION • Choose NUMERIC(your-desired-precision) Vertica default for NUMERIC is NUMERIC(37,15)
  • 17. LOCKING BEHAVIOR AUTOCOMMIT = ON (jdbc driver default) § Each statement treated as complete transaction § When statement completes changes automatically committed to database AUTOCOMMIT = OFF § Transaction continue until manually run COMMIT or ROLLBACK § Locks kept on objects for transaction duration
  • 18. CONTROLLING AUTOCOMMIT STATE JAVA: conn = DriverManager.getConnection("jdbc:vertica://DBHost:5433/MyDB", myProperties); // get the state of the auto commit parameter System.out.println("Autocommit state: " + conn.getAutoCommit()); // Change the auto commit state to false conn.setAutoCommit(false); SQL:
  • 19. IMPACT ON LOCK COUNTS BY CHANGING AUTOCOMMIT SETTING TO OFF
  • 20. HOW TO DISABLE – OBVIOUS METHOD
  • 21. HOW TO DISABLE – BETTER METHOD
  • 23. WHAT IS TUNABLE IN BIOS?
  • 24. HOW TO TUNE ? http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf
  • 25. DOES IT REALLY MATTER ? 0 100 200 300 400 500 600 700 800 900 1000 DSS BIOS settings with 1x DRAM refresh rate DSS BIOS settings with 4x DRAM refresh rate HPC BIOS settings with 4x DRAM refresh rate HPC + HyperThreading BIOS settings with 4x DRAM refresh rate HPC - NO TurboBoost BIOS settings with 4x DRAM refresh rate Sec DSS  BIOS  se)ngs  with  1x  DRAM  refresh  rate   738.949439   DSS  BIOS  se)ngs  with  4x  DRAM  refresh  rate   745.111176   HPC  BIOS  se)ngs  with  4x  DRAM  refresh  rate   552.148285   HPC  +  HyperThreading  BIOS  se)ngs  with  4x  DRAM  refresh  rate   877.838469   HPC  -­‐  NO  TurboBoost  BIOS  se)ngs  with  4x  DRAM  refresh  rate   561.260084   Performance increase potential about 40%
  • 26. WHAT TUNING DOC SAYS ?
  • 27. STAYING IN THE SAME “TECHNOLOGY SLICE”
  • 28. WHAT I WILL BE SLICING THROUGH ??? CPU and chipset Hardware Operating System (OS) Database Management System (DBMS)
  • 29. WHAT IS “TECHNOLOGY SLICE” ANYWAY ??? CPU Gen3 CPU Gen4 Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C OS v. 37 DBMS v. 7 CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 30. WHAT IS “TECHNOLOGY SLICE” ANYWAY ??? CPU Gen3 CPU Gen4 Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C OS v. 37 DBMS v. 7 CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 31. COMMON “TECHNOLOGY SLICE” TRAP CPU Gen3 CPU Gen4 ✔ Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C ✔ OS v. 37 ✔ DBMS v. 7 ✔ CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3
  • 32. COMMON “TECHNOLOGY SLICE” TRAP CPU Gen3 CPU Gen4 ✔ Server Gen-B OS v. 36 DBMS v. 6 Server Gen-C ✔ OS v. 37 ✔ DBMS v. 7 ✔ CPU Gen5 Server Gen-D CPU Gen6 CPU Gen7 Server Gen-E Srv Gen F OS v. 38 Server Gen-A OS v. 35OS v. 34 DBMS v. 5DBMS v. 4DBMS v. 3 ? ?
  • 33. SYMPTOMS OF “TECHNOLOGY SLICE” ISSUES System AVG: 57.90 Nice AVG: 46.56 System AVG > Nice AVG System AVG / Nice AVG = 1.24 System AVG: 11.19 Nice AVG: 57.38 System AVG < Nice AVG System AVG / Nice AVG = 0.19
  • 34. “TECHNOLOGY SLICE” PERFORMANCE IMPACT 0 20 40 60 80 100 120 140 different “TECHNOLOGY SLICE” kernel proper “TECHNOLOGY SLICE” kernel Sec
  • 36. DO I REALLY NEED MORE RAM ? select event_type, count(1) from query_events group by event_type order by 2 desc; Spilled events are very good indication of queries not fitting in RAM
  • 37. HOW I CAN QUANTIFY IMPACT ? select 'event_timestamp' as timestamp_type, min(event_timestamp) as min_timestamp, max(event_timestamp) as max_timestamp from query_events union select 'query_timestamp' as timestamp_type, min(start_timestamp) as min_timestamp, max(start_timestamp) as max_timestamp from query_requests; System tables in Vertica have individual rolling window. Make sure you understand relation of histories available.
  • 38. HOW I CAN QUANTIFY IMPACT ? CONT. select spilled_queries, total_qieries, round( spilled_queries / total_qieries * 100 , 2 ) as spilled_queries_percent from (select count(1) as total_qieries from query_requests where request_type = 'QUERY' and start_timestamp > (select min(event_timestamp) from query_events)) query_data, (select count(1) as spilled_queries from (select session_id, transaction_id, statement_id from query_events where event_type ilike '%SPILLED%' group by session_id, transaction_id, statement_id) spill_data) spill_data2; Amount of spilled queries in relation to entire query volume.
  • 39. CAN MY SPILLED DATA FIT IN TO RAM ? select min(counter_value) as min_bytes_spilled, max(counter_value) as max_bytes_spilled, avg(counter_value) as avg_bytes_spilled from execution_engine_profiles where counter_name = 'bytes spilled' and counter_value > 0; Understanding size of spillage to disk.
  • 40. WHO CAUSING SPILLS ? select user_name, count(1) as spill_event_count from query_events where event_type ilike '%SPILLED%' group by user_name order by 2 desc; In Vertica RAM allocated to queries through resource pools. Resource pools connected to users. Knowing user will point us to resource pool, which needs tuning.
  • 41. WHAT I SHOULD TUNE ? select distinct resource_pool from users where user_name in ('peter', 'john'); Identified resource pool with spilled queries. Now we know what to tune.
  • 42. The resource pool parameters of MEMORYSIZE and PLANNEDCONCURRENCY provide the options that let you tune the target memory allocated to queries. WHAT I SHOULD CHANGE ? HP Vertica Analytics Platform Version 7.1.x Documentation Administrator's Guide Managing the Database Managing Workloads Resource Pool Architecture Target Memory Determination for Queries in Concurrent Environments
  • 43. Q & A