SlideShare ist ein Scribd-Unternehmen logo
1 von 25
Downloaden Sie, um offline zu lesen
MAGENTO
SCALABILITY
from the trenches
Piotr Karwatka
AGENDA
1. General scalability rules
2. Action Plan – scalability framework
3. Magento B2B case
1. EAV and indexes,
2. Cache
3. Replication
4. Fine-tuning
4. Magento 2.0
2
THE CHALLENGE
- Good architecture – a rare good,
- There is no holy grail of scalability,
- Always take custom approach – measure before
optimizing,
- Start “cheap”, scale fast – risky
- Processes driven over improvisation,
- Redundancy – scalability goes with availability
- Divide and conquer – using layers
- Measure and examine bottlenecks,
- Scale only overloaded layers
- Good news: Magento is scalable by design
3
middleware
cache
storage
app
db
HARDWARE APPROACH
At start – optimize code & use cache (New Relic,
collected to catch bottlenecks); try HHVM, nginx,
OpCache
Vertical: more RAM, more CPUs
+ no code changes required, fast gain
- technology barriers,
- at some point very expensive
Horizontal: more cheap servers
+ high availability when done right,
+ cloud ready,
- often requires code refactoring,
- challenging configuration and dev-ops
4
Cost at scale
ACTION PLAN
Step 1
- use vertical scaling as far as it’s reasonable,
- optimize code to avoid bottlenecks,
- use caching where it’s possible,
- separate database server
- separate static files or/and use CDN,
Step 2
- add additional app servers,
- establish cache cluster,
- use reverse proxy (Varnish)
Step 3
- use database replication,
- scale up using horizontal scaling 5
First go vertical
Then go horizontal
MAGENTO CASE – THE CHALLENGE
TIM.PL – largest B2B site in Poland. About 100 000 000EUR /
year
Platform for customers – offers/inquiries, bulk orders, near real-time
CRM/WMS integration
6
- B2B e-Commerce site with external
integrations (CRM, PIM, ERP, WMS)
- Up to 1.5M SKU’s,
- Up to 2K active concurrent users,
average session time: 4h+,
- About 6000 attributes,
- About 2189 attribute sets,
- 1M+ website calls / day,
- Challenging read/write ratio: 50/50%
- B2B features, site used as
tool/platform; browse/checkout
scenario
We called it MVP.
It worked well to some point...
7
FIRST APPROACH – 3 years ago
- Cache for blocks enabled,
- FLAT enabled – but at 5000+ attributes InnoDB limits achieved,
- The code was optimized quite well (we’ve used Ivan’s tips: http://www.
slideshare.net/ivanchepurnyi/making-magento-flying-like-a-rocket-a-set-of-
valuable-tips-for-developers)
- Separated DB server + master-master replication (backup purposes),
- SSD disks (APP + DB), lot of RAM (16GB / server) – vertical scaling
approach,
- MySQL tuning (IO buffers, InnoDB buffers),
- Apache tuning (connection limits, FPM)
- HHVM tested – about +50% boost, but no profiling
8
OPTIMIZE AND PROFILE!
Always measure impact of change before implementing it to production
- JMeter – we used it to emulate throughput and conduct load tests after each
change,
- New Relic – to analyze application speed, track slow-queries and method-calls;
it can be used on production servers as well because of near-zero overhead
9
- Collectd – installed on both app and db servers –
we’ve discovered bottlenecks on IO and db-locking
on Magento’s product indexation,
- Logs – we used ELK (Kibana) and custom New
Relic integration to diagnose web-services
response times,
- htop, iotop – during IO problems it can be useful
to find what generates the problem exactly,
- Xdebug/XHProf profiler - on stage servers to
debug and profile code and discover cache gaps,
JMeter
2h load
tests
Fine
tuning
JMeter
24h load
tests
Optimize
one piece
at time
High availability is crucial – we switched to 2N model
10
master master
App servers + GlusterFS
both servers can handle user reqs.
Haproxy + Varnish – load balancer
load balancing and reverse proxy for caching and static files
APP & CACHE
- Redis is faster than memcached as backend cache,
- Varnish (with ESI) is a must for both static files and page caching (we used
Turpentine and Phoenix on some projects – both are fine) - VCL can be challenging,
- We managed to use HAProxy as load balancer (using automatic failover),
- We’ve added cache to Mage_Catalog_Model_Product::load
- Consider adding cache to Mage_Eav_Model_Entity_Abstract to avoid EAV at all – we
couldn’t use FLAT because of attributes count,
- We turned on FLAT to 900 most frequently used attributes (InnoDb limits),
- Sessions were moved to Redis,
- We discovered lot of queries to core_url_rewrite - cache should help here,
- We used Fast-Async Reindexing module while using Magento 1.x to avoid
database locking
- GlusterFS used to handle uploads and replication
11
VARNISH IMPACT
12
APP & CACHE
- Remarks
- GlusterFS/network file systems – stat(), open() without local caching are IO
exhausting,
- we had some issues with APC on PHP 5.4 (segfaults) – now everybody uses
OpCache ☺
- at some point we switched from Apache to nginx + php-fpm to gain speed req/s
throughput and lower memory usage (read more here: http://info.magento.
com/rs/magentocommerce/images/MagentoECG-
PoweringMagentowithNgnixandPHP-FPM.pdf)
- We had problems with Magento API (really slow responses – 0.5s);
optimizations = 0.2s + HHVM = 0.1s; next step – fast responding façade without
Magento overhead - http://divante.co/blog/magento-1-9-1-0-page-load-time-0-3s/
- We had problems with Redis clogging with cache Keys (http://divante.
co/blog/magento-clogged-redis-cache/)
13
HHVM IMPACT
14
THE HARD WAY
- Most challenging issues: EAV and indexing
- Will be great to use NoSQL DB (MongoDB, SOLR),
- At this point we use only model-level cache,
- We’ve disabled Magento logs and reports – less queries, less
useless data to store,
- Small configuration tips make big difference:
- query_cache_size - up to 128MB works well; furthermore – cache cleaning can
be really, REALLY slow
- innodb_thread_concurrency - setting to 0 prevents MySQL from clogging
worker threads (looks like it’s locking but it isn’t)
- We switched from MySQL to PerconaDB/XtraDB
- Great gain performance gain on peaks – queries count vs.
response time – up to + 275%,
- No code / SQL changes required – 100% compatible with
MySQL,
- MemSQL – looks really promising, not tested yet
15
DATABASE CAVEATS
16
Without FLAT in place – lot of EAV-related quires, also lot of URL-redirect related
queries. Those queries are unnecessary.
HOW TO DISABLE EAV?
– it will be great if we can switch to NoSQL DB (like MongoDB,
SOLR, Sphinx Search),
– one can overwrite EAV->FLAT indexers but it’s extremely hard
(relations, some modules works on RAW SQL),
– suggestions:
- Add cache to Product::load method – invalidation is
extremely important (you can use modification date in
cache-key or observer based mechanism to clear it up),
- Add cache to load EAV attributes – for products, product
categories,
- Overwrite/refactor Mage_Catalog – for searching and
browsing products – some search modules do this partially,
- Great knowledge base about EAV: http://www.
solvingmagento.com/magento-eav-system/
17
If you cannot use FLAT (categories + products are must) – it’s too slow or you
have too many attributes
DATABASE SCALABILITY - REPLICATION
With replicas one gets: high availability, more req/s.
It doesn’t fit all cases:
Caution: replication-lags
It’s possible to move selected tables to
external servers (like product catalogs).
Always consider using cache first!
18
:-
)
:-
(
master slave
mastermaster
master
master
master
TB: users
TB: photos
INDEXATION VS. REPLICATION
- Master-slave replication shall help
with db-locking issue;
- MySQL replicates only
UPDATE/INSERT operations using
binlogs
- this is extremely fast and doesn’t
lock replicas
19
public function processEntityAction(Varien_Object $entity,
$entityType, $eventType)
...
$resourceModel = Mage::getResourceSingleton
('index/process');
$resourceModel->beginTransaction();
$this->_allowTableChanges = false;
try {
$this->indexEvent($event);
$resourceModel->commit();
} catch (Exception $e) {
$resourceModel->rollBack();
if ($allowTableChanges) {
$this->_allowTableChanges = true;
$this->_changeKeyStatus(true);
$this->_currentEvent = null;
}
throw $e;
DATABASE – NEXT STEPS
- We’ve tested app-local master-slave replication to avoid network
latency and database-locking
– Magento supports this kind of replication out of the box,
– Next step – move catalog database to separate server,
– Route Admin panel requests to separated servers (using multi-
master Magento2 feature)
20
master master
App servers + GlusterFS + PerconaDB
local db-slave’s for read access
Each server can handle user requestsHaproxy & Varnish
load balancer + proxy
Indexing, updates,
Imports, RDBM
INTEGRATIONS
- We use queuing to avoid bottlenecks,
- On each app server there are Gearman workers
(PHP processes) – responsible for getting prices,
stocks, transferring orders,
- Workers exchange data with CRM, WMS, ERP,
PIM in both async and sync modes – using
priorities,
- We used Command/Task design pattern,
- We log everything using ELK – especially
Kibana and New Relic to analyze external
systems
- Magento API can be very challenging (it’s
extremely slow)
21
MONITORING
We use Kibana (ELK stack) and custom New Relic metrics to monitor real-time
integrations (CRM, WMS, ERP)
Zabbix with Sellenium scripts is used to monitor and alert website availability
22
FINAL ARCHITECTURE
23
master master
App servers + GlusterFS + PerconaDB
local db-slave’s for read access
Each server can handle user requests
Haproxy & Varnish
load balancer + proxy
Gearman queue workers
handle background jobs and external
integrations
API calls
Web requests
External sys. Calls
background jobs
WHAT I’VE MISSED + MAGENTO 2
- Search – we used FactFinder / SOLR,
- Details about Varnish and HHVM
- Life is going to be easier: What excites me in Magento2?
– Materialized views engine – smarter indexation,
– Full page caching in community,
– Multi master DB contexts,
– Checkout optimizations
24
THANK YOU! QUESTIONS?
25
Technical or scalability challenges?
Contact me to consult your case for free!
Piotr Karwatka (pkarwatka@divante.pl)
Divante – http://divante.co

Weitere ähnliche Inhalte

Was ist angesagt?

Speeding up your WordPress Site - WordCamp Toronto 2015
Speeding up your WordPress Site - WordCamp Toronto 2015Speeding up your WordPress Site - WordCamp Toronto 2015
Speeding up your WordPress Site - WordCamp Toronto 2015Alan Lok
 
Magento Performance Optimization 101
Magento Performance Optimization 101Magento Performance Optimization 101
Magento Performance Optimization 101Angus Li
 
How to reduce database load using Memcache
How to reduce database load using MemcacheHow to reduce database load using Memcache
How to reduce database load using Memcachevaluebound
 
JS digest. Decemebr 2017
JS digest. Decemebr 2017JS digest. Decemebr 2017
JS digest. Decemebr 2017ElifTech
 
Magento performance
Magento performanceMagento performance
Magento performanceDivante
 
PHP and MySQL
PHP and MySQLPHP and MySQL
PHP and MySQLbmani
 
Speeding up your WordPress site - WordCamp Hamilton 2015
Speeding up your WordPress site - WordCamp Hamilton 2015Speeding up your WordPress site - WordCamp Hamilton 2015
Speeding up your WordPress site - WordCamp Hamilton 2015Alan Lok
 
Improve WordPress performance with caching and deferred execution of code
Improve WordPress performance with caching and deferred execution of codeImprove WordPress performance with caching and deferred execution of code
Improve WordPress performance with caching and deferred execution of codeDanilo Ercoli
 
Grav CMS
Grav CMSGrav CMS
Grav CMSbtopro
 
Piecing Together the WordPress Puzzle
Piecing Together the WordPress PuzzlePiecing Together the WordPress Puzzle
Piecing Together the WordPress PuzzleBusiness Vitality LLC
 
WordPress Development Tools and Best Practices
WordPress Development Tools and Best PracticesWordPress Development Tools and Best Practices
WordPress Development Tools and Best PracticesDanilo Ercoli
 
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages ApplicationsIBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages Applicationsbeglee
 
Mobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressMobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressDanilo Ercoli
 
Wordcamp2009
Wordcamp2009Wordcamp2009
Wordcamp2009joetek
 
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATION
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATIONEXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATION
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATIONMysql User Camp
 
WordCamp RVA 2011 - Performance & Tuning
WordCamp RVA 2011 - Performance & TuningWordCamp RVA 2011 - Performance & Tuning
WordCamp RVA 2011 - Performance & TuningTimothy Wood
 
PHP Enhancement with Windows Server 2008
PHP Enhancement with Windows Server 2008PHP Enhancement with Windows Server 2008
PHP Enhancement with Windows Server 2008Krit Kamtuo
 
Anthony Somerset - Site Speed = Success!
Anthony Somerset - Site Speed = Success!Anthony Somerset - Site Speed = Success!
Anthony Somerset - Site Speed = Success!WordCamp Cape Town
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson
 

Was ist angesagt? (20)

Speeding up your WordPress Site - WordCamp Toronto 2015
Speeding up your WordPress Site - WordCamp Toronto 2015Speeding up your WordPress Site - WordCamp Toronto 2015
Speeding up your WordPress Site - WordCamp Toronto 2015
 
Magento Performance Optimization 101
Magento Performance Optimization 101Magento Performance Optimization 101
Magento Performance Optimization 101
 
How to reduce database load using Memcache
How to reduce database load using MemcacheHow to reduce database load using Memcache
How to reduce database load using Memcache
 
JS digest. Decemebr 2017
JS digest. Decemebr 2017JS digest. Decemebr 2017
JS digest. Decemebr 2017
 
Magento performance
Magento performanceMagento performance
Magento performance
 
PHP and MySQL
PHP and MySQLPHP and MySQL
PHP and MySQL
 
Speeding up your WordPress site - WordCamp Hamilton 2015
Speeding up your WordPress site - WordCamp Hamilton 2015Speeding up your WordPress site - WordCamp Hamilton 2015
Speeding up your WordPress site - WordCamp Hamilton 2015
 
Improve WordPress performance with caching and deferred execution of code
Improve WordPress performance with caching and deferred execution of codeImprove WordPress performance with caching and deferred execution of code
Improve WordPress performance with caching and deferred execution of code
 
Grav CMS
Grav CMSGrav CMS
Grav CMS
 
Piecing Together the WordPress Puzzle
Piecing Together the WordPress PuzzlePiecing Together the WordPress Puzzle
Piecing Together the WordPress Puzzle
 
WordPress Development Tools and Best Practices
WordPress Development Tools and Best PracticesWordPress Development Tools and Best Practices
WordPress Development Tools and Best Practices
 
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages ApplicationsIBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
 
Mobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressMobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPress
 
Wordcamp2009
Wordcamp2009Wordcamp2009
Wordcamp2009
 
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATION
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATIONEXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATION
EXPERIENCE WITH MYSQL HA SOLUTION AND GROUP REPLICATION
 
WordCamp RVA 2011 - Performance & Tuning
WordCamp RVA 2011 - Performance & TuningWordCamp RVA 2011 - Performance & Tuning
WordCamp RVA 2011 - Performance & Tuning
 
PHP Enhancement with Windows Server 2008
PHP Enhancement with Windows Server 2008PHP Enhancement with Windows Server 2008
PHP Enhancement with Windows Server 2008
 
The RDBMS You Should Be Using
The RDBMS You Should Be UsingThe RDBMS You Should Be Using
The RDBMS You Should Be Using
 
Anthony Somerset - Site Speed = Success!
Anthony Somerset - Site Speed = Success!Anthony Somerset - Site Speed = Success!
Anthony Somerset - Site Speed = Success!
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 

Andere mochten auch

Omnichannel Customer Experience
Omnichannel Customer ExperienceOmnichannel Customer Experience
Omnichannel Customer ExperienceDivante
 
Surprising failure factors when implementing eCommerce and Omnichannel eBusiness
Surprising failure factors when implementing eCommerce and Omnichannel eBusinessSurprising failure factors when implementing eCommerce and Omnichannel eBusiness
Surprising failure factors when implementing eCommerce and Omnichannel eBusinessDivante
 
Market Research Project on Retail Loyalty Cards
Market Research Project on Retail Loyalty CardsMarket Research Project on Retail Loyalty Cards
Market Research Project on Retail Loyalty CardsDeepali Agarwal
 
E-Commerce Integration and Implementation Issues
E-Commerce Integration and Implementation IssuesE-Commerce Integration and Implementation Issues
E-Commerce Integration and Implementation IssuesNurul Izzah
 
Growth hacking dla e-Commerce
Growth hacking dla e-CommerceGrowth hacking dla e-Commerce
Growth hacking dla e-CommerceDivante
 
Customer Experience Workshop
Customer Experience WorkshopCustomer Experience Workshop
Customer Experience Workshopmcrucera
 
Searching higher up the funnel
Searching higher up the funnelSearching higher up the funnel
Searching higher up the funnelJono Alderson
 
All You Need to Know About Customer Journey Mapping
All You Need to Know About Customer Journey MappingAll You Need to Know About Customer Journey Mapping
All You Need to Know About Customer Journey MappingRealtimeBoard
 
Venture Design Workshop: Business Model Canvas
Venture Design Workshop: Business Model CanvasVenture Design Workshop: Business Model Canvas
Venture Design Workshop: Business Model CanvasAlex Cowan
 

Andere mochten auch (9)

Omnichannel Customer Experience
Omnichannel Customer ExperienceOmnichannel Customer Experience
Omnichannel Customer Experience
 
Surprising failure factors when implementing eCommerce and Omnichannel eBusiness
Surprising failure factors when implementing eCommerce and Omnichannel eBusinessSurprising failure factors when implementing eCommerce and Omnichannel eBusiness
Surprising failure factors when implementing eCommerce and Omnichannel eBusiness
 
Market Research Project on Retail Loyalty Cards
Market Research Project on Retail Loyalty CardsMarket Research Project on Retail Loyalty Cards
Market Research Project on Retail Loyalty Cards
 
E-Commerce Integration and Implementation Issues
E-Commerce Integration and Implementation IssuesE-Commerce Integration and Implementation Issues
E-Commerce Integration and Implementation Issues
 
Growth hacking dla e-Commerce
Growth hacking dla e-CommerceGrowth hacking dla e-Commerce
Growth hacking dla e-Commerce
 
Customer Experience Workshop
Customer Experience WorkshopCustomer Experience Workshop
Customer Experience Workshop
 
Searching higher up the funnel
Searching higher up the funnelSearching higher up the funnel
Searching higher up the funnel
 
All You Need to Know About Customer Journey Mapping
All You Need to Know About Customer Journey MappingAll You Need to Know About Customer Journey Mapping
All You Need to Know About Customer Journey Mapping
 
Venture Design Workshop: Business Model Canvas
Venture Design Workshop: Business Model CanvasVenture Design Workshop: Business Model Canvas
Venture Design Workshop: Business Model Canvas
 

Ähnlich wie Magento scalability from the trenches (Meet Magento Sweden 2016)

AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09Chris Purrington
 
High Performance Web Sites
High Performance Web SitesHigh Performance Web Sites
High Performance Web SitesRavi Raj
 
One daytalk hbraun_oct2011
One daytalk hbraun_oct2011One daytalk hbraun_oct2011
One daytalk hbraun_oct2011hbraun
 
Extending Piwik At R7.com
Extending Piwik At R7.comExtending Piwik At R7.com
Extending Piwik At R7.comLeo Lorieri
 
01 demystifying mysq-lfororacledbaanddeveloperv1
01 demystifying mysq-lfororacledbaanddeveloperv101 demystifying mysq-lfororacledbaanddeveloperv1
01 demystifying mysq-lfororacledbaanddeveloperv1Ivan Ma
 
murakumo Cloud Controller
murakumo Cloud Controllermurakumo Cloud Controller
murakumo Cloud ControllerShingo Kawano
 
MySQL Performance - Best practices
MySQL Performance - Best practices MySQL Performance - Best practices
MySQL Performance - Best practices Ted Wennmark
 
KubeCon 2017: Kubernetes from Dev to Prod
KubeCon 2017: Kubernetes from Dev to ProdKubeCon 2017: Kubernetes from Dev to Prod
KubeCon 2017: Kubernetes from Dev to ProdSubhas Dandapani
 
Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012Wim Godden
 
Fundamentals of performance tuning PHP on IBM i
Fundamentals of performance tuning PHP on IBM i  Fundamentals of performance tuning PHP on IBM i
Fundamentals of performance tuning PHP on IBM i Zend by Rogue Wave Software
 
PLNOG Automation@Brainly
PLNOG Automation@BrainlyPLNOG Automation@Brainly
PLNOG Automation@Brainlyvespian_256
 
PLNOG14: Automation at Brainly - Paweł Rozlach
PLNOG14: Automation at Brainly - Paweł RozlachPLNOG14: Automation at Brainly - Paweł Rozlach
PLNOG14: Automation at Brainly - Paweł RozlachPROIDEA
 
OOW09 Ebs Tuning Final
OOW09 Ebs Tuning FinalOOW09 Ebs Tuning Final
OOW09 Ebs Tuning Finaljucaab
 
Integrating best of breed open source tools to vitess orchestrator pleu21
Integrating best of breed open source tools to vitess  orchestrator   pleu21Integrating best of breed open source tools to vitess  orchestrator   pleu21
Integrating best of breed open source tools to vitess orchestrator pleu21Alkin Tezuysal
 
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...varien
 
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...MagentoImagine
 
PHP Performance tuning for Drupal 8
PHP Performance tuning for Drupal 8PHP Performance tuning for Drupal 8
PHP Performance tuning for Drupal 8Acquia
 
Caching and tuning fun for high scalability @ FrOSCon 2011
Caching and tuning fun for high scalability @ FrOSCon 2011Caching and tuning fun for high scalability @ FrOSCon 2011
Caching and tuning fun for high scalability @ FrOSCon 2011Wim Godden
 

Ähnlich wie Magento scalability from the trenches (Meet Magento Sweden 2016) (20)

AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09
 
High Performance Web Sites
High Performance Web SitesHigh Performance Web Sites
High Performance Web Sites
 
One daytalk hbraun_oct2011
One daytalk hbraun_oct2011One daytalk hbraun_oct2011
One daytalk hbraun_oct2011
 
Extending Piwik At R7.com
Extending Piwik At R7.comExtending Piwik At R7.com
Extending Piwik At R7.com
 
01 demystifying mysq-lfororacledbaanddeveloperv1
01 demystifying mysq-lfororacledbaanddeveloperv101 demystifying mysq-lfororacledbaanddeveloperv1
01 demystifying mysq-lfororacledbaanddeveloperv1
 
Kash Kubernetified
Kash KubernetifiedKash Kubernetified
Kash Kubernetified
 
murakumo Cloud Controller
murakumo Cloud Controllermurakumo Cloud Controller
murakumo Cloud Controller
 
MySQL Performance - Best practices
MySQL Performance - Best practices MySQL Performance - Best practices
MySQL Performance - Best practices
 
KubeCon 2017: Kubernetes from Dev to Prod
KubeCon 2017: Kubernetes from Dev to ProdKubeCon 2017: Kubernetes from Dev to Prod
KubeCon 2017: Kubernetes from Dev to Prod
 
Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012Caching and tuning fun for high scalability @ FOSDEM 2012
Caching and tuning fun for high scalability @ FOSDEM 2012
 
Fundamentals of performance tuning PHP on IBM i
Fundamentals of performance tuning PHP on IBM i  Fundamentals of performance tuning PHP on IBM i
Fundamentals of performance tuning PHP on IBM i
 
ESIGate dev meeting #4 21-11-2013
ESIGate dev meeting #4 21-11-2013ESIGate dev meeting #4 21-11-2013
ESIGate dev meeting #4 21-11-2013
 
PLNOG Automation@Brainly
PLNOG Automation@BrainlyPLNOG Automation@Brainly
PLNOG Automation@Brainly
 
PLNOG14: Automation at Brainly - Paweł Rozlach
PLNOG14: Automation at Brainly - Paweł RozlachPLNOG14: Automation at Brainly - Paweł Rozlach
PLNOG14: Automation at Brainly - Paweł Rozlach
 
OOW09 Ebs Tuning Final
OOW09 Ebs Tuning FinalOOW09 Ebs Tuning Final
OOW09 Ebs Tuning Final
 
Integrating best of breed open source tools to vitess orchestrator pleu21
Integrating best of breed open source tools to vitess  orchestrator   pleu21Integrating best of breed open source tools to vitess  orchestrator   pleu21
Integrating best of breed open source tools to vitess orchestrator pleu21
 
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
 
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
 
PHP Performance tuning for Drupal 8
PHP Performance tuning for Drupal 8PHP Performance tuning for Drupal 8
PHP Performance tuning for Drupal 8
 
Caching and tuning fun for high scalability @ FrOSCon 2011
Caching and tuning fun for high scalability @ FrOSCon 2011Caching and tuning fun for high scalability @ FrOSCon 2011
Caching and tuning fun for high scalability @ FrOSCon 2011
 

Mehr von Divante

The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup	The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup Divante
 
eCommerce Trends 2020
eCommerce Trends 2020eCommerce Trends 2020
eCommerce Trends 2020Divante
 
Async & Bulk REST API new possibilities of communication between systems
Async & Bulk REST API new possibilities of communication  between systemsAsync & Bulk REST API new possibilities of communication  between systems
Async & Bulk REST API new possibilities of communication between systemsDivante
 
Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Divante
 
Die Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDie Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDivante
 
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 progressive web apps - pwa as a game changer for e-commerce - meet magento i... progressive web apps - pwa as a game changer for e-commerce - meet magento i...
progressive web apps - pwa as a game changer for e-commerce - meet magento i...Divante
 
Customer churn - how to stop it?
Customer churn - how to stop it?Customer churn - how to stop it?
Customer churn - how to stop it?Divante
 
eCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coeCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coDivante
 
How to create a Vue Storefront theme
How to create a Vue Storefront themeHow to create a Vue Storefront theme
How to create a Vue Storefront themeDivante
 
Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Divante
 
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechVue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechDivante
 
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...Divante
 
eCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coeCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coDivante
 
Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Divante
 
Why is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosWhy is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosDivante
 
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationvue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationDivante
 
Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Divante
 
Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Divante
 
The biggest stores on Magento
The biggest stores on MagentoThe biggest stores on Magento
The biggest stores on MagentoDivante
 
B2B Commerce - how to become successful
B2B Commerce - how to become successfulB2B Commerce - how to become successful
B2B Commerce - how to become successfulDivante
 

Mehr von Divante (20)

The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup	The eCommerce Platforms in the Global Setup
The eCommerce Platforms in the Global Setup
 
eCommerce Trends 2020
eCommerce Trends 2020eCommerce Trends 2020
eCommerce Trends 2020
 
Async & Bulk REST API new possibilities of communication between systems
Async & Bulk REST API new possibilities of communication  between systemsAsync & Bulk REST API new possibilities of communication  between systems
Async & Bulk REST API new possibilities of communication between systems
 
Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...Magento Functional Testing Framework a way to seriously write automated tests...
Magento Functional Testing Framework a way to seriously write automated tests...
 
Die Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der ModernbrancheDie Top 10 Progressive Web Apps in der Modernbranche
Die Top 10 Progressive Web Apps in der Modernbranche
 
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 progressive web apps - pwa as a game changer for e-commerce - meet magento i... progressive web apps - pwa as a game changer for e-commerce - meet magento i...
progressive web apps - pwa as a game changer for e-commerce - meet magento i...
 
Customer churn - how to stop it?
Customer churn - how to stop it?Customer churn - how to stop it?
Customer churn - how to stop it?
 
eCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.coeCommerce trends 2019 by Divante.co
eCommerce trends 2019 by Divante.co
 
How to create a Vue Storefront theme
How to create a Vue Storefront themeHow to create a Vue Storefront theme
How to create a Vue Storefront theme
 
Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa Game changer for e-commerce - Vue Storefront - open source pwa
Game changer for e-commerce - Vue Storefront - open source pwa
 
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speechVue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
Vue Storefront - Progressive Web App for Magento (1.9, 2.x) - MM18DE speech
 
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...How to successfully onboard  end-clients to a B2B Platform - Magento Imagine ...
How to successfully onboard end-clients to a B2B Platform - Magento Imagine ...
 
eCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.coeCommerce trends from 2017 to 2018 by Divante.co
eCommerce trends from 2017 to 2018 by Divante.co
 
Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)Designing for PWA (Progressive Web Apps)
Designing for PWA (Progressive Web Apps)
 
Why is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenariosWhy is crud a bad idea - focus on real scenarios
Why is crud a bad idea - focus on real scenarios
 
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentationvue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
vue-storefront - PWA eCommerce for Magento2 MM17NYC presentation
 
Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5Pimcore Overview - Pimcore5
Pimcore Overview - Pimcore5
 
Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5Pimcore E-Commerce Framework - Pimcore5
Pimcore E-Commerce Framework - Pimcore5
 
The biggest stores on Magento
The biggest stores on MagentoThe biggest stores on Magento
The biggest stores on Magento
 
B2B Commerce - how to become successful
B2B Commerce - how to become successfulB2B Commerce - how to become successful
B2B Commerce - how to become successful
 

Kürzlich hochgeladen

Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
Niche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusNiche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusSkylark Nobin
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdfShreedeep Rayamajhi
 
world Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxworld Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxnaveenithkrishnan
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpressssuser166378
 
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxA_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxjayshuklatrainer
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilitiesalihassaah1994
 

Kürzlich hochgeladen (15)

Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
Niche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus BonusNiche Domination Prodigy Review Plus Bonus
Niche Domination Prodigy Review Plus Bonus
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
 
world Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptxworld Tuberculosis day ppt 25-3-2024.pptx
world Tuberculosis day ppt 25-3-2024.pptx
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpress
 
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptxA_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
A_Z-1_0_4T_00A-EN_U-Po_w_erPoint_06.pptx
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilities
 

Magento scalability from the trenches (Meet Magento Sweden 2016)

  • 2. AGENDA 1. General scalability rules 2. Action Plan – scalability framework 3. Magento B2B case 1. EAV and indexes, 2. Cache 3. Replication 4. Fine-tuning 4. Magento 2.0 2
  • 3. THE CHALLENGE - Good architecture – a rare good, - There is no holy grail of scalability, - Always take custom approach – measure before optimizing, - Start “cheap”, scale fast – risky - Processes driven over improvisation, - Redundancy – scalability goes with availability - Divide and conquer – using layers - Measure and examine bottlenecks, - Scale only overloaded layers - Good news: Magento is scalable by design 3 middleware cache storage app db
  • 4. HARDWARE APPROACH At start – optimize code & use cache (New Relic, collected to catch bottlenecks); try HHVM, nginx, OpCache Vertical: more RAM, more CPUs + no code changes required, fast gain - technology barriers, - at some point very expensive Horizontal: more cheap servers + high availability when done right, + cloud ready, - often requires code refactoring, - challenging configuration and dev-ops 4 Cost at scale
  • 5. ACTION PLAN Step 1 - use vertical scaling as far as it’s reasonable, - optimize code to avoid bottlenecks, - use caching where it’s possible, - separate database server - separate static files or/and use CDN, Step 2 - add additional app servers, - establish cache cluster, - use reverse proxy (Varnish) Step 3 - use database replication, - scale up using horizontal scaling 5 First go vertical Then go horizontal
  • 6. MAGENTO CASE – THE CHALLENGE TIM.PL – largest B2B site in Poland. About 100 000 000EUR / year Platform for customers – offers/inquiries, bulk orders, near real-time CRM/WMS integration 6 - B2B e-Commerce site with external integrations (CRM, PIM, ERP, WMS) - Up to 1.5M SKU’s, - Up to 2K active concurrent users, average session time: 4h+, - About 6000 attributes, - About 2189 attribute sets, - 1M+ website calls / day, - Challenging read/write ratio: 50/50% - B2B features, site used as tool/platform; browse/checkout scenario
  • 7. We called it MVP. It worked well to some point... 7
  • 8. FIRST APPROACH – 3 years ago - Cache for blocks enabled, - FLAT enabled – but at 5000+ attributes InnoDB limits achieved, - The code was optimized quite well (we’ve used Ivan’s tips: http://www. slideshare.net/ivanchepurnyi/making-magento-flying-like-a-rocket-a-set-of- valuable-tips-for-developers) - Separated DB server + master-master replication (backup purposes), - SSD disks (APP + DB), lot of RAM (16GB / server) – vertical scaling approach, - MySQL tuning (IO buffers, InnoDB buffers), - Apache tuning (connection limits, FPM) - HHVM tested – about +50% boost, but no profiling 8
  • 9. OPTIMIZE AND PROFILE! Always measure impact of change before implementing it to production - JMeter – we used it to emulate throughput and conduct load tests after each change, - New Relic – to analyze application speed, track slow-queries and method-calls; it can be used on production servers as well because of near-zero overhead 9 - Collectd – installed on both app and db servers – we’ve discovered bottlenecks on IO and db-locking on Magento’s product indexation, - Logs – we used ELK (Kibana) and custom New Relic integration to diagnose web-services response times, - htop, iotop – during IO problems it can be useful to find what generates the problem exactly, - Xdebug/XHProf profiler - on stage servers to debug and profile code and discover cache gaps, JMeter 2h load tests Fine tuning JMeter 24h load tests Optimize one piece at time
  • 10. High availability is crucial – we switched to 2N model 10 master master App servers + GlusterFS both servers can handle user reqs. Haproxy + Varnish – load balancer load balancing and reverse proxy for caching and static files
  • 11. APP & CACHE - Redis is faster than memcached as backend cache, - Varnish (with ESI) is a must for both static files and page caching (we used Turpentine and Phoenix on some projects – both are fine) - VCL can be challenging, - We managed to use HAProxy as load balancer (using automatic failover), - We’ve added cache to Mage_Catalog_Model_Product::load - Consider adding cache to Mage_Eav_Model_Entity_Abstract to avoid EAV at all – we couldn’t use FLAT because of attributes count, - We turned on FLAT to 900 most frequently used attributes (InnoDb limits), - Sessions were moved to Redis, - We discovered lot of queries to core_url_rewrite - cache should help here, - We used Fast-Async Reindexing module while using Magento 1.x to avoid database locking - GlusterFS used to handle uploads and replication 11
  • 13. APP & CACHE - Remarks - GlusterFS/network file systems – stat(), open() without local caching are IO exhausting, - we had some issues with APC on PHP 5.4 (segfaults) – now everybody uses OpCache ☺ - at some point we switched from Apache to nginx + php-fpm to gain speed req/s throughput and lower memory usage (read more here: http://info.magento. com/rs/magentocommerce/images/MagentoECG- PoweringMagentowithNgnixandPHP-FPM.pdf) - We had problems with Magento API (really slow responses – 0.5s); optimizations = 0.2s + HHVM = 0.1s; next step – fast responding façade without Magento overhead - http://divante.co/blog/magento-1-9-1-0-page-load-time-0-3s/ - We had problems with Redis clogging with cache Keys (http://divante. co/blog/magento-clogged-redis-cache/) 13
  • 15. THE HARD WAY - Most challenging issues: EAV and indexing - Will be great to use NoSQL DB (MongoDB, SOLR), - At this point we use only model-level cache, - We’ve disabled Magento logs and reports – less queries, less useless data to store, - Small configuration tips make big difference: - query_cache_size - up to 128MB works well; furthermore – cache cleaning can be really, REALLY slow - innodb_thread_concurrency - setting to 0 prevents MySQL from clogging worker threads (looks like it’s locking but it isn’t) - We switched from MySQL to PerconaDB/XtraDB - Great gain performance gain on peaks – queries count vs. response time – up to + 275%, - No code / SQL changes required – 100% compatible with MySQL, - MemSQL – looks really promising, not tested yet 15
  • 16. DATABASE CAVEATS 16 Without FLAT in place – lot of EAV-related quires, also lot of URL-redirect related queries. Those queries are unnecessary.
  • 17. HOW TO DISABLE EAV? – it will be great if we can switch to NoSQL DB (like MongoDB, SOLR, Sphinx Search), – one can overwrite EAV->FLAT indexers but it’s extremely hard (relations, some modules works on RAW SQL), – suggestions: - Add cache to Product::load method – invalidation is extremely important (you can use modification date in cache-key or observer based mechanism to clear it up), - Add cache to load EAV attributes – for products, product categories, - Overwrite/refactor Mage_Catalog – for searching and browsing products – some search modules do this partially, - Great knowledge base about EAV: http://www. solvingmagento.com/magento-eav-system/ 17 If you cannot use FLAT (categories + products are must) – it’s too slow or you have too many attributes
  • 18. DATABASE SCALABILITY - REPLICATION With replicas one gets: high availability, more req/s. It doesn’t fit all cases: Caution: replication-lags It’s possible to move selected tables to external servers (like product catalogs). Always consider using cache first! 18 :- ) :- ( master slave mastermaster master master master TB: users TB: photos
  • 19. INDEXATION VS. REPLICATION - Master-slave replication shall help with db-locking issue; - MySQL replicates only UPDATE/INSERT operations using binlogs - this is extremely fast and doesn’t lock replicas 19 public function processEntityAction(Varien_Object $entity, $entityType, $eventType) ... $resourceModel = Mage::getResourceSingleton ('index/process'); $resourceModel->beginTransaction(); $this->_allowTableChanges = false; try { $this->indexEvent($event); $resourceModel->commit(); } catch (Exception $e) { $resourceModel->rollBack(); if ($allowTableChanges) { $this->_allowTableChanges = true; $this->_changeKeyStatus(true); $this->_currentEvent = null; } throw $e;
  • 20. DATABASE – NEXT STEPS - We’ve tested app-local master-slave replication to avoid network latency and database-locking – Magento supports this kind of replication out of the box, – Next step – move catalog database to separate server, – Route Admin panel requests to separated servers (using multi- master Magento2 feature) 20 master master App servers + GlusterFS + PerconaDB local db-slave’s for read access Each server can handle user requestsHaproxy & Varnish load balancer + proxy Indexing, updates, Imports, RDBM
  • 21. INTEGRATIONS - We use queuing to avoid bottlenecks, - On each app server there are Gearman workers (PHP processes) – responsible for getting prices, stocks, transferring orders, - Workers exchange data with CRM, WMS, ERP, PIM in both async and sync modes – using priorities, - We used Command/Task design pattern, - We log everything using ELK – especially Kibana and New Relic to analyze external systems - Magento API can be very challenging (it’s extremely slow) 21
  • 22. MONITORING We use Kibana (ELK stack) and custom New Relic metrics to monitor real-time integrations (CRM, WMS, ERP) Zabbix with Sellenium scripts is used to monitor and alert website availability 22
  • 23. FINAL ARCHITECTURE 23 master master App servers + GlusterFS + PerconaDB local db-slave’s for read access Each server can handle user requests Haproxy & Varnish load balancer + proxy Gearman queue workers handle background jobs and external integrations API calls Web requests External sys. Calls background jobs
  • 24. WHAT I’VE MISSED + MAGENTO 2 - Search – we used FactFinder / SOLR, - Details about Varnish and HHVM - Life is going to be easier: What excites me in Magento2? – Materialized views engine – smarter indexation, – Full page caching in community, – Multi master DB contexts, – Checkout optimizations 24
  • 25. THANK YOU! QUESTIONS? 25 Technical or scalability challenges? Contact me to consult your case for free! Piotr Karwatka (pkarwatka@divante.pl) Divante – http://divante.co