Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Wix Architecture at Scale

Aviran Mordo
Head of Back-End Engineering @ Wix

@aviranm
linkedin.com/in/aviran

aviransplace....
Wix in Numbers
Over 45,000,000 users
1M new users/month
Static storage is >800TB of data
1.5TB new files/day

3 data cente...
Initial Architecture
Tomcat, Hibernate, custom web framework
Built for fast development
Stateful login (Tomcat session), E...
The Monolithic Giant
One monolithic server that handled everything
Dependency between features
Changes in unrelated areas ...
Breaking the System Apart
Concerns and SLA
Edit websites
Data Validation
Security /
Authentication
Data consistency
Lots of data

View sites, create...
Wix Segmentation
Networking

1. Editor Segment

2. Media Segment

3. Public Segment
Making SOA Guidelines
Each service has its own database (if one is needed)
Only one service can write to a specific DB

Th...
1. Editor
Segment
Editor Server
Immutable JSON pages (~2.5M / day)
Site revisions
Active – standby MySQL cross datacenters

Editor Server

M...
Protect The Data
Protect against DB outage with fast recovery = replication
Protect against data poisoning/corruption = re...
Saving Editor Data
Browser

Save
Page(s)
200 OK

Editor
Server

Upload

Static
Grid

Notify
Save
Page

MySQL
Active
Sites
...
Self Healing Process
Browser

Save
Page(s)
200 OK

Editor
Server

Upload

Static
Grid

Notify
Save
Page

MySQL
Active
Site...
No DB Transactions
Save each page (JSON) as an atomic operation
Page ID is a content based hash (immutable/idempotent)
Fin...
2. Media
Segment
Prospero – Wix Media Storage
800TB user media files
3M files uploaded daily
500M metadata records
Dynamic media processing...
Prospero
Eventual consistent distributed file system
Multi datacenter aware
Automatic fallback cross DC

Run on commodity ...
Prospero – Wix Media Manager
Tampa

Google
Cloud

x36
x36
T x32
T

Second
fallback

First
fallback

Austin
CDN
get image.j...
3. Public
Segment
Public Segment Roles
Routing (resolve URLs)

www.example.com

HTML
Renderer

Dispatching (to a renderer)

HTML
SEO
Rendere...
Public SLA
Response time <100ms at peak traffic
Publish A Site
Publish site header (a map of pages for a site)
Publish routing table

Publish site header / routes
Editor ...
Built For Speed
Minimize out-of-service hops (2 DB, 1 RPC)
Lookup tables are cached in memory, updated every 5
minutes
Den...
How a Page Gets Rendered
Bootstrap HTML template that contains only data
Only JavaScript imports
JSON data (site-header + ...
Offload rendering work to the
browser
The average Intel
Core i750 can push
up to 7 GFLOPS
without
overclocking
Why JSON?
Easy to parse in JavaScript and Java/Scala
Fairly compact text format
Highly compressible (5:1 even for small pa...
Minimum Number of Public
Servers Needed to Serve 45M
Sites

4
Public SLA
Be Available 99.99999%
Serving a Site – Sunny Day
Browser
http://example.wix.com
HTTP
Request

HTML

Resources /
Media

CDN

Notify
site view
HTT...
Serving a Site – DC Lost
Browser
http://example.wix.com

CDN

Statics

HTTP
Request

LB

Archiv
e

LB

Public

Public

Ren...
Serving a Site – Public Lost
Browser
http://example.wix.com
HTTP
Request

CDN

HTML
Get
Cached HTML
Version

LB

Public

R...
Living in the Browser
Browser
http://example.wix.com
HTTP
Request

Fallback

Editor

LB

Public

Rendere
r

HTML

JSON /
M...
Summary
Identify your critical path and concerns
Build redundancy in critical path (for availability)
De-normalize data (f...
Q&A
http://goo.gl/Oo3lGr

Aviran Mordo
Head of Back-End Engineering @ Wix

@aviranm
linkedin.com/in/aviran

aviransplace.c...
Wix Architecture at Scale - QCon London 2014
Wix Architecture at Scale - QCon London 2014
Wix Architecture at Scale - QCon London 2014
Wix Architecture at Scale - QCon London 2014
Upcoming SlideShare
Loading in …5
×

of

Wix Architecture at Scale - QCon London 2014 Slide 1 Wix Architecture at Scale - QCon London 2014 Slide 2 Wix Architecture at Scale - QCon London 2014 Slide 3 Wix Architecture at Scale - QCon London 2014 Slide 4 Wix Architecture at Scale - QCon London 2014 Slide 5 Wix Architecture at Scale - QCon London 2014 Slide 6 Wix Architecture at Scale - QCon London 2014 Slide 7 Wix Architecture at Scale - QCon London 2014 Slide 8 Wix Architecture at Scale - QCon London 2014 Slide 9 Wix Architecture at Scale - QCon London 2014 Slide 10 Wix Architecture at Scale - QCon London 2014 Slide 11 Wix Architecture at Scale - QCon London 2014 Slide 12 Wix Architecture at Scale - QCon London 2014 Slide 13 Wix Architecture at Scale - QCon London 2014 Slide 14 Wix Architecture at Scale - QCon London 2014 Slide 15 Wix Architecture at Scale - QCon London 2014 Slide 16 Wix Architecture at Scale - QCon London 2014 Slide 17 Wix Architecture at Scale - QCon London 2014 Slide 18 Wix Architecture at Scale - QCon London 2014 Slide 19 Wix Architecture at Scale - QCon London 2014 Slide 20 Wix Architecture at Scale - QCon London 2014 Slide 21 Wix Architecture at Scale - QCon London 2014 Slide 22 Wix Architecture at Scale - QCon London 2014 Slide 23 Wix Architecture at Scale - QCon London 2014 Slide 24 Wix Architecture at Scale - QCon London 2014 Slide 25 Wix Architecture at Scale - QCon London 2014 Slide 26 Wix Architecture at Scale - QCon London 2014 Slide 27 Wix Architecture at Scale - QCon London 2014 Slide 28 Wix Architecture at Scale - QCon London 2014 Slide 29 Wix Architecture at Scale - QCon London 2014 Slide 30 Wix Architecture at Scale - QCon London 2014 Slide 31 Wix Architecture at Scale - QCon London 2014 Slide 32 Wix Architecture at Scale - QCon London 2014 Slide 33 Wix Architecture at Scale - QCon London 2014 Slide 34 Wix Architecture at Scale - QCon London 2014 Slide 35 Wix Architecture at Scale - QCon London 2014 Slide 36 Wix Architecture at Scale - QCon London 2014 Slide 37 Wix Architecture at Scale - QCon London 2014 Slide 38 Wix Architecture at Scale - QCon London 2014 Slide 39
Upcoming SlideShare
Scaling up to 30M users - The Wix Story
Next
Download to read offline and view in fullscreen.

11 Likes

Share

Download to read offline

Wix Architecture at Scale - QCon London 2014

Download to read offline

In this talk I will go over Wix's architecture, how we evolved our system to be highly available even at the worst case scenarios when everything can break, how we built a self-healing eventual consistency system for website data distribution and will show some of the patterns we use that helps us render 45M websites while maintaining a relatively low number of servers.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Wix Architecture at Scale - QCon London 2014

  1. 1. Wix Architecture at Scale Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com
  2. 2. Wix in Numbers Over 45,000,000 users 1M new users/month Static storage is >800TB of data 1.5TB new files/day 3 data centers + 2 clouds (Google, Amazon) 300 servers 700M HTTP requests/day 600 people work at Wix, of which ~ 200 in R&D
  3. 3. Initial Architecture Tomcat, Hibernate, custom web framework Built for fast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability and testing Intended for short-term use Lighttpd (file serving) Wix (Tomcat) MySQL DB
  4. 4. The Monolithic Giant One monolithic server that handled everything Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime
  5. 5. Breaking the System Apart
  6. 6. Concerns and SLA Edit websites Data Validation Security / Authentication Data consistency Lots of data View sites, created by Wix editor High availability Serving Media High availability High performance High performance High traffic volume Lots of static files Long tail Very high traffic volume Viewport optimization Cacheable data
  7. 7. Wix Segmentation Networking 1. Editor Segment 2. Media Segment 3. Public Segment
  8. 8. Making SOA Guidelines Each service has its own database (if one is needed) Only one service can write to a specific DB There may be additional read-only services that directly accesses the DB (for performance reasons) Services are stateless No DB transactions Cache is not a building block, but an optimization
  9. 9. 1. Editor Segment
  10. 10. Editor Server Immutable JSON pages (~2.5M / day) Site revisions Active – standby MySQL cross datacenters Editor Server MySQL Active Sites MySQL Archive
  11. 11. Protect The Data Protect against DB outage with fast recovery = replication Protect against data poisoning/corruption = revisions / backup Make the data available at all times = data distribution to multiple locations / providers
  12. 12. Saving Editor Data Browser Save Page(s) 200 OK Editor Server Upload Static Grid Notify Save Page MySQL Active Sites MySQL Archive DC replication MySQL Active Sites MySQL Archive Download Page Notify Archive (Google) Google Cloud Storage Archive (Amazon)
  13. 13. Self Healing Process Browser Save Page(s) 200 OK Editor Server Upload Static Grid Notify Save Page MySQL Active Sites MySQL Archive DC replication MySQL Active Sites MySQL Archive Download Page Notify Archive (Google) Google Cloud Storage Archive (Amazon)
  14. 14. No DB Transactions Save each page (JSON) as an atomic operation Page ID is a content based hash (immutable/idempotent) Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice
  15. 15. 2. Media Segment
  16. 16. Prospero – Wix Media Storage 800TB user media files 3M files uploaded daily 500M metadata records Dynamic media processing • Picture resize, crop and sharpen “on the fly” • Watermark • Audio format conversion
  17. 17. Prospero Eventual consistent distributed file system Multi datacenter aware Automatic fallback cross DC Run on commodity servers & cloud
  18. 18. Prospero – Wix Media Manager Tampa Google Cloud x36 x36 T x32 T Second fallback First fallback Austin CDN get image.jpg If not in CDN x36 x36 T x32 T
  19. 19. 3. Public Segment
  20. 20. Public Segment Roles Routing (resolve URLs) www.example.com HTML Renderer Dispatching (to a renderer) HTML SEO Renderer Flash SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer Rendering (HTML,XML,TXT) Public Server
  21. 21. Public SLA Response time <100ms at peak traffic
  22. 22. Publish A Site Publish site header (a map of pages for a site) Publish routing table Publish site header / routes Editor Segment Public Segment
  23. 23. Built For Speed Minimize out-of-service hops (2 DB, 1 RPC) Lookup tables are cached in memory, updated every 5 minutes Denormalized data – optimize for read by primary key (MySQL) Minimize business logic
  24. 24. How a Page Gets Rendered Bootstrap HTML template that contains only data Only JavaScript imports JSON data (site-header + dynamic data) No “real” HTML view
  25. 25. Offload rendering work to the browser
  26. 26. The average Intel Core i750 can push up to 7 GFLOPS without overclocking
  27. 27. Why JSON? Easy to parse in JavaScript and Java/Scala Fairly compact text format Highly compressible (5:1 even for small payloads) Easy to fix rendering bugs (just deploy a new client code)
  28. 28. Minimum Number of Public Servers Needed to Serve 45M Sites 4
  29. 29. Public SLA Be Available 99.99999%
  30. 30. Serving a Site – Sunny Day Browser http://example.wix.com HTTP Request HTML Resources / Media CDN Notify site view HTTP Request LB Store HTML to cache Public Rendere r Archiv e Statics
  31. 31. Serving a Site – DC Lost Browser http://example.wix.com CDN Statics HTTP Request LB Archiv e LB Public Public Rendere r Rendere r Change DNS
  32. 32. Serving a Site – Public Lost Browser http://example.wix.com HTTP Request CDN HTML Get Cached HTML Version LB Public Rendere r Archiv e Statics
  33. 33. Living in the Browser Browser http://example.wix.com HTTP Request Fallback Editor LB Public Rendere r HTML JSON / Media CDN Fallback Archiv e Statics
  34. 34. Summary Identify your critical path and concerns Build redundancy in critical path (for availability) De-normalize data (for performance) Minimize out-of-process hops (for performance) Take advantage of client’s CPU power
  35. 35. Q&A http://goo.gl/Oo3lGr Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com
  • kr2smile

    Aug. 25, 2018
  • PrinceRaj30

    Oct. 6, 2015
  • hamidramazani

    Apr. 2, 2015
  • sumekey

    Mar. 3, 2015
  • sheeplogh

    Feb. 23, 2015
  • puinha

    Nov. 18, 2014
  • karenfornari

    Nov. 14, 2014
  • TonyFabeen

    Nov. 14, 2014
  • omergar

    Nov. 4, 2014
  • Ramzi_Alqrainy

    May. 21, 2014
  • rubale

    Mar. 9, 2014

In this talk I will go over Wix's architecture, how we evolved our system to be highly available even at the worst case scenarios when everything can break, how we built a self-healing eventual consistency system for website data distribution and will show some of the patterns we use that helps us render 45M websites while maintaining a relatively low number of servers.

Views

Total views

31,910

On Slideshare

0

From embeds

0

Number of embeds

114

Actions

Downloads

85

Shares

0

Comments

0

Likes

11

×