Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1trOqf6. Aviran Mordo introduces Wix's architecture, a highly available eventually consistent system, along with patterns for rendering many websites with a relatively small number of servers. Filmed at qconlondon.com. Aviran Mordo is the head of back-end engineering at Wix. He has over 20 years of experience in the software industry and has filled many engineering roles and leading positions, from designing and building the US National Electronic Records Archives prototype to building the infrastructure that powers indeed.com, the leading job search engine in the world.

1. Wix Architecture at Scale
Aviran Mordo
Head of Back-End Engineering @ Wix
@aviranm
linkedin.com/in/aviran
aviransplace.com

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/wix-architecture
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

5. Wix in Numbers
Over 45,000,000 users
1M new users/month
Static storage is >800TB of data
1.5TB new files/day
3 data centers + 2 clouds (Google, Amazon)
300 servers
700M HTTP requests/day
600 people work at Wix, of which ~ 200 in R&D

6. Initial Architecture
Tomcat, Hibernate, custom web framework
Built for fast development
Stateful login (Tomcat session), Ehcache, file uploads
No consideration for performance, scalability and testing
Intended for short-term use
Lighttpd
(file serving) MySQL
DB
Wix
(Tomcat)

7. The Monolithic Giant
One monolithic server that handled everything
Dependency between features
Changes in unrelated areas of the system caused deployment
of the whole system
Failure in unrelated areas will cause system wide downtime

8. Breaking the System Apart

9. Concerns and SLA
Edit websites
Data Validation
Security / Authentication
Data consistency
Lots of data
Serving Media
High availability
High performance
Lots of static files
Very high traffic volume
Viewport optimization
Cacheable data
View sites, created
by Wix editor
High availability
High performance
High traffic volume
Long tail

10. Wix Segmentation
Networking
1. Editor Segment 2. Media Segment 3. Public Segment

11. Making SOA Guidelines
Each service has its own database (if one is needed)
Only one service can write to a specific DB
There may be additional read-only services that directly
accesses the DB (for performance reasons)
Services are stateless
No DB transactions
Cache is not a building block, but an optimization

12. 1. Editor Segment

13. Editor Server
Immutable JSON pages (~2.5M / day)
Site revisions
Active – standby MySQL cross datacenters
Editor Server
MySQL
Active
Sites
MySQL
Archive

15. Protect The Data
Protect against DB outage with fast recovery = replication
Protect against data poisoning/corruption = revisions / backup
Make the data available at all times = data distribution to
multiple locations / providers

16. Saving Editor Data
Browser
Editor
Server
Static
Grid
Notify
Google
Cloud
Storage
MySQL
Active
Sites
MySQL
Archive
Notify
Archive
(Amazon)
Archive
(Google)
Save Page(s)
200 OK
Upload
Save Page
DC replication
Download Page
MySQL
Active
Sites
MySQL
Archive

17. Self Healing Process
Browser
Editor
Server
Static
Grid
Save Page(s)
Save Page
Upload
Notify
Download Page
Google
Cloud
Storage
MySQL
Archive
MySQL
Active
Sites
MySQL
Archive
DC replication
Notify
Archive
(Amazon)
Archive
(Google)
MySQL
Active
Sites
200 OK

18. No DB Transactions
Save each page (JSON) as an atomic operation
Page ID is a content based hash (immutable/idempotent)
Finalize transaction by sending site header (list of pages)
Can generate orphaned pages, not a problem in practice

19. 2. Media Segment

20. Prospero – Wix Media Storage
800TB user media files
3M files uploaded daily
500M metadata records
Dynamic media processing
• Picture resize, crop and sharpen “on the fly”
• Watermark
• Audio format conversion

21. Prospero
Eventual consistent distributed file system
Multi datacenter aware
Automatic fallback cross DC
Run on commodity servers & cloud

22. Prospero – Wix Media Manager
Austin
x36
T
x36
T
x32
get image.jpg
First
fallback
Second
fallback
If not in
CDN
Google
Cloud
Tampa
x36
T
x36
T
x32
CDN

23. 3. Public Segment

24. Public Segment Roles
Routing (resolve URLs)
Dispatching (to a renderer)
Rendering (HTML,XML,TXT)
Public
Server
HTML
Renderer
HTML
SEO
Renderer
Flash
Renderer
Sitemap
Renderer
Robots.txt
Renderer
www.example.com
Flash
SEO
Renderer

25. Public SLA
Response time <100ms at peak traffic

26. Publish A Site
Publish site header (a map of pages for a site)
Publish routing table
Publish site header / routes
Editor Segment Public Segment

27. Built For Speed
Minimize out-of-service hops (2 DB, 1 RPC)
Lookup tables are cached in memory, updated every 5 minutes
Denormalized data – optimize for read by primary key (MySQL)
Minimize business logic

28. How a Page Gets Rendered
Bootstrap HTML template that contains only data
Only JavaScript imports
JSON data (site-header + dynamic data)
No “real” HTML view

29. Offload rendering work to the browser

30. The average Intel
Core i750 can push
up to 7 GFLOPS
without overclocking

31. Why JSON?
Easy to parse in JavaScript and Java/Scala
Fairly compact text format
Highly compressible (5:1 even for small payloads)
Easy to fix rendering bugs (just deploy a new client code)

32. Minimum Number of Public Servers
Needed to Serve 45M Sites
4

33. Public SLA
Be Available 99.99999%

34. Serving a Site – Sunny Day
CDN Statics
Archive
Browser
http://example.wix.com
HTTP
Request
Notify
site view
Store HTML
to cache
LB
Public
Renderer
HTML
Resources / Media
HTTP
Request

35. Serving a Site – DC Lost
CDN Statics
Archive
Browser
http://example.wix.com
LB
Public
Renderer
LB
Public
Renderer
Change DNS
HTTP
Request

36. Serving a Site – Public Lost
CDN Statics
Archive
Browser
http://example.wix.com
LB
Public
Renderer
Get
Cached HTML
Version
HTML
HTTP
Request

37. Living in the Browser
CDN Statics
Archive
Browser
http://example.wix.com
LB
Public
Renderer
Fallback
Editor
JSON /
Media
HTML
HTTP
Request
Fallback

38. Summary
Identify your critical path and concerns
Build redundancy in critical path (for availability)
De-normalize data (for performance)
Minimize out-of-process hops (for performance)
Take advantage of client’s CPU power

40. Q&A
http://goo.gl/Oo3lGr
Aviran Mordo
Head of Back-End Engineering @ Wix
@aviranm
linkedin.com/in/aviran
aviransplace.com

42. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/wix-architecture