SlideShare ist ein Scribd-Unternehmen logo
1 von 41
Downloaden Sie, um offline zu lesen
Flickr and PHP
Cal Henderson
What’s Flickr
• Photo sharing
• Open APIs
Logical Architecture
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Physical Architecture
Static Servers Database Servers Node Servers
Web Servers
Users
Where is PHP?
Page Logic
Application Logic
DatabasePhoto Storage
API
EndpointsTemplates
3rd Party Apps Flickr Apps
Node Service
Flickr.comEmail
Users
Other than PHP?
• Smarty for templating
• PEAR for XML and Email parsing
• Perl for controlling…
• ImageMagick, for image processing
• MySQL (4.0 / InnoDb)
• Java, for the node service
• Apache 2, Redhat, etc. etc.
Big Application?
• One programmer, one designer, etc.
• ~60,000 lines of PHP code
• ~60,000 lines of templates
• ~70 custom smarty functions/modifiers
• ~25,000 DB transactions/second at peak
• ~1000 pages per second at peak
Thinking outside the web app
• Services
– Atom/RSS/RDF Feeds
– APIs
• SOAP
• XML-RPC
• REST
• PEAR::XML::Tree
More cool stuff
• Email interface
– Postfix
– PHP
– PEAR::Mail::mimeDecode
• FTP
• Uploading API
• Authentication API
• Unicode
Even more stuff
• Real time application
• Cool flash apps
• Blogging
– Blogger API (1 & 2)
– Metaweblog API
– Atom
– LiveJournal
APIs are simple!
• Modeled on XML-RPC (Sort of)
• Method calls with XML responses
• SOAP, XML-RPC and REST are just transports
• PHP endpoints mean we can use the same application
logic as the website
XML isn’t simple :(
• PHP 4 doesn’t have good a XML parser
• Expat is cool though (PEAR::XML::Parser)
• Why doesn’t PEAR have XPath?
– Because PEAR is stupid!
– PHP 4 sucks!
I love XPath
if ($tree->root->name == 'methodResponse'){
if (($tree->root->children[0]->name == 'params')
&& ($tree->root->children[0]->children[0]->name == 'param')
&& ($tree->root->children[0]->children[0]->children[0]->name == 'value')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array')
&& ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){
$rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0];
}
if ($tree->root->children[0]->name == 'fault'){
$fault = $tree->root->children[0];
return $fault;
}
}
$nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()');
if (count($nodes)){
$rsp = array_pop($nodes);
}else{
list($fault) = $tree->select_nodes('/methodResponse/fault');
return $fault;
}
Creating API methods
• Stateless method-call APIs are easy to extend
• Adding a method requires no knowledge of the transport
• Adding a method once makes it available to all the
interfaces
• Self documenting
Red Hot Unicode Action
• UTF-8 pages
• CJKV support
• It’s really cool
flickr's architecture & php
Unicode for all
• It’s really easy
– Don’t need PHP support
– Don’t need MySQL support
– Just need the right headers
– UTF-8 is 7-bit transparent
– (Just don’t mess with high characters)
• Don’t use HtmlEntities()!
• But bear in mind…
• JavaScript has patchy Unicode support
• People using your APIs might be stupid
Scaling the beast
• Why PHP is great
• MySQL scaling
• Search scaling
• Horizontal scaling
Why PHP is great
• Stateless
– We can bounce people around servers
– Everything is stored in the database
– Even the smarty cache
– “Shared nothing”
– (so long as we avoid PHP sessions)
MySQL Scaling
• Our database server started to slow
• Load of 200
• Replication!
MySQL Replication
• But it only gives you more SELECT’s
• Else you need to partition vertically
• Re-architecting sucks :(
Looking at usage
• Snapshot of db1.flickr.com
– SELECT’s 44,220,588
– INSERT’s 1,349,234
– UPDATE’s 1,755,503
– DELETE’s 318,439
– 13 SELECT’s per I/U/D
Replication is really cool
• A bunch of slave servers handle all the SELECT’s
• A single master just handles I/U/D’s
• It can scale horizontally, at least for a while.
Searching
• A simple text search
• We were using RLIKE
• Then switched to LIKE
• Then disabled it all together
FULLTEXT Indexes
• MySQL saves the day!
• But they’re only supported my MyISAM tables
• We use InnoDb, because it’s a lot faster
• We’re doomed
But wait!
• Partial replication saves the day
• Replicate the portion of the database we want to search.
• But change the table types on the slave to MyISAM
• It can keep up because it’s only handling I/U/D’s on a
couple of tables
• And we can reduce the I/U/D’s with a little bit of vertical
partitioning
JOIN’s are slow
• Normalised data is for sissies
• Keep multiple copies of data around
• Makes searching faster
• Have to ensure consistency in the application logic
Our current setup
Slave Farm
DB1
Master I/U/D’s
SELECT’s
Search Slave
Farm
Search
SELECT’s
DB3
Main Search
slave
DB2
Main Slave
Horizontal scaling
• At the core of our design
• Just add hardware!
• Inexpensive
• Not exponential
• Avoid redesigns
Talking to the Node Service
• Everyone speaks XML (badly)
• Just TCP/IP - fsockopen()
• We’re issuing commands, not requesting data, so we
don’t bother to parse the response
– Just substring search for state=“ok”
• Don’t rely on it!
RSS / Atom / RDF
• Different formats
• All quite bad
• We’re generating a lot of different feeds
• Abstract the difference away using templates
• No good way to do private feeds. Why is nobody working
on this? (WSSE maybe?)
Receiving email
• Want users to be able to email photos to Flickr
• Just get postfix to pipe each mail to a PHP script
• Parse the mail and find any photos
• Cellular phone companies hate you
• Lots of mailers are retarded
– Photos as text/plain attachments :/
Upload via FTP
• PHP isn’t so great at being a daemon
• Leaks memory like a sieve
• No threads
• Java to the rescue
• Java just acts as an FTPd and passes all uploaded files
to PHP for processing
• (This isn’t actually public)
• Bricolage does this I think. Maybe Zope?
Blogs
• Why does everyone loves blogs so much?
• Only a few APIs really
– Blogger
– Metaweblog
– Blogger2
– Movable Type
– Atom
– Live Journal
It’s all broken
• Lots of blog software has broken interfaces
• It’s a support nightmare
• Manila is tricky
• But it all works, more or less
• Abstracted in the application logic
• We just call blogs_post_message();
Back to those APIs
• We opened up the Flickr APIs a few weeks ago
• Programmers mainly build tools for other programmers
• We have Perl, python, PHP, ActionScript, XMLHTTP and
.NET interface libraries
• But also a few actual applications
Flickr Rainbow
Tag Wallpaper
So what next?
• Much more scaling
• PHP 5?
• MySQL 5?
• Taking over the world
Flickr and PHP
Cal Henderson
Any Questions?

Weitere ähnliche Inhalte

Was ist angesagt?

PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databaseBarry Jones
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPressTaylor Lovett
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettextNgoc Dao
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web AppsTakuto Wada
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP PerspectiveBarry Jones
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPressTaylor Lovett
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Manish Pandit
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfAlfresco Software
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucenewolf4ood
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Taylor Lovett
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013Curtiss Grymala
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseTaylor Lovett
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsTaylor Lovett
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Wen-Tien Chang
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013Curtiss Grymala
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open SourceOpusVL
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHPTaylor Lovett
 

Was ist angesagt? (19)

Apache Jackrabbit
Apache JackrabbitApache Jackrabbit
Apache Jackrabbit
 
PostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty databasePostgreSQL - It's kind've a nifty database
PostgreSQL - It's kind've a nifty database
 
JSON REST API for WordPress
JSON REST API for WordPressJSON REST API for WordPress
JSON REST API for WordPress
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
 
Reviewing RESTful Web Apps
Reviewing RESTful Web AppsReviewing RESTful Web Apps
Reviewing RESTful Web Apps
 
Go from a PHP Perspective
Go from a PHP PerspectiveGo from a PHP Perspective
Go from a PHP Perspective
 
The JSON REST API for WordPress
The JSON REST API for WordPressThe JSON REST API for WordPress
The JSON REST API for WordPress
 
Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2Building Apis in Scala with Playframework2
Building Apis in Scala with Playframework2
 
PLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring SurfPLAT-7 Spring Web Scripts and Spring Surf
PLAT-7 Spring Web Scripts and Spring Surf
 
OrientDB & Lucene
OrientDB & LuceneOrientDB & Lucene
OrientDB & Lucene
 
Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch Transforming WordPress Search and Query Performance with Elasticsearch
Transforming WordPress Search and Query Performance with Elasticsearch
 
WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013WordPress Themes 101 - HighEdWeb New England 2013
WordPress Themes 101 - HighEdWeb New England 2013
 
Best Practices for WordPress in Enterprise
Best Practices for WordPress in EnterpriseBest Practices for WordPress in Enterprise
Best Practices for WordPress in Enterprise
 
Best Practices for Building WordPress Applications
Best Practices for Building WordPress ApplicationsBest Practices for Building WordPress Applications
Best Practices for Building WordPress Applications
 
Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3Service-Oriented Design and Implement with Rails3
Service-Oriented Design and Implement with Rails3
 
WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013WordPress Themes 101 - dotEduGuru Summit 2013
WordPress Themes 101 - dotEduGuru Summit 2013
 
Introduction to Apache Solr
Introduction to Apache SolrIntroduction to Apache Solr
Introduction to Apache Solr
 
Software Development with Open Source
Software Development with Open SourceSoftware Development with Open Source
Software Development with Open Source
 
You Got React.js in My PHP
You Got React.js in My PHPYou Got React.js in My PHP
You Got React.js in My PHP
 

Andere mochten auch

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approachesadunne
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architectureElizabeth Smith
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the WebCal Henderson
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid TemplatesAbhishek Shah
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love SongAbhishek Shah
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Jennifer Bell
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLJoe Stump
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Aaron Saray
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture phpcefour
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture AppDynamics
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tigerElizabeth Smith
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government DataJennifer Bell
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Fwdays
 
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravelwajrcs
 

Andere mochten auch (20)

Scalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and ApproachesScalable Web Architectures: Common Patterns and Approaches
Scalable Web Architectures: Common Patterns and Approaches
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
Php internal architecture
Php internal architecturePhp internal architecture
Php internal architecture
 
Building Big on the Web
Building Big on the WebBuilding Big on the Web
Building Big on the Web
 
Fifty Rock Solid Templates
Fifty Rock Solid TemplatesFifty Rock Solid Templates
Fifty Rock Solid Templates
 
Head Over Heals - My Love Song
Head Over Heals - My Love SongHead Over Heals - My Love Song
Head Over Heals - My Love Song
 
Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)Benefits of Open Government Data (Expanded)
Benefits of Open Government Data (Expanded)
 
Nature
NatureNature
Nature
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
 
Going Places
Going PlacesGoing Places
Going Places
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
 
6 3 tier architecture php
6 3 tier architecture php6 3 tier architecture php
6 3 tier architecture php
 
Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture Anatomy of a Modern PHP Application Architecture
Anatomy of a Modern PHP Application Architecture
 
Thassos - Greece
Thassos - GreeceThassos - Greece
Thassos - Greece
 
Taming the resource tiger
Taming the resource tigerTaming the resource tiger
Taming the resource tiger
 
Benefits of Open Government Data
Benefits of Open Government DataBenefits of Open Government Data
Benefits of Open Government Data
 
Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"Sep Nasiri "Upwork PHP Architecture"
Sep Nasiri "Upwork PHP Architecture"
 
Aids Out Life In
Aids Out Life InAids Out Life In
Aids Out Life In
 
Slab
SlabSlab
Slab
 
Domain Driven Design using Laravel
Domain Driven Design using LaravelDomain Driven Design using Laravel
Domain Driven Design using Laravel
 

Ähnlich wie flickr's architecture & php

Flickr Php
Flickr PhpFlickr Php
Flickr Phproyans
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentationweb25
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Hendersonkangaro10a
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Servicesroyans
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Servicesroyans
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfPatiento Del Mar
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1Stefan Schmidt
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesBrett Meyer
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the CloudTony Tam
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approachOren Eini
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Aaron Blythe
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?DATAVERSITY
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101Samantha Geitz
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoopbddmoscow
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeDavid Boike
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineDavid Keener
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlySarah Guido
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerIke Ellis
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionJeff Smith
 

Ähnlich wie flickr's architecture & php (20)

Flickr Php
Flickr PhpFlickr Php
Flickr Php
 
Flickr Architecture Presentation
Flickr Architecture PresentationFlickr Architecture Presentation
Flickr Architecture Presentation
 
Flickr and PHP - Cal Henderson
Flickr and PHP - Cal HendersonFlickr and PHP - Cal Henderson
Flickr and PHP - Cal Henderson
 
Top ten-list
Top ten-listTop ten-list
Top ten-list
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
Flickr Services
Flickr ServicesFlickr Services
Flickr Services
 
hibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdfhibernateormfeatures-140223193044-phpapp02.pdf
hibernateormfeatures-140223193044-phpapp02.pdf
 
New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1New Persistence Features in Spring Roo 1.1
New Persistence Features in Spring Roo 1.1
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
 
Running MongoDB in the Cloud
Running MongoDB in the CloudRunning MongoDB in the Cloud
Running MongoDB in the Cloud
 
Hard Coding as a design approach
Hard Coding as a design approachHard Coding as a design approach
Hard Coding as a design approach
 
Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013Devops kc meetup_5_20_2013
Devops kc meetup_5_20_2013
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
REST APIs in Laravel 101
REST APIs in Laravel 101REST APIs in Laravel 101
REST APIs in Laravel 101
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
Big Data Developers Moscow Meetup 1  - sql on hadoopBig Data Developers Moscow Meetup 1  - sql on hadoop
Big Data Developers Moscow Meetup 1 - sql on hadoop
 
Modeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught MeModeling Tricks My Relational Database Never Taught Me
Modeling Tricks My Relational Database Never Taught Me
 
Rails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search EngineRails and the Apache SOLR Search Engine
Rails and the Apache SOLR Search Engine
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute Beginner
 
Oracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG EditionOracle REST Data Services: POUG Edition
Oracle REST Data Services: POUG Edition
 

Mehr von coolpics

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33fcoolpics
 
Funny Planes
Funny PlanesFunny Planes
Funny Planescoolpics
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Bannerscoolpics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating coolpics
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from spacecoolpics
 
Cool Pictures
Cool PicturesCool Pictures
Cool Picturescoolpics
 
Funny pictures
Funny picturesFunny pictures
Funny picturescoolpics
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't misscoolpics
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash evercoolpics
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continentcoolpics
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?coolpics
 

Mehr von coolpics (15)

J9gumn6ysd33f
J9gumn6ysd33fJ9gumn6ysd33f
J9gumn6ysd33f
 
Funny Planes
Funny PlanesFunny Planes
Funny Planes
 
Funny Bus Banners
Funny Bus BannersFunny Bus Banners
Funny Bus Banners
 
cool pics
cool picscool pics
cool pics
 
coolpics
coolpicscoolpics
coolpics
 
coolpics
coolpicscoolpics
coolpics
 
cool pics
cool picscool pics
cool pics
 
awesome pictures of transistors copulating
awesome pictures of transistors copulating awesome pictures of transistors copulating
awesome pictures of transistors copulating
 
Superb pictures of the earth from space
Superb pictures of the earth from spaceSuperb pictures of the earth from space
Superb pictures of the earth from space
 
Cool Pictures
Cool PicturesCool Pictures
Cool Pictures
 
Funny pictures
Funny picturesFunny pictures
Funny pictures
 
Cool art that will mess with your head...don't miss
Cool art that will mess with your head...don't missCool art that will mess with your head...don't miss
Cool art that will mess with your head...don't miss
 
The scariest car crash ever
The scariest car crash everThe scariest car crash ever
The scariest car crash ever
 
Great snaps from the African continent
Great snaps from the African continentGreat snaps from the African continent
Great snaps from the African continent
 
What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?What happens when you take a photo at a right angle?
What happens when you take a photo at a right angle?
 

Kürzlich hochgeladen

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Kürzlich hochgeladen (20)

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

flickr's architecture & php

  • 1. Flickr and PHP Cal Henderson
  • 2. What’s Flickr • Photo sharing • Open APIs
  • 3. Logical Architecture Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 4. Physical Architecture Static Servers Database Servers Node Servers Web Servers Users
  • 5. Where is PHP? Page Logic Application Logic DatabasePhoto Storage API EndpointsTemplates 3rd Party Apps Flickr Apps Node Service Flickr.comEmail Users
  • 6. Other than PHP? • Smarty for templating • PEAR for XML and Email parsing • Perl for controlling… • ImageMagick, for image processing • MySQL (4.0 / InnoDb) • Java, for the node service • Apache 2, Redhat, etc. etc.
  • 7. Big Application? • One programmer, one designer, etc. • ~60,000 lines of PHP code • ~60,000 lines of templates • ~70 custom smarty functions/modifiers • ~25,000 DB transactions/second at peak • ~1000 pages per second at peak
  • 8. Thinking outside the web app • Services – Atom/RSS/RDF Feeds – APIs • SOAP • XML-RPC • REST • PEAR::XML::Tree
  • 9. More cool stuff • Email interface – Postfix – PHP – PEAR::Mail::mimeDecode • FTP • Uploading API • Authentication API • Unicode
  • 10. Even more stuff • Real time application • Cool flash apps • Blogging – Blogger API (1 & 2) – Metaweblog API – Atom – LiveJournal
  • 11. APIs are simple! • Modeled on XML-RPC (Sort of) • Method calls with XML responses • SOAP, XML-RPC and REST are just transports • PHP endpoints mean we can use the same application logic as the website
  • 12. XML isn’t simple :( • PHP 4 doesn’t have good a XML parser • Expat is cool though (PEAR::XML::Parser) • Why doesn’t PEAR have XPath? – Because PEAR is stupid! – PHP 4 sucks!
  • 13. I love XPath if ($tree->root->name == 'methodResponse'){ if (($tree->root->children[0]->name == 'params') && ($tree->root->children[0]->children[0]->name == 'param') && ($tree->root->children[0]->children[0]->children[0]->name == 'value') && ($tree->root->children[0]->children[0]->children[0]->children[0]->name == 'array') && ($tree->root->children[0]->children[0]->children[0]->children[0]->children[0]->name == 'data')){ $rsp = $tree->root->children[0]->children[0]->children[0]->children[0]->children[0]; } if ($tree->root->children[0]->name == 'fault'){ $fault = $tree->root->children[0]; return $fault; } } $nodes = $tree->select_nodes('/methodResponse/params/param[1]/value[1]/array[1]/data[1]/text()'); if (count($nodes)){ $rsp = array_pop($nodes); }else{ list($fault) = $tree->select_nodes('/methodResponse/fault'); return $fault; }
  • 14. Creating API methods • Stateless method-call APIs are easy to extend • Adding a method requires no knowledge of the transport • Adding a method once makes it available to all the interfaces • Self documenting
  • 15. Red Hot Unicode Action • UTF-8 pages • CJKV support • It’s really cool
  • 17. Unicode for all • It’s really easy – Don’t need PHP support – Don’t need MySQL support – Just need the right headers – UTF-8 is 7-bit transparent – (Just don’t mess with high characters) • Don’t use HtmlEntities()! • But bear in mind… • JavaScript has patchy Unicode support • People using your APIs might be stupid
  • 18. Scaling the beast • Why PHP is great • MySQL scaling • Search scaling • Horizontal scaling
  • 19. Why PHP is great • Stateless – We can bounce people around servers – Everything is stored in the database – Even the smarty cache – “Shared nothing” – (so long as we avoid PHP sessions)
  • 20. MySQL Scaling • Our database server started to slow • Load of 200 • Replication!
  • 21. MySQL Replication • But it only gives you more SELECT’s • Else you need to partition vertically • Re-architecting sucks :(
  • 22. Looking at usage • Snapshot of db1.flickr.com – SELECT’s 44,220,588 – INSERT’s 1,349,234 – UPDATE’s 1,755,503 – DELETE’s 318,439 – 13 SELECT’s per I/U/D
  • 23. Replication is really cool • A bunch of slave servers handle all the SELECT’s • A single master just handles I/U/D’s • It can scale horizontally, at least for a while.
  • 24. Searching • A simple text search • We were using RLIKE • Then switched to LIKE • Then disabled it all together
  • 25. FULLTEXT Indexes • MySQL saves the day! • But they’re only supported my MyISAM tables • We use InnoDb, because it’s a lot faster • We’re doomed
  • 26. But wait! • Partial replication saves the day • Replicate the portion of the database we want to search. • But change the table types on the slave to MyISAM • It can keep up because it’s only handling I/U/D’s on a couple of tables • And we can reduce the I/U/D’s with a little bit of vertical partitioning
  • 27. JOIN’s are slow • Normalised data is for sissies • Keep multiple copies of data around • Makes searching faster • Have to ensure consistency in the application logic
  • 28. Our current setup Slave Farm DB1 Master I/U/D’s SELECT’s Search Slave Farm Search SELECT’s DB3 Main Search slave DB2 Main Slave
  • 29. Horizontal scaling • At the core of our design • Just add hardware! • Inexpensive • Not exponential • Avoid redesigns
  • 30. Talking to the Node Service • Everyone speaks XML (badly) • Just TCP/IP - fsockopen() • We’re issuing commands, not requesting data, so we don’t bother to parse the response – Just substring search for state=“ok” • Don’t rely on it!
  • 31. RSS / Atom / RDF • Different formats • All quite bad • We’re generating a lot of different feeds • Abstract the difference away using templates • No good way to do private feeds. Why is nobody working on this? (WSSE maybe?)
  • 32. Receiving email • Want users to be able to email photos to Flickr • Just get postfix to pipe each mail to a PHP script • Parse the mail and find any photos • Cellular phone companies hate you • Lots of mailers are retarded – Photos as text/plain attachments :/
  • 33. Upload via FTP • PHP isn’t so great at being a daemon • Leaks memory like a sieve • No threads • Java to the rescue • Java just acts as an FTPd and passes all uploaded files to PHP for processing • (This isn’t actually public) • Bricolage does this I think. Maybe Zope?
  • 34. Blogs • Why does everyone loves blogs so much? • Only a few APIs really – Blogger – Metaweblog – Blogger2 – Movable Type – Atom – Live Journal
  • 35. It’s all broken • Lots of blog software has broken interfaces • It’s a support nightmare • Manila is tricky • But it all works, more or less • Abstracted in the application logic • We just call blogs_post_message();
  • 36. Back to those APIs • We opened up the Flickr APIs a few weeks ago • Programmers mainly build tools for other programmers • We have Perl, python, PHP, ActionScript, XMLHTTP and .NET interface libraries • But also a few actual applications
  • 39. So what next? • Much more scaling • PHP 5? • MySQL 5? • Taking over the world
  • 40. Flickr and PHP Cal Henderson