SlideShare ist ein Scribd-Unternehmen logo
1 von 57
Getting Distributed
               With Ruby (On Rails)

                  by Martin Sadler


Implementing distributed processing at Working With Rails
dsc.net
DSC
• Hosting, Web application development, and
  consultancy
• Host the crew email system and carried out
  the intranet integration for Virgin Atlantic.
• Runs several large forums.
• e.g. pprune.org - 150k members, 2600 at one
  time
• Also AskDirect, Mumsnet
http://www.workingwithrails.com
Working With Rails
• Largest index of Ruby on Rails in the world
• Over 7000 people listed
• From 104 countries
• Find out who’s who?
• Connect with others
• Find a developer for a project / employment
• Also lists groups, companies, and sites
Why Distributed?
Working With Rails
Getting Distributed (With Ruby On Rails)
Distributed Ruby?
Some background
Some background
Current

• Uses FeedTools (nice lib!)
• Great at parsing feed formats
• Good for small sites
Issues
Issues
• No longer supported by author
Issues
• No longer supported by author
• Feeds fetched at the users expense   (and
  therefore Mongrels)
Issues
• No longer supported by author
• Feeds fetched at the users expense   (and
  therefore Mongrels)
• Feeds are cached locally but are parsed on
  request
Issues
• No longer supported by author
• Feeds fetched at the users expense   (and
  therefore Mongrels)
• Feeds are cached locally but are parsed on
  request
• Known probs when scaling (search on
  Google)
The Result
The Result

• Occasional slow loading pages that include
  third party feeds
The Result

• Occasional slow loading pages that include
  third party feeds
• Stale feed items
The Result

• Occasional slow loading pages that include
  third party feeds
• Stale feed items
• Inconstant feed items
The Result

• Occasional slow loading pages that include
  third party feeds
• Stale feed items
• Inconstant feed items
• No good!
The Challenge
• To keep content fresh, push traffic to WWR
  and out to the blog owners
• Different feeds and sources to consider:
  Flickr, Twitter, Blog, Delicious
• Each need to display in multiple places in
  many ways
• But also want to do some funkier stuff (as
  you’ll see a bit later)
Ruby on Rails distributed processing choices

     RingyDingy
                            AP4R

    Rinda         DRB                 Starfish



 BackgroundRB           Reliable-Message
DRB
Basic building block of all other Ruby
distributed libs.
“DRb literally stands for quot;Distributed Rubyquot;. It is a library that allows you
to send and receive messages from remote Ruby objects via TCP/IP. Sound
kind of like RPC, CORBA or Java's RMI? Probably so. This is Ruby's simple
as dirt answer to all of the above.”

http://chadfowler.com/ruby/drb.html
Quick DRB Example
Server
                                                        Client
require 'drb'
                                                        require 'drb'
class TestServer                                        DRb.start_service()
                                                        obj = DRbObject.new(nil, 'druby://localhost:9000')
  def doit                                              # Now use obj
    quot;Hello, Distributed Worldquot;                          p obj.doit
  end
end

aServerObject = TestServer.new
DRb.start_service('druby://localhost:9000', aServerObject)
DRb.thread.join # Don't exit just yet!
Quick DRB Example
Server
                                                        Client
require 'drb'
                                                        require 'drb'
class TestServer                                        DRb.start_service()
                                                        obj = DRbObject.new(nil, 'druby://localhost:9000')
  def doit                                              # Now use obj
    quot;Hello, Distributed Worldquot;                          p obj.doit
  end
end

aServerObject = TestServer.new
DRb.start_service('druby://localhost:9000', aServerObject)
DRb.thread.join # Don't exit just yet!




> ruby server.rb
Quick DRB Example
Server
                                                        Client
require 'drb'
                                                        require 'drb'
class TestServer                                        DRb.start_service()
                                                        obj = DRbObject.new(nil, 'druby://localhost:9000')
  def doit                                              # Now use obj
    quot;Hello, Distributed Worldquot;                          p obj.doit
  end
end

aServerObject = TestServer.new
DRb.start_service('druby://localhost:9000', aServerObject)
DRb.thread.join # Don't exit just yet!




> ruby server.rb                      > ruby client.rb
                                      “Hello Distributed World”
Basics

    • Server
    • Clients / Workers
    • Communicate via messages
http://en.wikipedia.org/wiki/Distributed_computing
BackgroundRB
• Ruby job server and scheduler.
• Integrates with Rails
• Quite complex
• Some issues between versions but many
  favor it above the other libs
• Most well known
              http://backgroundrb.rubyforge.org/
Starfish
• Inspired by Google’s MapReduce
• Easy to understand code
• Stability?
• No longer supported by author?

                     http://rufy.com/starfish/doc/
reliable-message
   • Solid library
   • Easy to understand API
   • Bit more involved to setup
   • Can be integrated with Rails
   • On going development
http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/ReliableMessaging
AP4R

• Asynchronous Processing for Ruby
• Lesser known lib from Japan (new kid on the
  block)
• Integrates with Rails
• Built on top of reliable-message
AP4R
• AP4R, Asynchronous Processing for Ruby, is
  the implementation of reliable asynchronous
  message processing. It provides message
  queuing, and message dispatching.
• Using asynchronous processing, we can cut
  down turn-around-time of web applications
  by queuing, or can utilize more machine
  power by load-balancing.
AP4R Features

•   Business logic can be implemented as simple Web applications, or ruby code, whether it's called
    asynchronously or synchronously.

•   Asynchronous messaging is reliable by RDBMS persistence (now MySQL only) or file
    persistence, under the favor of reliable-msg.

•   Load balancing over multiple AP4R processes on single/multiple servers is supported.

•   Asynchronous logics are called via various protocols, such as XML-RPC, SOAP, HTTP PUT, and
    more.

•   Using store and forward function, at-least-omce QoS level is provided.
AP4R Process Flow

•   A client(e.g. a web browser) makes a request to a web server (Apache, Lighttpd, etc...).

•   A rails application (a synchronous logic) is executed on mongrel via mod_proxy or something.

•   At the last of the synchronous logic, message(s) are put to AP4R (AP4R provides a helper).

•   Once the synchronous logic is done, the clients receives a response immediately.

•   AP4R queues the message, and requests it to the web server asynchronously.

•   An asynchronous logic, implemented as usual rails action, is executed.
AP4R example
Hello World app comes with AP4R to get
you started.


Nice guide also here
http://rubyforge.org/frs/download.php/13312/AP4R_Users_Guide_EN.pdf
Complimentary
  Services
Rinda
   • Rinda::Ring allows DRb services and clients
      to automatically find each other without
      knowing where they live.
   • DRb servers register themselves with a
      RingServer which allows clients to find the
      servers they need. Many servers may
      register themselves with the RingServer. The
      DRb servers don't need to run on the same
      machine.
http://segment7.net/projects/ruby/drb/rinda/ringserver.html
RingyDingy
• RingyDingy automatically registers a service
  with a RingServer. If communication between
  the RingServer and the RingyDingy is lost,
  RingyDingy will re-register its service with
  the RingServer when it reappears.



http://seattlerb.rubyforge.org/RingyDingy/
Feeds in WWR
 AP4R
 Server
                               Feed
@queue
                              Queue



Feed Fetcher   Feed Fetcher     Feed Fetcher
     1              2                N
Running the code

ruby script/ap4r_start -c config/queues_mysql.cfg
rake background:feed_queue
rake background:feed_retrieve
Key points

• The Feed Queue fetches the urls of stale
  feeds
• Each worker (client) has the Rails
  environment loaded
With this solution


• Can scale as demand grows
• Flexible for any type of feed data
• Still - room for improvement
Possible Improvements

• Automatic spawning and killing of workers
  as queue size grows or decreases
• Better handling of feed errors
• Dynamic polling intervals based on user
  defined prefs or some intelligent logic.
When to go distributed?

• Long running process or task
 • Fetching external data
 • Complex computations
 • .... that can be broken into chunks or work
• You care about the user experience
Pitfalls
• Dependencies
• `connection closed' errors on Mac (IPV6) -
  change all refs of localhost to 127.0.0.1 to
  avoid. (had to patch reliable-message)
• Terminology to understand
• Memory requirements
Do you need
         distributed?

• Maybe you would be better scheduling
  instead?
• http://www.igvita.com/blog/2007/03/29/
  scheduling-tasks-in-ruby-rails/
So where is all this
  leading us to?
Contextual Feed
 Aggregation
Getting Distributed (With Ruby On Rails)
Group feed aggregation
Getting Distributed (With Ruby On Rails)
Group blog posts
Group blog posts
Twitters
Group blog posts
   Twitters
and so on.......
Thanks!
             http://www.dsc.net
             http://www.workingwithrails.com
             Blog: http://beyondthetype.com


Enjoyed the talk? Recommend me on WWR
http://workingwithrails.com/person/5152-martin-sadler

Weitere ähnliche Inhalte

Was ist angesagt?

[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager
[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager
[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client ManagerDrupalDay
 
Modern Web Development with Perl
Modern Web Development with PerlModern Web Development with Perl
Modern Web Development with PerlDave Cross
 
Ruby on Rails 2.1 What's New
Ruby on Rails 2.1 What's NewRuby on Rails 2.1 What's New
Ruby on Rails 2.1 What's NewLibin Pan
 
Introduction to Web Programming with Perl
Introduction to Web Programming with PerlIntroduction to Web Programming with Perl
Introduction to Web Programming with PerlDave Cross
 
Fast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on OracleFast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on OracleRaimonds Simanovskis
 
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Alberto Perdomo
 
Deploying Perl apps on dotCloud
Deploying Perl apps on dotCloudDeploying Perl apps on dotCloud
Deploying Perl apps on dotClouddaoswald
 
Introducing RaveJS: Spring Boot concepts for JavaScript applications
Introducing RaveJS: Spring Boot concepts for JavaScript applicationsIntroducing RaveJS: Spring Boot concepts for JavaScript applications
Introducing RaveJS: Spring Boot concepts for JavaScript applicationsJohn Hann
 
Build Your Own CMS with Apache Sling
Build Your Own CMS with Apache SlingBuild Your Own CMS with Apache Sling
Build Your Own CMS with Apache SlingBob Paulin
 
Perl in the Internet of Things
Perl in the Internet of ThingsPerl in the Internet of Things
Perl in the Internet of ThingsDave Cross
 
Creating And Consuming Web Services In Php 5
Creating And Consuming Web Services In Php 5Creating And Consuming Web Services In Php 5
Creating And Consuming Web Services In Php 5Michael Girouard
 
Introduction to CouchDB
Introduction to CouchDBIntroduction to CouchDB
Introduction to CouchDBOpusVL
 
How to dockerize rails application compose and rails tutorial
How to dockerize rails application compose and rails tutorialHow to dockerize rails application compose and rails tutorial
How to dockerize rails application compose and rails tutorialKaty Slemon
 
Developing cross platform desktop application with Ruby
Developing cross platform desktop application with RubyDeveloping cross platform desktop application with Ruby
Developing cross platform desktop application with RubyAnis Ahmad
 
Getting started with Catalyst and extjs
Getting started with Catalyst and extjsGetting started with Catalyst and extjs
Getting started with Catalyst and extjsPeter Edwards
 
Introduction to Rails - presented by Arman Ortega
Introduction to Rails - presented by Arman OrtegaIntroduction to Rails - presented by Arman Ortega
Introduction to Rails - presented by Arman Ortegaarman o
 

Was ist angesagt? (20)

[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager
[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager
[drupalday2017] - Drupal come frontend che consuma servizi: HTTP Client Manager
 
Modern Web Development with Perl
Modern Web Development with PerlModern Web Development with Perl
Modern Web Development with Perl
 
Ruby on Rails 2.1 What's New
Ruby on Rails 2.1 What's NewRuby on Rails 2.1 What's New
Ruby on Rails 2.1 What's New
 
Ruby on Rails
Ruby on RailsRuby on Rails
Ruby on Rails
 
Introduction to Web Programming with Perl
Introduction to Web Programming with PerlIntroduction to Web Programming with Perl
Introduction to Web Programming with Perl
 
Fast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on OracleFast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on Oracle
 
Os Bunce
Os BunceOs Bunce
Os Bunce
 
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
 
Deploying Perl apps on dotCloud
Deploying Perl apps on dotCloudDeploying Perl apps on dotCloud
Deploying Perl apps on dotCloud
 
Introducing RaveJS: Spring Boot concepts for JavaScript applications
Introducing RaveJS: Spring Boot concepts for JavaScript applicationsIntroducing RaveJS: Spring Boot concepts for JavaScript applications
Introducing RaveJS: Spring Boot concepts for JavaScript applications
 
Build Your Own CMS with Apache Sling
Build Your Own CMS with Apache SlingBuild Your Own CMS with Apache Sling
Build Your Own CMS with Apache Sling
 
Os Pruett
Os PruettOs Pruett
Os Pruett
 
Perl in the Internet of Things
Perl in the Internet of ThingsPerl in the Internet of Things
Perl in the Internet of Things
 
Develop webservice in PHP
Develop webservice in PHPDevelop webservice in PHP
Develop webservice in PHP
 
Creating And Consuming Web Services In Php 5
Creating And Consuming Web Services In Php 5Creating And Consuming Web Services In Php 5
Creating And Consuming Web Services In Php 5
 
Introduction to CouchDB
Introduction to CouchDBIntroduction to CouchDB
Introduction to CouchDB
 
How to dockerize rails application compose and rails tutorial
How to dockerize rails application compose and rails tutorialHow to dockerize rails application compose and rails tutorial
How to dockerize rails application compose and rails tutorial
 
Developing cross platform desktop application with Ruby
Developing cross platform desktop application with RubyDeveloping cross platform desktop application with Ruby
Developing cross platform desktop application with Ruby
 
Getting started with Catalyst and extjs
Getting started with Catalyst and extjsGetting started with Catalyst and extjs
Getting started with Catalyst and extjs
 
Introduction to Rails - presented by Arman Ortega
Introduction to Rails - presented by Arman OrtegaIntroduction to Rails - presented by Arman Ortega
Introduction to Rails - presented by Arman Ortega
 

Ähnlich wie Getting Distributed (With Ruby On Rails)

Running Containerized Node.js Services on AWS Elastic Beanstalk
Running Containerized Node.js Services on AWS Elastic BeanstalkRunning Containerized Node.js Services on AWS Elastic Beanstalk
Running Containerized Node.js Services on AWS Elastic Beanstalkzupzup.org
 
aws lambda & api gateway
aws lambda & api gatewayaws lambda & api gateway
aws lambda & api gatewayfumihiko hata
 
A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)Flowdock
 
Rails Conf Europe 2007 Notes
Rails Conf  Europe 2007  NotesRails Conf  Europe 2007  Notes
Rails Conf Europe 2007 NotesRoss Lawley
 
TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011Lance Ball
 
Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS drupalcampest
 
Ruby on Rails survival guide of an aged Java developer
Ruby on Rails survival guide of an aged Java developerRuby on Rails survival guide of an aged Java developer
Ruby on Rails survival guide of an aged Java developergicappa
 
Cloud Foundry Open Tour China
Cloud Foundry Open Tour ChinaCloud Foundry Open Tour China
Cloud Foundry Open Tour Chinamarklucovsky
 
General Assembly Workshop: Advanced JavaScript
General Assembly Workshop: Advanced JavaScriptGeneral Assembly Workshop: Advanced JavaScript
General Assembly Workshop: Advanced JavaScriptSpike Brehm
 
Connecting the Worlds of Java and Ruby with JRuby
Connecting the Worlds of Java and Ruby with JRubyConnecting the Worlds of Java and Ruby with JRuby
Connecting the Worlds of Java and Ruby with JRubyNick Sieger
 
Cloud Foundry Open Tour China (english)
Cloud Foundry Open Tour China (english)Cloud Foundry Open Tour China (english)
Cloud Foundry Open Tour China (english)marklucovsky
 
java database connectivity for java programming
java database connectivity for java programmingjava database connectivity for java programming
java database connectivity for java programmingrinky1234
 
Jruby synergy-of-ruby-and-java
Jruby synergy-of-ruby-and-javaJruby synergy-of-ruby-and-java
Jruby synergy-of-ruby-and-javaKeith Bennett
 
Ruby Performance - The Last Mile - RubyConf India 2016
Ruby Performance - The Last Mile - RubyConf India 2016Ruby Performance - The Last Mile - RubyConf India 2016
Ruby Performance - The Last Mile - RubyConf India 2016Charles Nutter
 
Ruby on Rails : First Mile
Ruby on Rails : First MileRuby on Rails : First Mile
Ruby on Rails : First MileGourab Mitra
 
Everything ruby
Everything rubyEverything ruby
Everything rubyajeygore
 
FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceEvan McGee
 
Ruby on Rails All Hands Meeting
Ruby on Rails All Hands MeetingRuby on Rails All Hands Meeting
Ruby on Rails All Hands MeetingDan Davis
 

Ähnlich wie Getting Distributed (With Ruby On Rails) (20)

Running Containerized Node.js Services on AWS Elastic Beanstalk
Running Containerized Node.js Services on AWS Elastic BeanstalkRunning Containerized Node.js Services on AWS Elastic Beanstalk
Running Containerized Node.js Services on AWS Elastic Beanstalk
 
aws lambda & api gateway
aws lambda & api gatewayaws lambda & api gateway
aws lambda & api gateway
 
Get your teeth into Plack
Get your teeth into PlackGet your teeth into Plack
Get your teeth into Plack
 
A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)A Tale of a Server Architecture (Frozen Rails 2012)
A Tale of a Server Architecture (Frozen Rails 2012)
 
Rails Conf Europe 2007 Notes
Rails Conf  Europe 2007  NotesRails Conf  Europe 2007  Notes
Rails Conf Europe 2007 Notes
 
TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011
 
Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS Developing realtime apps with Drupal and NodeJS
Developing realtime apps with Drupal and NodeJS
 
Ruby on Rails survival guide of an aged Java developer
Ruby on Rails survival guide of an aged Java developerRuby on Rails survival guide of an aged Java developer
Ruby on Rails survival guide of an aged Java developer
 
Cloud Foundry Open Tour China
Cloud Foundry Open Tour ChinaCloud Foundry Open Tour China
Cloud Foundry Open Tour China
 
General Assembly Workshop: Advanced JavaScript
General Assembly Workshop: Advanced JavaScriptGeneral Assembly Workshop: Advanced JavaScript
General Assembly Workshop: Advanced JavaScript
 
Connecting the Worlds of Java and Ruby with JRuby
Connecting the Worlds of Java and Ruby with JRubyConnecting the Worlds of Java and Ruby with JRuby
Connecting the Worlds of Java and Ruby with JRuby
 
Cloud Foundry Open Tour China (english)
Cloud Foundry Open Tour China (english)Cloud Foundry Open Tour China (english)
Cloud Foundry Open Tour China (english)
 
Ugo Cei Presentation
Ugo Cei PresentationUgo Cei Presentation
Ugo Cei Presentation
 
java database connectivity for java programming
java database connectivity for java programmingjava database connectivity for java programming
java database connectivity for java programming
 
Jruby synergy-of-ruby-and-java
Jruby synergy-of-ruby-and-javaJruby synergy-of-ruby-and-java
Jruby synergy-of-ruby-and-java
 
Ruby Performance - The Last Mile - RubyConf India 2016
Ruby Performance - The Last Mile - RubyConf India 2016Ruby Performance - The Last Mile - RubyConf India 2016
Ruby Performance - The Last Mile - RubyConf India 2016
 
Ruby on Rails : First Mile
Ruby on Rails : First MileRuby on Rails : First Mile
Ruby on Rails : First Mile
 
Everything ruby
Everything rubyEverything ruby
Everything ruby
 
FreeSWITCH as a Microservice
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
 
Ruby on Rails All Hands Meeting
Ruby on Rails All Hands MeetingRuby on Rails All Hands Meeting
Ruby on Rails All Hands Meeting
 

Kürzlich hochgeladen

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 

Kürzlich hochgeladen (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 

Getting Distributed (With Ruby On Rails)

  • 1. Getting Distributed With Ruby (On Rails) by Martin Sadler Implementing distributed processing at Working With Rails
  • 3. DSC • Hosting, Web application development, and consultancy • Host the crew email system and carried out the intranet integration for Virgin Atlantic. • Runs several large forums. • e.g. pprune.org - 150k members, 2600 at one time • Also AskDirect, Mumsnet
  • 5. Working With Rails • Largest index of Ruby on Rails in the world • Over 7000 people listed • From 104 countries • Find out who’s who? • Connect with others • Find a developer for a project / employment • Also lists groups, companies, and sites
  • 12. Current • Uses FeedTools (nice lib!) • Great at parsing feed formats • Good for small sites
  • 14. Issues • No longer supported by author
  • 15. Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels)
  • 16. Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels) • Feeds are cached locally but are parsed on request
  • 17. Issues • No longer supported by author • Feeds fetched at the users expense (and therefore Mongrels) • Feeds are cached locally but are parsed on request • Known probs when scaling (search on Google)
  • 19. The Result • Occasional slow loading pages that include third party feeds
  • 20. The Result • Occasional slow loading pages that include third party feeds • Stale feed items
  • 21. The Result • Occasional slow loading pages that include third party feeds • Stale feed items • Inconstant feed items
  • 22. The Result • Occasional slow loading pages that include third party feeds • Stale feed items • Inconstant feed items • No good!
  • 23. The Challenge • To keep content fresh, push traffic to WWR and out to the blog owners • Different feeds and sources to consider: Flickr, Twitter, Blog, Delicious • Each need to display in multiple places in many ways • But also want to do some funkier stuff (as you’ll see a bit later)
  • 24. Ruby on Rails distributed processing choices RingyDingy AP4R Rinda DRB Starfish BackgroundRB Reliable-Message
  • 25. DRB Basic building block of all other Ruby distributed libs. “DRb literally stands for quot;Distributed Rubyquot;. It is a library that allows you to send and receive messages from remote Ruby objects via TCP/IP. Sound kind of like RPC, CORBA or Java's RMI? Probably so. This is Ruby's simple as dirt answer to all of the above.” http://chadfowler.com/ruby/drb.html
  • 26. Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet!
  • 27. Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet! > ruby server.rb
  • 28. Quick DRB Example Server Client require 'drb' require 'drb' class TestServer DRb.start_service() obj = DRbObject.new(nil, 'druby://localhost:9000') def doit # Now use obj quot;Hello, Distributed Worldquot; p obj.doit end end aServerObject = TestServer.new DRb.start_service('druby://localhost:9000', aServerObject) DRb.thread.join # Don't exit just yet! > ruby server.rb > ruby client.rb “Hello Distributed World”
  • 29. Basics • Server • Clients / Workers • Communicate via messages http://en.wikipedia.org/wiki/Distributed_computing
  • 30. BackgroundRB • Ruby job server and scheduler. • Integrates with Rails • Quite complex • Some issues between versions but many favor it above the other libs • Most well known http://backgroundrb.rubyforge.org/
  • 31. Starfish • Inspired by Google’s MapReduce • Easy to understand code • Stability? • No longer supported by author? http://rufy.com/starfish/doc/
  • 32. reliable-message • Solid library • Easy to understand API • Bit more involved to setup • Can be integrated with Rails • On going development http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/ReliableMessaging
  • 33. AP4R • Asynchronous Processing for Ruby • Lesser known lib from Japan (new kid on the block) • Integrates with Rails • Built on top of reliable-message
  • 34. AP4R • AP4R, Asynchronous Processing for Ruby, is the implementation of reliable asynchronous message processing. It provides message queuing, and message dispatching. • Using asynchronous processing, we can cut down turn-around-time of web applications by queuing, or can utilize more machine power by load-balancing.
  • 35. AP4R Features • Business logic can be implemented as simple Web applications, or ruby code, whether it's called asynchronously or synchronously. • Asynchronous messaging is reliable by RDBMS persistence (now MySQL only) or file persistence, under the favor of reliable-msg. • Load balancing over multiple AP4R processes on single/multiple servers is supported. • Asynchronous logics are called via various protocols, such as XML-RPC, SOAP, HTTP PUT, and more. • Using store and forward function, at-least-omce QoS level is provided.
  • 36. AP4R Process Flow • A client(e.g. a web browser) makes a request to a web server (Apache, Lighttpd, etc...). • A rails application (a synchronous logic) is executed on mongrel via mod_proxy or something. • At the last of the synchronous logic, message(s) are put to AP4R (AP4R provides a helper). • Once the synchronous logic is done, the clients receives a response immediately. • AP4R queues the message, and requests it to the web server asynchronously. • An asynchronous logic, implemented as usual rails action, is executed.
  • 37. AP4R example Hello World app comes with AP4R to get you started. Nice guide also here http://rubyforge.org/frs/download.php/13312/AP4R_Users_Guide_EN.pdf
  • 39. Rinda • Rinda::Ring allows DRb services and clients to automatically find each other without knowing where they live. • DRb servers register themselves with a RingServer which allows clients to find the servers they need. Many servers may register themselves with the RingServer. The DRb servers don't need to run on the same machine. http://segment7.net/projects/ruby/drb/rinda/ringserver.html
  • 40. RingyDingy • RingyDingy automatically registers a service with a RingServer. If communication between the RingServer and the RingyDingy is lost, RingyDingy will re-register its service with the RingServer when it reappears. http://seattlerb.rubyforge.org/RingyDingy/
  • 41. Feeds in WWR AP4R Server Feed @queue Queue Feed Fetcher Feed Fetcher Feed Fetcher 1 2 N
  • 42. Running the code ruby script/ap4r_start -c config/queues_mysql.cfg rake background:feed_queue rake background:feed_retrieve
  • 43. Key points • The Feed Queue fetches the urls of stale feeds • Each worker (client) has the Rails environment loaded
  • 44. With this solution • Can scale as demand grows • Flexible for any type of feed data • Still - room for improvement
  • 45. Possible Improvements • Automatic spawning and killing of workers as queue size grows or decreases • Better handling of feed errors • Dynamic polling intervals based on user defined prefs or some intelligent logic.
  • 46. When to go distributed? • Long running process or task • Fetching external data • Complex computations • .... that can be broken into chunks or work • You care about the user experience
  • 47. Pitfalls • Dependencies • `connection closed' errors on Mac (IPV6) - change all refs of localhost to 127.0.0.1 to avoid. (had to patch reliable-message) • Terminology to understand • Memory requirements
  • 48. Do you need distributed? • Maybe you would be better scheduling instead? • http://www.igvita.com/blog/2007/03/29/ scheduling-tasks-in-ruby-rails/
  • 49. So where is all this leading us to?
  • 56. Group blog posts Twitters and so on.......
  • 57. Thanks! http://www.dsc.net http://www.workingwithrails.com Blog: http://beyondthetype.com Enjoyed the talk? Recommend me on WWR http://workingwithrails.com/person/5152-martin-sadler