Overview of Google Cloud products for Developers, to build, sell and monetize web apps: Google Apps Marketplace, App Engine, App Engine for Business, Google Storage, Prediction and BigQuery APIs.
3. The benefits of Cloud Computing
Economics
Pay for only what you use
TCO
OPEX vs CAPEX
Operations
Day to day: no maintenance
Fighting fires: no Pagers
Elasticity
Focus on your Business
3
Tuesday, July 6, 2010
4. Build and Buy all your enterprise cloud apps...
Google Apps Platform
Enterprise Firewall
4
Enterprise Data Authentication Enterprise Services User Management
Tuesday, July 6, 2010
5. Build and Buy all your enterprise cloud apps...
Buy from Google
Google Apps
for Business
Google Apps Platform
Enterprise Firewall
4
Enterprise Data Authentication Enterprise Services User Management
Tuesday, July 6, 2010
6. Build and Buy all your enterprise cloud apps...
Buy from others Buy from Google
Google Apps Google Apps
Marketplace for Business
Google Apps Platform
Enterprise Firewall
4
Enterprise Data Authentication Enterprise Services User Management
Tuesday, July 6, 2010
7. Build and Buy all your enterprise cloud apps...
Buy from others Buy from Google Build your own
Google Apps Google Apps Google App Engine
Marketplace for Business for Business
Google Apps Platform
Enterprise Firewall
4
Enterprise Data Authentication Enterprise Services User Management
Tuesday, July 6, 2010
8. Customers want more Apps
Business
in the cloud
Google
Apps
5
Tuesday, July 6, 2010
11. Google Apps is Focused on Messaging &
Collaboration
Gives every employee powerful messaging and collaboration
tools without the usual IT hassle and cost
Tuesday, July 6, 2010
12. These customers want more great cloud apps...
Messaging
Document Management
Collaboration
Productivity
Accounting & Finance Project Management
Admin Tools Sales & Marketing
Calendaring / Meetings Security & Compliance
Customer Management Workflow
Tuesday, July 6, 2010
13. Steps to sell your app to Google Apps customers
1. Build your app:
- with any tools and hosting provider you want
2. Integrate your app:
- add Single Sign On using OpenID (required)
- access over a dozen integration points from Calendar, Contacts,
Docs, etc. using OAuth (optional)
3. Sell your app:
- to 2M+ businesses & 25M+ users
- through Google Apps resellers
- to wherever Google Apps goes
- One-time fee of $100, 20% rev share starting 2H'10
Tuesday, July 6, 2010
17. How it's done - Single Sign On
Tuesday, July 6, 2010
18. Single Sign On
OpenID with Google Apps
bob@foo.com Welcome Bob!
***********
Sounds complicated, but not hard in practice!
Tuesday, July 6, 2010
19. Single Sign On
OpenID Libraries
Language Libraries
Java OpenID4Java, Step2
.NET DotNetOpenAuth
php-openid,
PHP
php-openid-apps-discovery
ruby-openid,
Ruby
ruby-openid-apps-discovery
Any RPX, Ping Identity
Tuesday, July 6, 2010
20. Single Sign On - Code
Step 1 of 3 - Initialize library
function getOpenIDConsumer() {
  // Initialize client, storing associations
  // in memcache
  $store = new Auth_OpenID_MemcachedStore($MEMCACHE);
  $consumer = new Auth_OpenID_Consumer($store);
 Â
  // Enable Google Apps supportÂ
  GApps_OpenID_EnableDiscovery($consumer);
  return $consumer;
}
Tuesday, July 6, 2010
21. Single Sign On - Code
Step 2 of 3 - Make the request
// Create an auth request to the user's domainÂ
$auth_request =Â getOpenIDConsumer()->begin($domain);
// Request email address & name during loginÂ
$ax = new Auth_OpenID_AX_FetchRequest;Â
$attr = Auth_OpenID_AX_AttrInfo::make(
      $AX_SCHEMA_EMAIL, 1, 1, 'email');
$attr = Auth_OpenID_AX_AttrInfo::make(
      $AX_SCHEMA_FIRSTNAME, 1, 1, 'ďŹrst');
$attr = Auth_OpenID_AX_AttrInfo::make(
      $AX_SCHEMA_LASTNAME, 1, 1, 'last');
$auth_request->addExtension($ax);
// Render Javascript/form to post request
$form_html = $auth_request->htmlMarkup($REALM,
    $RETURN_TO, false,Â
    array('id' => 'openid_message'));Â
print $form_html;
Tuesday, July 6, 2010
22. Single Sign On - Code
Step 3 of 3 - Handle response
// Parse the response from identity providerÂ
$response = getOpenIDConsumer()->complete($RETURN_TO);Â
if ($response->status == Auth_OpenID_SUCCESS) {
  // Extract data from responseÂ
  $openid = $response->getDisplayIdentiďŹer();
  $ax = new Auth_OpenID_AX_FetchResponse();
  $ax_resp = $ax->fromSuccessResponse($response);
  $email = $ax_resp->data[$AX_SCHEMA_EMAIL][0];
  $ďŹrstName = $ax_resp->data[$AX_SCHEMA_FIRSTNAME][0];
  $lastName = $ax_resp->data[$AX_SCHEMA_LASTTNAME][0];
  // Map to user in DBÂ
  $user = fetch_user($openid, $email, $ďŹrstName,
    $lastName);
}Â
Tuesday, July 6, 2010
23. Single Sign On - Code
Manifest
<ApplicationManifest>
  <Name>Sassy Voice</Name>
  <Description>Voice Mail & Messaging</Description>
  <Admin>
   <Link rel="setup">
http://voice.saasyapp.com/setup.php?domain=${DOMAIN_NAME}
   </Link>
  </Admin> Â
  <Extension id="navlink" type="link">
   <Name>SaasyVoice</Name>
   <Url>
http://voice.saasyapp.com/login.php?domain=${DOMAIN_NAME}
   </Url>Â
  </Extension>
  <Extension id="realm" type="openIdRealm">
   <Url>http://voice.saasyapp.com/</Url>
  </Extension>
</ApplicationManifest>
Tuesday, July 6, 2010
24. Single Sign On
What we learned about users
Claimed ID: http://example.com/openid?id=12345
Email: bob@example.com*
First Name: Bob
Last Name: Dobbs
* Be sure to confirm or white-list trusted providers
Tuesday, July 6, 2010
26. 2-Legged OAuth
Delegating data access
<ApplicationManifest>
  ...Â
  <Scope id="userFeed">
   <Url>https://apps-apis.google...
   <Reason>To get a list of user...
  </Scope>
  <Scope id="contactsFeed">
   <Url>https://www.google.com...Â
   <Reason>To display names of...Â
  </Scope>
  <Scope id="docsFeed">Â
   <Url>https://docs.google.com...Â
   <Reason>To export a call log...Â
  </Scope>Â
</ApplicationManifest>
Tuesday, July 6, 2010
27. Data Access - Code
Step 1 of 2 - Initialize the client
function getOauthClient() {
  $options = array(
   'consumerKey' => $CONSUMER_KEY,
   'consumerSecret' => $CONSUMER_SECRET,
   'signatureMethod' => 'HMAC-SHA1',
    'requestScheme' =>Zend_Oauth::REQUEST_SCHEME_HEADER,
    'version' => '1.0');
  $consumer = new Zend_Oauth_Consumer($options);
  $token = new Zend_Oauth_Token_Access();
  $httpClient = $token->getHttpClient($options);
  return $httpClient;
}
Tuesday, July 6, 2010
28. Data Access - Code
Step 2 of 2 - Fetching Users with Provisioning API
// Initialize client
$userClient = new Zend_Gdata_Gapps(getOauthClient());
// Query feed for current user's domain
$userQuery = new Zend_Gdata_Gapps_UserQuery(
  getCurrentUserDomain());
$usersFeed = $userClient->getUserFeed($userQuery);
// Extract data from user feed
$users =Â array();
foreach ($usersFeed as $userEntry)Â {
  $login = $userEntry->getLogin();
  $name = $userEntry->getName();
  $users[] = array(
   'username' => $login->getUsername(),
   'ďŹrstName' => $name->getGivenName(),
   'lastName' => $name->getFamilyName(),
   'admin' => $login->getAdmin());
}
Tuesday, July 6, 2010
29. Integration Recap
With not a lot of code we added:
⢠Single Sign On with OpenID
⢠Quicker setup with Attribute Exchange, Provisioning API
⢠Integrated user data with Contacts API
⢠Uploading spreadsheets with Docs API
Tuesday, July 6, 2010
30. Gadgets - your real estate in Google Apps
Tuesday, July 6, 2010
31. Gadgets
⢠Many types of gadgets
o Gmail sidebar
o Calendar sidebar
o Sites
o Spreadsheets
⢠Two new types of gadgets available this week!
o Gmail contextual!
o Wave!
⢠All use the OpenSocial gadgets specification
Tuesday, July 6, 2010
32. Gmail contextual gadgets
⢠Detect e-mail content via regular expressions
⢠Display and collect actionable business information
directly in Gmail, below each message
Tuesday, July 6, 2010
35. Calendar sidebar gadgets
⢠Can detect:
o currently-displayed date range
o currently-selected event
ď§title
ď§attendees
ď§date/time
Tuesday, July 6, 2010
39. Complete Manifest
<ApplicationManifest xmlns="http://schemas.google.com/
ApplicationManifest/2009">
<Name>SaaSy Voice</Name>
  <Description>SaaSy Voice</Description>
  <Support>
   <Link rel="setup" href="https://.../setup.php?domain=${DOMAIN_NAME}"/
>
   <Link rel="manage" href="https://.../conďŹg.php?domain=$
{DOMAIN_NAME}"/>
   <Link rel="deletion-policy" href="http://.../deletion-policy.php"/>
   <Link rel="support" href="http://.../support.php"/>
  </Support>
Tuesday, July 6, 2010
40. Complete Manifest
(continued)
  <!-- Link in universal navigation -->Â
  <Extension id="oneBarLink" type="link">Â
   <Name>SaaSy Voice</Name>
   <Url>http://.../index.php?domain=${DOMAIN_NAME}</Url>
   <Scope ref="provisioningFeed"/>
   <Scope ref="contactsFeed"/>Â
   <Scope ref="docsFeed"/>
  </Extension>Â
  <!-- Declare our openid.realm -->Â
  <Extension id="realm" type="openIdRealm">
   <Url>http://voice.saasyapp.com/marketplace/</Url>
  </Extension>
Tuesday, July 6, 2010
41. Complete Manifest
(continued)
  <Scope id="provisioningFeed">
   <Url>https://apps-apis.google.com/a/feeds/user/#readonly</Url>Â
   <Reason>To get a list of users to provision accounts</Reason>
  </Scope>
  <Scope id="contactsFeed">Â
   <Url>https://www.google.com/m8/feeds/</Url>
   <Reason>To display names of people who called</Reason>
  </Scope>
  <Scope id="docsFeed">
   <Url>https://docs.google.com/feeds/</Url>Â
   <Reason>To export a call log</Reason>
  </Scope>
</ApplicationManifest>
Tuesday, July 6, 2010
42. Listing in the Google Apps Marketplace
Tuesday, July 6, 2010
43. Listing in the Google Apps Marketplace
Tuesday, July 6, 2010
44. Reach 25 million Google Apps Users
"We've seen a very meaningful increase in high quality, non-paid lead
flow directly attributable to Google Apps customers...Our customers
cite Smartsheet's tight integration with Google's Data APIs as a key
factor in their decision to purchase."
Brent Frei, Smartsheet
Tuesday, July 6, 2010
46. Roadmap - General
⢠Recent Launches
o Gmail contextual gadgets
o OAuth support for Gmail's IMAP and SMTP
Tuesday, July 6, 2010
47. Roadmap - Billing
⢠Goals
o Simplified invoicing & payments for customers
o Consistent experience across apps
⢠Planned 2nd-half of 2010
o Required for installable apps after transition period
o Revenue sharing starts once APIs adopted
⢠Features
o Unified billing for apps purchased in marketplace
o Support for a variety of pricing & licensing models
o APIs for customization and up-selling
Tuesday, July 6, 2010
49. Resources
⢠Business and Marketing
o http://developer.googleapps.com/marketplace
⢠Technical and Code
o http://code.google.com/googleapps/marketplace
⢠Shopping!
o http://www.google.com/appsmarketplace
⢠Don't have Google Apps?
o http://www.google.com/a
Tuesday, July 6, 2010
50. Build and Buy all your enterprise cloud apps...
Buy from others Buy from Google Build your own
Google Apps Google Apps Google App Engine
Marketplace for Business for Business
Google Apps Platform
Enterprise Firewall
47
Enterprise Data Authentication Enterprise Services User Management
Tuesday, July 6, 2010
52. Leveraging Google's Leadership in
Cloud Computing
⢠Massive data center operations
⢠Purpose built hardware
⢠Multi tenant software platform at Internet scale
49
Tuesday, July 6, 2010
53. Google App Engine
- Easy to build
- Easy to maintain
- Easy to scale
50
Tuesday, July 6, 2010
54. Cloud development in a box
⢠SDK & âThe Cloudâ
⢠Hardware
⢠Networking
⢠Operating system
⢠Application runtime
Java, Python
o
⢠Static file serving
⢠Services
⢠Fault tolerance
⢠Load balancing
51
Tuesday, July 6, 2010
55. App Engine Services
Memcache Datastore URL Fetch
Mail XMPP Task Queue
Images Blobstore User Service
52
Tuesday, July 6, 2010
56. Always free to get started
~5M pageviews/month
⢠6.5 CPU hrs/day
⢠1 GB storage
⢠650K URL Fetch calls/day
⢠2,000 recipients emailed
⢠1 GB/day bandwidth
⢠100,000 tasks enqueued
⢠650K XMPP messages/day
53
Tuesday, July 6, 2010
57. Purchase additional resources *
* free monthly quota of ~5 million page views still in full effect
54
Tuesday, July 6, 2010
58. By the numbers
250,000+
Developers
100,000+ Apps
0.5B+ daily
Pageviews
55
Tuesday, July 6, 2010
62. Social networking at scale
>62M Users
3.6M DAUs on Facebook
1.9M DAUs on MySpace
Orkut, Bebo, Hi5,
Friendster, Hyves, Ning, âŚ
58
Tuesday, July 6, 2010
63. Chillingo Crystal
59
Gaming meets Social
Zombie Dash Angry Birds LITE Underground Meltdown Cogs
Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds
Extreme The Fallen King
Tuesday, July 6, 2010
69. Build your Enterprise Apps on Google
⢠Easy to Build - Java standards
⢠Easy to Deploy - push-button deployment
⢠Easy to Scale - from small apps to millions of users
64
Tuesday, July 6, 2010
70. Google App Engine for Business
⢠Centralized administration - controls
⢠Reliability and support - SLA, Premium support
⢠Secure by default - only your users
⢠Pricing that makes sense - pay only for what you use
⢠Enterprise features - hosted SQL, SSL on your domain
65
Tuesday, July 6, 2010
71. Understanding the Cloud Computing
Landscape
SaaS
PaaS
IaaS
66
Source: Gartner AADI Summit Dec 2009
Tuesday, July 6, 2010
72. Google's Cloud Offerings
1. Our Apps
2. 3rd party Apps:
Google Apps Marketplace
SaaS 3. ________
PaaS Google App Engine
Google Storage for Devs
IaaS Machine Learning
BigQuery
67
Tuesday, July 6, 2010
73. Google's Cloud Offerings
Your Apps
1. Our Apps
2. 3rd party Apps:
Google Apps Marketplace
SaaS 3. ________
PaaS Google App Engine
Google Storage for Devs
IaaS Machine Learning
BigQuery
67
Tuesday, July 6, 2010
75. Domain Console
Like the regular admin console
Designed to manage enterprises with a portfolio of apps
⢠Keep track of all apps in a domain
⢠Access Control: view apps, deploy
⢠Global Settings: apply to all apps in the domain
⢠Billing rolling up to single account
⢠DNS configuration done only once: *.ext.example.com
⢠All apps by default for logged in users from domain
69
Tuesday, July 6, 2010
76. Google Apps Integration
⢠SSO/SSO delegation
⢠APIs for most Google Apps for integration
Â
70
Tuesday, July 6, 2010
80. Using Secure Data Connector
Installation
- Determine access rules
- Configure and install SDC
73
Tuesday, July 6, 2010
81. Using Secure Data Connector
Installation
- Determine access rules
- Configure and install SDC
Getting ready to serve
- SDC opens SSL tunnel
73
Tuesday, July 6, 2010
82. Using Secure Data Connector
Installation
- Determine access rules
- Configure and install SDC
Getting ready to serve
- SDC opens SSL tunnel
Serving
- User request sent to App Engine
- User authenticated
- App makes request through tunnel
- SDC performs access checks
- Results returned
73
Tuesday, July 6, 2010
83. App Engine for Business Pricing
Intranet apps:
Each app costs $8 / active user / month
Capped at $1,000 / month (i.e. users above 125 are free)
Apps are auth-restricted to domain users
Development is free
Overage charges on Background Analysis/Storage
Non intranet apps (external/public/ISV apps):
Pricing TBD
Postpaid (i.e. billed at the end of month)
74
Tuesday, July 6, 2010
84. App Engine for Business Support and
SLA
Paid Support
Email based
1000$/month
1h response time on operational issues
8h on development issues
SLA
99.9% uptime
Service credits from 10% to 100% refund of monthly bill
75
Tuesday, July 6, 2010
86. Two years in review
Apr 2008 Python launch
May 2008 Memcache, Images API
Jul 2008 Logs export
Aug 2008 Batch write/delete
Oct 2008 HTTPS support
Dec 2008 Status dashboard, quota details
Feb 2009 Billing, larger files
Apr 2009 Java launch, DB import, cron support, SDC
May 2009 Key-only queries
Jun 2009 Task queues
Aug 2009 Kindless queries
Sep 2009 XMPP
Oct 2009 Incoming email
Dec 2009 Blobstore
Feb 2010 Datastore cursors, Appstats
Mar 2010 Read policies, IPv6
May 2010 App Engine for Business
77
Tuesday, July 6, 2010
88. App Engine
Roadmap
Improved monitoring/alerting
Background servers
SSL for your domain
Control datastore availability vs. latency trade-offs
Datastore dump and restore facility
Mapping operations across datasets
Raise request/response size limits for some APIs
Reserved instances
Built-in support for OAuth & OpenID
Channel API 79
Tuesday, July 6, 2010
90. Google Storage
⢠Cloud-based binary object store
o Structured as buckets and objects
o Many buckets, many objects, large objects
⢠You control your data
o Private, shared, or public
o Get your data back out at any time
⢠For developers
o RESTful API
o Many SDKs + tools
o Integration with other Google services
Tuesday, July 6, 2010
91. Google Storage Benefits
High Performance and Scalability
backed by Google infrastructure
Flexible Authentication &
Sharing Models
Get Started Fast with
Google & 3rd Party Utilities
Tuesday, July 6, 2010
92. Demo: Get Started In < 1 Minute
⢠click emailed invitation link
⢠read & accept Terms Of Service
⢠start using GS Manager
Tuesday, July 6, 2010
93. Google services using Google Storage
Partner Reporting
Data Liberation Haiti Relief Imagery Partner Reporting
Google Google
BigQuery Prediction API
Tuesday, July 6, 2010
96. Google Storage Overview
⢠Fast, scalable, highly available object store
o Objects of any type and practically any size
o Lots of objects, lots of buckets
o All data replicated to multiple US data centers
o Read-your-writes data consistency
⢠Easy, flexible authentication and sharing
o Key-based authentication
o Authenticated downloads from a web browser
o Sharing with individuals and groups
⢠Google products and 3rd party tools/services
o Compatible with many available tools and libraries
o Getting started toolkit
Tuesday, July 6, 2010
97. API Concepts
⢠RESTful API
o Verbs: GET, PUT, POST, HEAD, DELETE
o Resources: identified by URI
⢠Buckets
o Flat containers
⢠Objects
o Any type, practically any size
⢠Access Control for Google Accounts
o Google Groups
⢠Two Ways to Authenticate Requests
o Sign request using access keys
o Web browser login
Tuesday, July 6, 2010
98. Sample Signed Request
PUT /mybucket/My/Long/Object/Name HTTP/1.1
Host: commondatastorage.googleapis.com:443
Accept-Encoding: identity
Date: Sat, 08 May 2010 19:04:21 GMT
Content-Length: 28
Content-Type: text/plain
Authorization:GOOG1
GOOG4622809698762217:J+y3mj5GThfI6Ed1MqLi7JpCq5Y=
Â
This is my object's content.
Tuesday, July 6, 2010
99. Sharing and ACLs
⢠Data can be private or shared
⢠Bucket ACL determines:
o who can list objects (READ)
o who can create / delete objects (WRITE)
o who can read / write bucket ACL (FULL_CONTROL)
⢠Object ACL determines:
o who can read objects (READ)
o who can read / write object ACL (FULL_CONTROL)
Tuesday, July 6, 2010
100. Read-Your-Writes Consistency
⢠Once a write succeeds, all future reads will see a snapshot
that includes that write... no matter what replica they talk to
⢠Once any reader sees a result (even if the write previously
appeared to fail) then all future readers will see a snapshot
that includes the write
Tuesday, July 6, 2010
101. Interoperability Gives You Choice
⢠Data Liberation
o You shouldn't be locked in by your choice of
storage provider
⢠Choice of tools
o You should be able to use the same tools to
manage your data, regardless of where you keep it
Tuesday, July 6, 2010
103. Computing in the Cloud: App Engine Sample
os.environ['BOTO_CONFIG'] = 'boto.cfg'
from boto import storage_uri
# <other imports omitted>
class MainPage(webapp.RequestHandler):
  def get(self):
   self.response.out.write('<html><body>')
   uri = storage_uri('gs://pub/shakespeare/rose.txt')
   poem = uri.get_contents_as_string()
   self.response.out.write('<pre>' + poem + '</pre>')
   self.response.out.write('</body></html>')
def main():
  application = webapp.WSGIApplication([('/', MainPage)])
  run_wsgi_app(application)
if __name__ == "__main__":
  main()
Tuesday, July 6, 2010
104. Pricing and Availability
⢠Pay as you go pricing
⢠Storage - $0.17/GB/month
⢠Network
o Upload data to Google
ď§ $0.10/GB
o Download data from Google
ď§ $0.15/GB for Americas and EMEA
ď§ $0.30/GB for APAC
⢠Requests
o PUT, POST, LIST - $0.01 per 1,000 Requests
o GET, HEAD - $0.01 per 10,000 Requests
⢠Free storage (up to 100GB) during preview period
o No SLA
⢠http://code.google.com/apis/storage
Tuesday, July 6, 2010
105. What's Coming Up
⢠Service Level Agreement
⢠Support
⢠Available to Premium Apps Customers
⢠Technical Features:
o Group support in ACLs
o Resumable uploads
o Additional regions
Tuesday, July 6, 2010
107. Overview
⢠Big Data - Challenging and Important
⢠Google has tools for deep data analysis
⢠Now you can use these tools
Tuesday, July 6, 2010
108. Overview
⢠Big Data - Challenging and Important
⢠Google has tools for deep data analysis
⢠Now you can use these tools
⢠Announcing two new APIs to get more from your data:
1.BigQuery
2.Prediction API
Tuesday, July 6, 2010
109. Benefits
⢠Built on Google technology
⢠Scalability
⢠Security
⢠Sharing
⢠Easy integration with Google App Engine, Google
Spreadsheets, ....
Tuesday, July 6, 2010
110. Using Your Data with BigQuery & Prediction API
1. Upload Upload your data Your Data
to Google Storage
Import to tables BigQuery
2. Process Train a model Prediction API
Run queries Your Apps
3. Act Make predictions
Tuesday, July 6, 2010
111. BigQuery
Interactive Analysis of Big Data
Tuesday, July 6, 2010
112. Big Data is Challenging
Starts with Scale
Tuesday, July 6, 2010
113. Many Use Cases ...
Interactive Tools Trends
Spam Detection
Web Network
Dashboards Optimization
Tuesday, July 6, 2010
114. Demo: Analyzing M-Lab
An open platform for advanced network researchďťż
http://www.measurementlab.net/
Tuesday, July 6, 2010
116. Key Capabilities of BigQuery
⢠Scalable: Billions of rows
⢠Fast: Response in seconds
⢠Simple: Queries in SQL
⢠Web Service
o REST
o JSON-RPC
o Google App Scripts
Tuesday, July 6, 2010
117. Using Your Data with BigQuery
Upload to Google Storage
1. Upload
Import data into a BigQuery Table
2. Import - No need to define indices, keys, etc..
Execute queries via APIs
3. Query - No provisioning machines or resources
Tuesday, July 6, 2010
118. Writing Queries
Compact subset of SQL
o SELECT ... FROM ...
WHERE ...
GROUP BY ... ORDER BY ...
LIMIT ...;
Common functions
o Math, String, Time, ...
Statistical approximations
o TOP
o COUNT DISTINCT
Tuesday, July 6, 2010
119. API in a Minute
GET /bigquery/v1/tables/{table name}
GET /bigquery/v1/query?q={query}
Sample JSON Reply:
{
  "results": {
   "ďŹelds": { [
    {"id":"COUNT(*)","type":"uint64"}, ... ]
   },
   "rows": [
    {"f":[{"v":"2949"}, ...]},
    {"f":[{"v":"5387"}, ...]}, ... ]
  }
}
Also supports JSON-RPC
Tuesday, July 6, 2010
120. Security and Privacy
Standard Google Authentication
⢠Client Login
⢠OAuth
⢠AuthSub
HTTPS support
⢠protects your credentials
⢠protects your data
Use Google Storage for Developers to manage access
Tuesday, July 6, 2010
121. Large Corpus Analysis
Wikimedia Revision History
Wikimedia Revision history data from: http://download.wikimedia.org/enwiki/latest/
enwiki-latest-pages-meta-history.xml.7z
Tuesday, July 6, 2010
122. Using BigQuery Shell
Python DB API 2.0 + B. Clapper's sqlcmd
http://www.clapper.org/software/python/sqlcmd/
Tuesday, July 6, 2010
125. BigQuery Recap
⢠Interactive analysis of very large data sets
⢠Simple SQL query language
⢠APIs enable a variety of use cases
Tuesday, July 6, 2010
127. Prediction API 101
⢠Google's sophisticated machine learning algorithms
⢠Available as an on-demand RESTful HTTP web service
⢠Train a model offline/asynchronously
⢠Predict results in real-time
"Tous pour un,
un pour tous, Prediction API "french"
c'est notre
devise."
Tuesday, July 6, 2010
128. How does it work?
"english" The quick brown fox jumped over the
The Prediction API lazy dog.
finds relevant
"english" To err is human, but to really foul things
features in the up you need a computer.
sample data during
"spanish" No hay mal que por bien no venga.
training.
"spanish" La tercera es la vencida.
The Prediction API
later searches for ? To be or not to be, that is the question.
those features
during prediction. ? La fe mueve montaĂąas.
Tuesday, July 6, 2010
129. A virtually endless number of applications...
Customer Transaction Species Message Diagnostics
Sentiment Risk Identification Routing
Churn Legal Docket Suspicious Work Roster Inappropriate
Prediction Classification Activity Assignment Content
Recommend Political Uplift Email Career
Products Bias Marketing Filtering Counselling
... and many more ...
Tuesday, July 6, 2010
130. Three simple steps to use the Prediction API
Upload your Use the API, gsutil or any
1. Upload training data to compatible utility to upload
Google Storage your data to Google Storage
Build a model prediction/v1/train/{}
2. Train from your data POST : a training request
prediction/v1/query/{}
Make new GET : model info
3. Predict predictions POST : a prediction request
Tuesday, July 6, 2010
131. Prediction API Demo
Automatically categorize and respond to emails by language
⢠Customer: ACME Corp, a multinational organization
⢠Goal: Respond to customer emails in their language
⢠Data: Many emails, tagged with their languages
⢠Outcome: Predict language and respond accordingly
Tuesday, July 6, 2010
132. Step 1: Upload
Upload your training data to Google Storage
⢠Training data: outputs and input features
⢠Data format: comma separated value format (CSV)
$ head -n 2 ${data}
"english","To err is human, but to really ..."
"spanish","No hay mal que por bien no venga."
Upload to Google Storage
$ gsutil cp ${data} gs://io10/${data}
Tuesday, July 6, 2010
133. Step 2: Train
Create a new model by training on data
To train a model:
POST prediction/v1/train/${data}
Training runs asynchronously. To see if it has finished:
GET prediction/v1/query/${data}
{"data": {
   "resource": {
    "data": "${data}",Â
    "modelinfo": "estimated accuracy: ${acc}"}}}
Tuesday, July 6, 2010
134. Step 3: Predict
Apply the trained model to make predictions on new data
POSTÂ prediction/v1/query/${data}
{ data : {Â
  "instance" : {
   "input" : { "text" : [
    "J'aime X! C'est le meilleur" ]}}}}
Tuesday, July 6, 2010
135. Step 3: Predict
Apply the trained model to make predictions on new data
POSTÂ prediction/v1/query/${data}
{ data : {Â
  "instance" : {
   "input" : { "text" : [
    "J'aime X! C'est le meilleur" ]}
   "output" :
   {"output_label" : "french"}}}}
Tuesday, July 6, 2010
136. Step 3: Predict
Apply the trained model to make predictions on new data
import httplib
header = {"Content-Type" : "application/json"}#...put new data in
JSON format in params variable
conn =
httplib.HTTPConnection("www.googleapis.com")conn.request("P
OST",
  "/prediction/v1/query/${data}",  params, header)print
conn.getresponse()
Tuesday, July 6, 2010
137. Prediction API Capabilities
Data
⢠Input Features: numeric or unstructured text
⢠Output: up to 100s of discrete categories
Training
⢠Many machine learning techniques
⢠Automatically selected
⢠Performed asynchronously
Access from many platforms:
⢠Web app from Google App Engine
⢠Apps Script (e.g. from Google Spreadsheet)
⢠Desktop app
Tuesday, July 6, 2010
138. Prediction API and BigQuery Demo: Tagger
Input Data: http://delic.io.us/chanezon
â 6000 urls, 14000 tags in 6 years
Analyze my delicious tags
â use delicious API to get all tagged urls
â cleanup data, resize (100Mb limit)
â PUT data in Google storage
â Define table
â analyze
Predict how I would tag a technology article
â input is tag,url,text
â send new url and text
â get predicted tag
Tuesday, July 6, 2010
139. Get the BigQuery & Prediction APIs
⢠Preview, opened to a limited number of developers
⢠You need a Google Storage for Developers account
⢠To request access and get more information, go to:
o http://code.google.com/apis/bigquery
o http://code.google.com/apis/prediction
Tuesday, July 6, 2010
140. Acknowledgement
Thanks to many Googlers from their slides, most
available at the Google IO 2010 website
131
Tuesday, July 6, 2010
141. Thank you
Read more
http://code.google.com/appengine/
Contact info
Patrick Chanezon
Developer Advocate
chanezon@google.com
http://twitter.com/chanezon
Questions
?
132
Tuesday, July 6, 2010