Suche senden
Hochladen
ソーシャルアプリでの Amazon Elastic MapReduce 活用事例
•
11 gefällt mir
•
4,443 views
T
Takahiro Kamatani
Folgen
TokyoWebmining #7 buhii
Weniger lesen
Mehr lesen
Melden
Teilen
Melden
Teilen
1 von 37
Jetzt herunterladen
Downloaden Sie, um offline zu lesen
Empfohlen
800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理
Tatsuya Sasaki
961万人の食卓を支えるデータ解析
961万人の食卓を支えるデータ解析
Tatsuya Sasaki
Hadoopを業務で使ってみた
Hadoopを業務で使ってみた
Tatsuya Sasaki
2015 FOSS4G Track - Building Lightweight Mapping Apps with Esri Leaflet by An...
2015 FOSS4G Track - Building Lightweight Mapping Apps with Esri Leaflet by An...
GIS in the Rockies
Gunosy.go#2 package/compress
Gunosy.go#2 package/compress
Shunsuke Aihara
MapReduce解説
MapReduce解説
Shunsuke Aihara
The Outcome Economy
The Outcome Economy
Helge Tennø
The Six Highest Performing B2B Blog Post Formats
The Six Highest Performing B2B Blog Post Formats
Barry Feldman
Empfohlen
800万人の"食べたい"をHadoopで分散処理
800万人の"食べたい"をHadoopで分散処理
Tatsuya Sasaki
961万人の食卓を支えるデータ解析
961万人の食卓を支えるデータ解析
Tatsuya Sasaki
Hadoopを業務で使ってみた
Hadoopを業務で使ってみた
Tatsuya Sasaki
2015 FOSS4G Track - Building Lightweight Mapping Apps with Esri Leaflet by An...
2015 FOSS4G Track - Building Lightweight Mapping Apps with Esri Leaflet by An...
GIS in the Rockies
Gunosy.go#2 package/compress
Gunosy.go#2 package/compress
Shunsuke Aihara
MapReduce解説
MapReduce解説
Shunsuke Aihara
The Outcome Economy
The Outcome Economy
Helge Tennø
The Six Highest Performing B2B Blog Post Formats
The Six Highest Performing B2B Blog Post Formats
Barry Feldman
GeekCamp SG 2009 - CouchApps with CouchDB
GeekCamp SG 2009 - CouchApps with CouchDB
Arun Thampi
Community in action leroy merlin case study - nuxeo world 2010
Community in action leroy merlin case study - nuxeo world 2010
Nuxeo
Responsive web design - Drupal theming
Responsive web design - Drupal theming
adifferentdesign
ScaleCamp 2009 - Last.fm vs Xbox
ScaleCamp 2009 - Last.fm vs Xbox
davidsingleton
COOKPADでのHadoop利用
COOKPADでのHadoop利用
Tatsuya Sasaki
Migrando do App Engine para o Heroku
Migrando do App Engine para o Heroku
Filipe Ximenes
Rango
Rango
elliando dias
Chef - Configuration Management for the Cloud
Chef - Configuration Management for the Cloud
James Casey
Weitere ähnliche Inhalte
Ähnlich wie ソーシャルアプリでの Amazon Elastic MapReduce 活用事例
GeekCamp SG 2009 - CouchApps with CouchDB
GeekCamp SG 2009 - CouchApps with CouchDB
Arun Thampi
Community in action leroy merlin case study - nuxeo world 2010
Community in action leroy merlin case study - nuxeo world 2010
Nuxeo
Responsive web design - Drupal theming
Responsive web design - Drupal theming
adifferentdesign
ScaleCamp 2009 - Last.fm vs Xbox
ScaleCamp 2009 - Last.fm vs Xbox
davidsingleton
COOKPADでのHadoop利用
COOKPADでのHadoop利用
Tatsuya Sasaki
Migrando do App Engine para o Heroku
Migrando do App Engine para o Heroku
Filipe Ximenes
Rango
Rango
elliando dias
Chef - Configuration Management for the Cloud
Chef - Configuration Management for the Cloud
James Casey
Ähnlich wie ソーシャルアプリでの Amazon Elastic MapReduce 活用事例
(8)
GeekCamp SG 2009 - CouchApps with CouchDB
GeekCamp SG 2009 - CouchApps with CouchDB
Community in action leroy merlin case study - nuxeo world 2010
Community in action leroy merlin case study - nuxeo world 2010
Responsive web design - Drupal theming
Responsive web design - Drupal theming
ScaleCamp 2009 - Last.fm vs Xbox
ScaleCamp 2009 - Last.fm vs Xbox
COOKPADでのHadoop利用
COOKPADでのHadoop利用
Migrando do App Engine para o Heroku
Migrando do App Engine para o Heroku
Rango
Rango
Chef - Configuration Management for the Cloud
Chef - Configuration Management for the Cloud
ソーシャルアプリでの Amazon Elastic MapReduce 活用事例
1.
Amazon Elastic MapReduce
Takahiro Kamatani gumi, Inc. 2010/09/26 Sunday, September 26, 2010
2.
•
• • Amazon Elastic MapReduce • Sunday, September 26, 2010
3.
• Twitter: @buhii
• • gumi @ http://www.kansei.tsukuba.ac.jp/~uchiyamalab/beacon • • beacon • gumi @ynil Sunday, September 26, 2010
4.
Sunday, September 26,
2010
5.
gumi
• mixi, , GREE • python Django • Amazon Web Services (EC2 + RDS) • • DB Sunday, September 26, 2010
6.
• PV
, UU • DAU Daily Active Users • • • ÷ DAU • ARPU Average Revenue Per User • Sunday, September 26, 2010
7.
Amazon Web Service
AWS Sunday, September 26, 2010
8.
Sunday, September 26,
2010
9.
Sunday, September 26,
2010
10.
Sunday, September 26,
2010
11.
Sunday, September 26,
2010
12.
Sunday, September 26,
2010
13.
Amazon Elastic MapReduce Sunday,
September 26, 2010
14.
MapReduce
Mapper Key, Value Mapper key Sort / Shuffle Reducer Reducer key, value Mapper, Reducer Sunday, September 26, 2010
15.
Amazon Elastic MapReduce
• Hadoop • Hadoop Streaming Mapper Reducer Ruby, Perl, Python, PHP, R, Bash, C++ • EC2 job • Sunday, September 26, 2010
16.
Example Task
• • • Sunday, September 26, 2010
17.
• Mapper
• Apache Log • ID key value Reducer Sunday, September 26, 2010
18.
• Reducer
• sort/shuffle ID Reducer • ID Sunday, September 26, 2010
19.
Reducer
31758623 2010-08-20 42346572 2010-09-05,2010-09-06 31977736 2010-08-11,2010-08-12,2010-08-13,2010-08-14 14007991 2010-08-16 35995849 2010-08-12,2010-08-13,2010-08-14 34246688 2010-08-21,2010-08-22,2010-08-23,2010-08-27 ... PC Sunday, September 26, 2010
20.
Amazon Elastic Mapreduce
• AWS • S3 Mapper, Reducer → s3cmd, S3Fox Organizer, Cyberduck • Job OK • Sunday, September 26, 2010
21.
Sunday, September 26,
2010
22.
Sunday, September 26,
2010
23.
Streaming Sunday, September 26,
2010
24.
{Input, Output} Location,
Mapper, Reducer S3 gzip Hadoop Extra Args -jobconf stream.recordreader.compression=gzip input Location Extra Args -input s3n://(bucket )/( )/access_log.* Sunday, September 26, 2010
25.
Sunday, September 26,
2010
26.
Sunday, September 26,
2010
27.
Sunday, September 26,
2010
28.
Debug Sunday, September 26,
2010
29.
Sunday, September 26,
2010
30.
• Hadoop
• MapReduce • • Sunday, September 26, 2010
31.
Hadoop
• • S3 gzip • • hadoop EC2 • ( 20 ...) Sunday, September 26, 2010
32.
@ynil
MapReduce http://nlpyutori.g.hatena.ne.jp/yaruki_nil/20100911/1284089305 Sunday, September 26, 2010
33.
Sunday, September 26,
2010
34.
Sunday, September 26,
2010
35.
Sunday, September 26,
2010
36.
MapReduce
MapReduce Google Map Reduce Map Reduce MapReduce C++ Java Python Wikipedia “MapReduce” http://ja.wikipedia.org/wiki/MapReduce Sunday, September 26, 2010
37.
cron
• PV, UU NFS CSV • DB → DB • PV, UU Sunday, September 26, 2010
Jetzt herunterladen