SlideShare ist ein Scribd-Unternehmen logo

Jupyter, A Platform for Data Science at Scale

•
•
Matthias Bussonnier
Matthias Bussonnier

This document discusses Jupyter, an open-source tool for interactive data science and scientific computing. Jupyter allows for interactive exploration, development, and communication through code, equations, visualizations and narrative text. It supports over 50 programming languages and has found widespread adoption in academia and industry for individual and collaborative work across the entire workflow of a scientific idea from data collection to publication. The document outlines Jupyter's history and architecture, ecosystem of related projects, and future development plans to enhance collaboration and software engineering capabilities.

Technologie
1 von 45
Downloaden Sie, um offline zu lesen
1 A tool for datascience at scale. Matthias Bussonnier (UC Berkeley mbussonnier@berkeley.edu) Slides examples on GitHub: https://github.com/Carreau/talks/tree/master/labtech-2015
Jupyter A bit of History The Notebook Application The document format & Publication Multi-User & scaling The Ecosystem 2
The Lifecycle of a Scientific Idea 1. Individual exploratory work 2. Collaborative development 3. Parallel production runs (HPC, cloud, …) 4. Publication & communication (reproducibly!) 5. Education 6. Goto 1 3
The Lifecycle of a Scientific Idea 1. Individual exploratory work – Matlab command line 2. Collaborative development – email scripts back and forth ? 3. Parallel production runs (HPC, cloud, …) – rewrite Fortran/MPI 4. Publication & communication (reproducibly!) – Copy Past in PPT 5. Education – Specific tools 6. Goto 1 4
Can we have a single tool that cover all the lifecycle of a scientific idea, from data collection to publication ? 5
A bit of History Fernando Perez, 2001, CU Boulder (instead of writing a physics dissertation): Python can replace the collection of bash, perl, C/C++ Script. But the Python REPL can be better. 6
NOVEMBER 2001: "JUST AN AFTERNOON HACK" 259 Line Python script. (https://gist.github.com/fperez/1579699) sys.ps1 -> In [N]. sys.displayhook -> Out[N], caches results. Plotting, Numeric, etc. 2014 (OPENHUB STATS) 19,279 commits 442 contributors Total Lines: 187,326 Number of Languages : 7 (JS, CSS, HTML, …) 7
Improve over the terminal ❖ The REPL as a network protocol ❖ Kernels ❖ execute code ❖ Clients ❖ Read input ❖ Present output Simple abstractions enable rich, sophisticated clients 8
❖ Rich web client ❖ Text & math ❖ Code ❖ Results ❖ Share, reproduce. 2011: The IPython Notebook 9
The Team (people that spend a noticeable amount of time on the project, subjective of course) Fernando Perez (UC Berkeley LBL) Brian Granger (CalPoly) Oberon Lopez (summer student) Cameron Oelsen (summer student) SimonVurens (summer student) Ryan Morshed (summer student) Min Ragan-Kelley (Simula) Thomas Kluyver (UK) Matthias Bussonnier (UC Berkeley) Jon Frederic (Cal Poly) Jess Hamrick (UC Berkeley) Kyle Kelley (Rackspace) Jason Grout (Bloomberg) Sylvain Corlay (Bloomberg) Kester Tong (Google) Nicholas Bollweg Will Whitney (MIT) Damián Avila (Continuum) Steven Silvester (Continuum) Chris Colbert (Continuum) David Willmer (Continuum) Peter Parente (IBM) Dan Gisolfi (IBM) Gino Bustelo (IBM) All 400+ GitHub contributors. Bold:Working full time on IPython/Jupyter, underline: Contribute to Jupyter/IPython with corporate agreement 10
Funding 11
Jupyter vs IPython Network protocol for interactive computing Clients for protocol Console Qt Console Notebook Notebook file format & tools (nbconvert…) JupyterHub Nbviewer NbGrader Tmpnb … Interactive Python shell at the terminal Kernel for this protocol in Python Tools for Cross-Language integration Tools for Interactive Parallel computing The “reference” kernel for Jupyter 12
Why ? Don’t reinvent the wheel: reimplement 1 piece, get the rest for free. You don’t like the frontend, write a new one for Python get 50+ languages that work out of the box with it. (https://github.com/ ipython/ipython/wiki/IPython-kernels-for-other-languages) You don’t like a language, write your own kernel, get all the IDEs, conversion tools. Etc.. 13
The Notebook Try it on https://try.jupyter.org Demo Notebook app, also have a terminal, text editor, increasing number of plugins, and of course support 50 languages. 14
The notebook Web Application, that allow code to produce web-rich representation (images, sound, video, math, …) The Browser, Server, and kernel(s) can be on separate machines. The default application to edit `.ipynb` files. `.ipynb` file are JSON based files embeding input and output, so which can be read & converted without a running kernel. 15
The Notebook Fileformat (`.ipynb`) 16
NbViewer Zero-install reading of notebooks Just share a URL nbviewer.org Under the hood: get raw URL and convert to HTML on the fly. Sharing: git push, or dropbox sync. 17
Nbviewer on GitHub Since May GitHub renders Notebooks Powered by `nbconvert`, the library that deals with `.ipynb` -> * Over 200,000 notebooks on GitHub 18
What content as notebook ? http://www.nature.com/ismej/journal/v7/n3/full/ ismej2012123a.html http://qiime.org/home_static/nih-cloud-apr2012 Papers with code as AMI/VMs 19
Blogs Jake van der Plas @ UW http://blogs.scientificamerican.com/ sa-visual/2014/09/16/visualizing-4- dimensional-asteroids 20
Course, MOOCS 21
Books By Cameron Davidson-PilonBy Matthew Russell By José Unpingco You can download and execute the books locally. 22
Check The Gallery https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks23
Replicating, simpler for readers http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 What if you didn’t had to install anything ? Docker Container Just for you. 24
Replicating, simpler for authors 1. Fixed set of notebooks/dependencies 1. Tmpnb, (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/) 2. CodeNeuro, (http://codeneuro.org/) 2. Build on demand 1. Everware, (https://github.com/everware) 2. Binder, (MyBinder.org) 25
https://github.com/binder-project by Jeremy Freeman (Demo mybinder.org ) 26
The networking architecture (single user) Https/ websocket ZMQ 27
MULTI-USER Jupyter Notebook is Single-User by design. Multi-User enable through JupyterHub - Allow Better scalability - resources monitoring/user - Per-user configuration/version of IPython - Better integration with existing infrastructure 28
Hub Https/websocket proxy Auth & Security Hub 29
Everything* is a plugin • Auth: • Unix PAM (default), OAuth, LDAP… (ie, not yet another thing to manage) • Spawner - Start a single user server for each user. • Localhost, Rackspace, EC2, Docker • Meant for sysadmin, • Deployments relatively involved • Recent software stack (Node.js/Python3) Hub 30
Quick Demo 31
JupyterHub in education Jess Hamrick @ Cal K. Kelley Rackspace M. Ragan-Kelley Simula B. Granger Cal Poly https://developer.rackspace.com/blog/deploying-jupyterhub-for-education ❖ Computationally intensive course, ~220 students ❖ Fully hosted environment, zero-install ❖ Integration with autograding. 32
Deploy at larger scale this fall at UC Berkeley - Data Science 101 - Everyone with CalNet account. New Jupyter in Education Mailing List: https://groups.google.com/forum/#!forum/jupyter-education 33
Ecosystem 34
Non-Notebook projects IDE/Frontends: Atom Hydrogen EIN VIM IPython Rodeo PyCharm MicrosoftVisual Studio 35
Non-Notebook projects RISE (interactive slideshow) runipy (notebooks are report templates) ipymd (store notebook as markdown) NbGrader (grade assignments) Jupyter-Drive (store notebooks on G-drive) pgcontents (store notebooks on PostGres) urth (declarative widget, + dashboard from notebook) 36
Google CoLaboratory Kayur Patel, Kester Tong, Mark Sanders, Corinna Cortes @ Google Matt Turk @ NCSA/UIUC Currently being merge into Jupyter itself. 37
O’Reilly: authoring and delivering executable books Atlas, ipymd and Thebe beta.oreilly.com 38
The Future 39
40
Future work Interactive Computing Notebooks as interactive applications Modular, reusable UI/UX Software engineering with notebooks Computational Narratives nbconvert Element filtering Documentation Collaboration Real time collaboration JupyterHub Sustainability People Events 41
Future work Component and tiled-layout are oft requested feature. Collaboration with Continuum Analytics. Plan on adding panels for Text editor, output, variable inspectors, debuggers, … Discussion with Microsoft PTVS team for “debugger protocol” 42
43
Hiring At UC Berkeley Two new postdocs Project manager Web developer, tech writer (short contracts) One administrative assistant. At Cal Poly Three software engineers (one already hired) One designer One administrative assistant. 44
Time for questions ? Thanks 45

Empfohlen

Introduction to IPython & Jupyter Notebooks
Introduction to IPython & Jupyter NotebooksIntroduction to IPython & Jupyter Notebooks
Introduction to IPython & Jupyter NotebooksEueung Mulyana
 
Best Python Libraries For Data Science & Machine Learning | Edureka
Best Python Libraries For Data Science & Machine Learning | EdurekaBest Python Libraries For Data Science & Machine Learning | Edureka
Best Python Libraries For Data Science & Machine Learning | EdurekaEdureka!
 
Python for data science
Python for data sciencePython for data science
Python for data scienceTanzeel Ahmad Mujahid
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesMatt Harrison
 
Tensorflow presentation
Tensorflow presentationTensorflow presentation
Tensorflow presentationAhmed rebai
 
Python Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaPython Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaEdureka!
 
Introduction to NumPy (PyData SV 2013)
Introduction to NumPy (PyData SV 2013)Introduction to NumPy (PyData SV 2013)
Introduction to NumPy (PyData SV 2013)PyData
 
Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN)Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN)Muhammad Haroon
 

Empfohlen

Introduction to IPython & Jupyter Notebooks
Introduction to IPython & Jupyter NotebooksIntroduction to IPython & Jupyter Notebooks
Introduction to IPython & Jupyter NotebooksEueung Mulyana
 
Best Python Libraries For Data Science & Machine Learning | Edureka
Best Python Libraries For Data Science & Machine Learning | EdurekaBest Python Libraries For Data Science & Machine Learning | Edureka
Best Python Libraries For Data Science & Machine Learning | EdurekaEdureka!
 
Python for data science
Python for data sciencePython for data science
Python for data scienceTanzeel Ahmad Mujahid
 
Learn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesLearn 90% of Python in 90 Minutes
Learn 90% of Python in 90 MinutesMatt Harrison
 
Tensorflow presentation
Tensorflow presentationTensorflow presentation
Tensorflow presentationAhmed rebai
 
Python Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaPython Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaEdureka!
 
Introduction to NumPy (PyData SV 2013)
Introduction to NumPy (PyData SV 2013)Introduction to NumPy (PyData SV 2013)
Introduction to NumPy (PyData SV 2013)PyData
 
Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN)Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN)Muhammad Haroon
 
Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Koan-Sin Tan
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual IntroductionLukas Masuch
 
Machine learning
Machine learningMachine learning
Machine learningRajib Kumar De
 
Web scraping in python
Web scraping in python Web scraping in python
Web scraping in python Viren Rajput
 
Deep Neural Networks for Machine Learning
Deep Neural Networks for Machine LearningDeep Neural Networks for Machine Learning
Deep Neural Networks for Machine LearningJustin Beirold
 
Machine Learning Algorithms
Machine Learning AlgorithmsMachine Learning Algorithms
Machine Learning AlgorithmsHichem Felouat
 
Machine learning in image processing
Machine learning in image processingMachine learning in image processing
Machine learning in image processingData Science Thailand
 
Python Anaconda Tutorial | Edureka
Python Anaconda Tutorial | EdurekaPython Anaconda Tutorial | Edureka
Python Anaconda Tutorial | EdurekaEdureka!
 
PPT on Data Science Using Python
PPT on Data Science Using PythonPPT on Data Science Using Python
PPT on Data Science Using PythonNishantKumar1179
 
Machine Learning
Machine LearningMachine Learning
Machine LearningVivek Garg
 
Image classification using cnn
Image classification using cnnImage classification using cnn
Image classification using cnnDebarko De
 
Intro To Machine Learning in Python
Intro To Machine Learning in PythonIntro To Machine Learning in Python
Intro To Machine Learning in PythonRussel Mahmud
 
Machine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree LearningMachine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree Learningbutest
 
GAN - Theory and Applications
GAN - Theory and ApplicationsGAN - Theory and Applications
GAN - Theory and ApplicationsEmanuele Ghelfi
 
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...Edureka!
 
Introduction to python 3
Introduction to python 3Introduction to python 3
Introduction to python 3Youhei Sakurai
 
Python - the basics
Python - the basicsPython - the basics
Python - the basicsUniversity of Technology
 
Introduction To TensorFlow
Introduction To TensorFlowIntroduction To TensorFlow
Introduction To TensorFlowSpotle.ai
 
Machine Learning Using Python
Machine Learning Using PythonMachine Learning Using Python
Machine Learning Using PythonSavitaHanchinal
 
Object Detection with Tensorflow
Object Detection with TensorflowObject Detection with Tensorflow
Object Detection with TensorflowElifTech
 
LSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopLSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopKorkrid Akepanidtaworn
 
Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorLevi Waldron
 

Weitere ähnliche Inhalte

Was ist angesagt?

Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Koan-Sin Tan
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual IntroductionLukas Masuch
 
Web scraping in python
Web scraping in python Web scraping in python
Web scraping in python Viren Rajput
 
Deep Neural Networks for Machine Learning
Deep Neural Networks for Machine LearningDeep Neural Networks for Machine Learning
Deep Neural Networks for Machine LearningJustin Beirold
 
Machine Learning Algorithms
Machine Learning AlgorithmsMachine Learning Algorithms
Machine Learning AlgorithmsHichem Felouat
 
Machine learning in image processing
Machine learning in image processingMachine learning in image processing
Machine learning in image processingData Science Thailand
 
Python Anaconda Tutorial | Edureka
Python Anaconda Tutorial | EdurekaPython Anaconda Tutorial | Edureka
Python Anaconda Tutorial | EdurekaEdureka!
 
PPT on Data Science Using Python
PPT on Data Science Using PythonPPT on Data Science Using Python
PPT on Data Science Using PythonNishantKumar1179
 
Machine Learning
Machine LearningMachine Learning
Machine LearningVivek Garg
 
Image classification using cnn
Image classification using cnnImage classification using cnn
Image classification using cnnDebarko De
 
Intro To Machine Learning in Python
Intro To Machine Learning in PythonIntro To Machine Learning in Python
Intro To Machine Learning in PythonRussel Mahmud
 
Machine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree LearningMachine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree Learningbutest
 
GAN - Theory and Applications
GAN - Theory and ApplicationsGAN - Theory and Applications
GAN - Theory and ApplicationsEmanuele Ghelfi
 
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...Edureka!
 
Introduction to python 3
Introduction to python 3Introduction to python 3
Introduction to python 3Youhei Sakurai
 
Introduction To TensorFlow
Introduction To TensorFlowIntroduction To TensorFlow
Introduction To TensorFlowSpotle.ai
 
Machine Learning Using Python
Machine Learning Using PythonMachine Learning Using Python
Machine Learning Using PythonSavitaHanchinal
 
Object Detection with Tensorflow
Object Detection with TensorflowObject Detection with Tensorflow
Object Detection with TensorflowElifTech
 

Was ist angesagt? (20)

Introduction to TensorFlow Lite
Introduction to TensorFlow Lite Introduction to TensorFlow Lite
Introduction to TensorFlow Lite
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual Introduction
 
Machine learning
Machine learningMachine learning
Machine learning
 
Web scraping in python
Web scraping in python Web scraping in python
Web scraping in python
 
Deep Neural Networks for Machine Learning
Deep Neural Networks for Machine LearningDeep Neural Networks for Machine Learning
Deep Neural Networks for Machine Learning
 
Machine Learning Algorithms
Machine Learning AlgorithmsMachine Learning Algorithms
Machine Learning Algorithms
 
Machine learning in image processing
Machine learning in image processingMachine learning in image processing
Machine learning in image processing
 
Python Anaconda Tutorial | Edureka
Python Anaconda Tutorial | EdurekaPython Anaconda Tutorial | Edureka
Python Anaconda Tutorial | Edureka
 
PPT on Data Science Using Python
PPT on Data Science Using PythonPPT on Data Science Using Python
PPT on Data Science Using Python
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
 
Image classification using cnn
Image classification using cnnImage classification using cnn
Image classification using cnn
 
Intro To Machine Learning in Python
Intro To Machine Learning in PythonIntro To Machine Learning in Python
Intro To Machine Learning in Python
 
Machine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree LearningMachine Learning 3 - Decision Tree Learning
Machine Learning 3 - Decision Tree Learning
 
GAN - Theory and Applications
GAN - Theory and ApplicationsGAN - Theory and Applications
GAN - Theory and Applications
 
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
 
Introduction to python 3
Introduction to python 3Introduction to python 3
Introduction to python 3
 
Python - the basics
Python - the basicsPython - the basics
Python - the basics
 
Introduction To TensorFlow
Introduction To TensorFlowIntroduction To TensorFlow
Introduction To TensorFlow
 
Machine Learning Using Python
Machine Learning Using PythonMachine Learning Using Python
Machine Learning Using Python
 
Object Detection with Tensorflow
Object Detection with TensorflowObject Detection with Tensorflow
Object Detection with Tensorflow
 

Andere mochten auch

Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorLevi Waldron
 
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformAnalytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformRising Media Ltd.
 
Computational Approaches to Systems Biology
Computational Approaches to Systems BiologyComputational Approaches to Systems Biology
Computational Approaches to Systems BiologyMike Hucka
 
Computational Biology and Bioinformatics
Computational Biology and BioinformaticsComputational Biology and Bioinformatics
Computational Biology and BioinformaticsSharif Shuvo
 
Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102remko caprio
 
MongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceMongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceB1 Systems GmbH
 
The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4turingfan
 
Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Harald Erb
 
Systems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelSystems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelLars Juhl Jensen
 
DNA Information and Creation (PDF)
DNA Information and Creation (PDF)DNA Information and Creation (PDF)
DNA Information and Creation (PDF)Hans Rudolf Tremp
 
IBM - Big Value from Big Data
IBM - Big Value from Big DataIBM - Big Value from Big Data
IBM - Big Value from Big DataWilfried Hoge
 
Systems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemSystems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemLars Juhl Jensen
 
Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?IoT User Group Hamburg
 
Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Annex Publishers
 
Tutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopTutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopVivek Krishnakumar
 
System biology and its tools
System biology and its toolsSystem biology and its tools
System biology and its toolsGaurav Diwakar
 

Andere mochten auch (20)

LSESU a Taste of R Language Workshop
LSESU a Taste of R Language WorkshopLSESU a Taste of R Language Workshop
LSESU a Taste of R Language Workshop
 
Multi-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/BioconductorMulti-omics infrastructure and data for R/Bioconductor
Multi-omics infrastructure and data for R/Bioconductor
 
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-PlattformAnalytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
Analytics meets Big Data – R/Python auf der Hadoop/Spark-Plattform
 
Computational Approaches to Systems Biology
Computational Approaches to Systems BiologyComputational Approaches to Systems Biology
Computational Approaches to Systems Biology
 
Computational Biology and Bioinformatics
Computational Biology and BioinformaticsComputational Biology and Bioinformatics
Computational Biology and Bioinformatics
 
Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102Apps for Science - Elsevier Developer Network Workshop 201102
Apps for Science - Elsevier Developer Network Workshop 201102
 
COMPUTATIONAL BIOLOGY
COMPUTATIONAL BIOLOGYCOMPUTATIONAL BIOLOGY
COMPUTATIONAL BIOLOGY
 
MongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open SourceMongoDB - Big Data mit Open Source
MongoDB - Big Data mit Open Source
 
The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4The Computer Scientist and the Cleaner v4
The Computer Scientist and the Cleaner v4
 
Donald Knuth
Donald KnuthDonald Knuth
Donald Knuth
 
Zwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
Zwischen Browser, Code & Photoshop - aus dem Leben eines WebworkersZwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
Zwischen Browser, Code & Photoshop - aus dem Leben eines Webworkers
 
Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen Do you know what k-Means? Cluster-Analysen
Do you know what k-Means? Cluster-Analysen
 
Systems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems levelSystems biology - Understanding biology at the systems level
Systems biology - Understanding biology at the systems level
 
DNA Information and Creation (PDF)
DNA Information and Creation (PDF)DNA Information and Creation (PDF)
DNA Information and Creation (PDF)
 
IBM - Big Value from Big Data
IBM - Big Value from Big DataIBM - Big Value from Big Data
IBM - Big Value from Big Data
 
Systems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological systemSystems biology: Bioinformatics on complete biological system
Systems biology: Bioinformatics on complete biological system
 
Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?Data Scientist - The Sexiest Job of the 21st Century?
Data Scientist - The Sexiest Job of the 21st Century?
 
Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)Computational Systems Biology (JCSB)
Computational Systems Biology (JCSB)
 
Tutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer WorkshopTutorial 1: Your First Science App - Araport Developer Workshop
Tutorial 1: Your First Science App - Araport Developer Workshop
 
System biology and its tools
System biology and its toolsSystem biology and its tools
System biology and its tools
 

Ähnlich wie Jupyter, A Platform for Data Science at Scale

Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Domino Data Lab
 
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsUsing Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsLuciano Resende
 
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)PyData
 
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoReproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoEGI Federation
 
Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroidsJose Enrique Ruiz
 
Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1inside-BigData.com
 
Behold the Power of Python
Behold the Power of PythonBehold the Power of Python
Behold the Power of PythonSarah Dutkiewicz
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsHenry Schreiner
 
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet
 
Reproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilReproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilChristian Frech
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net DeveloperSarah Dutkiewicz
 
1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptxFEG
 
Python 101 for the .NET Developer
Python 101 for the .NET DeveloperPython 101 for the .NET Developer
Python 101 for the .NET DeveloperSarah Dutkiewicz
 
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Luciano Resende
 
Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Carlos Miguel Ferreira
 
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinalProf. Wim Van Criekinge
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayStrata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayLuciano Resende
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Codemotion
 

Ähnlich wie Jupyter, A Platform for Data Science at Scale (20)

Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...Computable content: Notebooks, containers, and data-centric organizational le...
Computable content: Notebooks, containers, and data-centric organizational le...
 
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsUsing Elyra for COVID-19 Analytics
Using Elyra for COVID-19 Analytics
 
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
IPython: A Modern Vision of Interactive Computing (PyData SV 2013)
 
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and ZenodoReproducible Open Science with EGI Notebooks, Binder and Zenodo
Reproducible Open Science with EGI Notebooks, Binder and Zenodo
 
Jupyter notebooks on steroids
Jupyter notebooks on steroidsJupyter notebooks on steroids
Jupyter notebooks on steroids
 
Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1Introduction to EasyBuild: Tutorial Part 1
Introduction to EasyBuild: Tutorial Part 1
 
Behold the Power of Python
Behold the Power of PythonBehold the Power of Python
Behold the Power of Python
 
PyCon2022 - Building Python Extensions
PyCon2022 - Building Python ExtensionsPyCon2022 - Building Python Extensions
PyCon2022 - Building Python Extensions
 
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
Puppet Camp Boston 2014: Continuous Integration for Hyper-V with Puppet (Begi...
 
Reproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and AndurilReproducible bioinformatics pipelines with Docker and Anduril
Reproducible bioinformatics pipelines with Docker and Anduril
 
Python 101 For The Net Developer
Python 101 For The Net DeveloperPython 101 For The Net Developer
Python 101 For The Net Developer
 
1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx1_International_Google_CoLab_20220307.pptx
1_International_Google_CoLab_20220307.pptx
 
Python 101 for the .NET Developer
Python 101 for the .NET DeveloperPython 101 for the .NET Developer
Python 101 for the .NET Developer
 
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
 
Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.Python 3.5: An agile, general-purpose development language.
Python 3.5: An agile, general-purpose development language.
 
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
2015 bioinformatics python_introduction_wim_vancriekinge_vfinal
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayStrata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
 
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
Luciano Resende - Scaling Big Data Interactive Workloads across Kubernetes Cl...
 
London level39
London level39London level39
London level39
 

Kürzlich hochgeladen

Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 

Kürzlich hochgeladen (20)

Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 

Jupyter, A Platform for Data Science at Scale

  • 1. 1 A tool for datascience at scale. Matthias Bussonnier (UC Berkeley mbussonnier@berkeley.edu) Slides examples on GitHub: https://github.com/Carreau/talks/tree/master/labtech-2015
  • 2. Jupyter A bit of History The Notebook Application The document format & Publication Multi-User & scaling The Ecosystem 2
  • 3. The Lifecycle of a Scientific Idea 1. Individual exploratory work 2. Collaborative development 3. Parallel production runs (HPC, cloud, …) 4. Publication & communication (reproducibly!) 5. Education 6. Goto 1 3
  • 4. The Lifecycle of a Scientific Idea 1. Individual exploratory work – Matlab command line 2. Collaborative development – email scripts back and forth ? 3. Parallel production runs (HPC, cloud, …) – rewrite Fortran/MPI 4. Publication & communication (reproducibly!) – Copy Past in PPT 5. Education – Specific tools 6. Goto 1 4
  • 5. Can we have a single tool that cover all the lifecycle of a scientific idea, from data collection to publication ? 5
  • 6. A bit of History Fernando Perez, 2001, CU Boulder (instead of writing a physics dissertation): Python can replace the collection of bash, perl, C/C++ Script. But the Python REPL can be better. 6
  • 7. NOVEMBER 2001: "JUST AN AFTERNOON HACK" 259 Line Python script. (https://gist.github.com/fperez/1579699) sys.ps1 -> In [N]. sys.displayhook -> Out[N], caches results. Plotting, Numeric, etc. 2014 (OPENHUB STATS) 19,279 commits 442 contributors Total Lines: 187,326 Number of Languages : 7 (JS, CSS, HTML, …) 7
  • 8. Improve over the terminal ❖ The REPL as a network protocol ❖ Kernels ❖ execute code ❖ Clients ❖ Read input ❖ Present output Simple abstractions enable rich, sophisticated clients 8
  • 9. ❖ Rich web client ❖ Text & math ❖ Code ❖ Results ❖ Share, reproduce. 2011: The IPython Notebook 9
  • 10. The Team (people that spend a noticeable amount of time on the project, subjective of course) Fernando Perez (UC Berkeley LBL) Brian Granger (CalPoly) Oberon Lopez (summer student) Cameron Oelsen (summer student) SimonVurens (summer student) Ryan Morshed (summer student) Min Ragan-Kelley (Simula) Thomas Kluyver (UK) Matthias Bussonnier (UC Berkeley) Jon Frederic (Cal Poly) Jess Hamrick (UC Berkeley) Kyle Kelley (Rackspace) Jason Grout (Bloomberg) Sylvain Corlay (Bloomberg) Kester Tong (Google) Nicholas Bollweg Will Whitney (MIT) Damián Avila (Continuum) Steven Silvester (Continuum) Chris Colbert (Continuum) David Willmer (Continuum) Peter Parente (IBM) Dan Gisolfi (IBM) Gino Bustelo (IBM) All 400+ GitHub contributors. Bold:Working full time on IPython/Jupyter, underline: Contribute to Jupyter/IPython with corporate agreement 10
  • 12. Jupyter vs IPython Network protocol for interactive computing Clients for protocol Console Qt Console Notebook Notebook file format & tools (nbconvert…) JupyterHub Nbviewer NbGrader Tmpnb … Interactive Python shell at the terminal Kernel for this protocol in Python Tools for Cross-Language integration Tools for Interactive Parallel computing The “reference” kernel for Jupyter 12
  • 13. Why ? Don’t reinvent the wheel: reimplement 1 piece, get the rest for free. You don’t like the frontend, write a new one for Python get 50+ languages that work out of the box with it. (https://github.com/ ipython/ipython/wiki/IPython-kernels-for-other-languages) You don’t like a language, write your own kernel, get all the IDEs, conversion tools. Etc.. 13
  • 14. The Notebook Try it on https://try.jupyter.org Demo Notebook app, also have a terminal, text editor, increasing number of plugins, and of course support 50 languages. 14
  • 15. The notebook Web Application, that allow code to produce web-rich representation (images, sound, video, math, …) The Browser, Server, and kernel(s) can be on separate machines. The default application to edit `.ipynb` files. `.ipynb` file are JSON based files embeding input and output, so which can be read & converted without a running kernel. 15
  • 16. The Notebook Fileformat (`.ipynb`) 16
  • 17. NbViewer Zero-install reading of notebooks Just share a URL nbviewer.org Under the hood: get raw URL and convert to HTML on the fly. Sharing: git push, or dropbox sync. 17
  • 18. Nbviewer on GitHub Since May GitHub renders Notebooks Powered by `nbconvert`, the library that deals with `.ipynb` -> * Over 200,000 notebooks on GitHub 18
  • 19. What content as notebook ? http://www.nature.com/ismej/journal/v7/n3/full/ ismej2012123a.html http://qiime.org/home_static/nih-cloud-apr2012 Papers with code as AMI/VMs 19
  • 20. Blogs Jake van der Plas @ UW http://blogs.scientificamerican.com/ sa-visual/2014/09/16/visualizing-4- dimensional-asteroids 20
  • 22. Books By Cameron Davidson-PilonBy Matthew Russell By José Unpingco You can download and execute the books locally. 22
  • 24. Replicating, simpler for readers http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 What if you didn’t had to install anything ? Docker Container Just for you. 24
  • 25. Replicating, simpler for authors 1. Fixed set of notebooks/dependencies 1. Tmpnb, (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/) 2. CodeNeuro, (http://codeneuro.org/) 2. Build on demand 1. Everware, (https://github.com/everware) 2. Binder, (MyBinder.org) 25
  • 27. The networking architecture (single user) Https/ websocket ZMQ 27
  • 28. MULTI-USER Jupyter Notebook is Single-User by design. Multi-User enable through JupyterHub - Allow Better scalability - resources monitoring/user - Per-user configuration/version of IPython - Better integration with existing infrastructure 28
  • 30. Everything* is a plugin • Auth: • Unix PAM (default), OAuth, LDAP… (ie, not yet another thing to manage) • Spawner - Start a single user server for each user. • Localhost, Rackspace, EC2, Docker • Meant for sysadmin, • Deployments relatively involved • Recent software stack (Node.js/Python3) Hub 30
  • 32. JupyterHub in education Jess Hamrick @ Cal K. Kelley Rackspace M. Ragan-Kelley Simula B. Granger Cal Poly https://developer.rackspace.com/blog/deploying-jupyterhub-for-education ❖ Computationally intensive course, ~220 students ❖ Fully hosted environment, zero-install ❖ Integration with autograding. 32
  • 33. Deploy at larger scale this fall at UC Berkeley - Data Science 101 - Everyone with CalNet account. New Jupyter in Education Mailing List: https://groups.google.com/forum/#!forum/jupyter-education 33
  • 36. Non-Notebook projects RISE (interactive slideshow) runipy (notebooks are report templates) ipymd (store notebook as markdown) NbGrader (grade assignments) Jupyter-Drive (store notebooks on G-drive) pgcontents (store notebooks on PostGres) urth (declarative widget, + dashboard from notebook) 36
  • 37. Google CoLaboratory Kayur Patel, Kester Tong, Mark Sanders, Corinna Cortes @ Google Matt Turk @ NCSA/UIUC Currently being merge into Jupyter itself. 37
  • 38. O’Reilly: authoring and delivering executable books Atlas, ipymd and Thebe beta.oreilly.com 38
  • 40. 40
  • 41. Future work Interactive Computing Notebooks as interactive applications Modular, reusable UI/UX Software engineering with notebooks Computational Narratives nbconvert Element filtering Documentation Collaboration Real time collaboration JupyterHub Sustainability People Events 41
  • 42. Future work Component and tiled-layout are oft requested feature. Collaboration with Continuum Analytics. Plan on adding panels for Text editor, output, variable inspectors, debuggers, … Discussion with Microsoft PTVS team for “debugger protocol” 42
  • 43. 43
  • 44. Hiring At UC Berkeley Two new postdocs Project manager Web developer, tech writer (short contracts) One administrative assistant. At Cal Poly Three software engineers (one already hired) One designer One administrative assistant. 44
  • 45. Time for questions ? Thanks 45