SlideShare ist ein Scribd-Unternehmen logo
1 von 82
Downloaden Sie, um offline zu lesen
Python, Geodata and
Maps
PyDX Conference, Oct 11th 2015
Hannes Hapke
@hanneshapke
Hello, my name is 

Hannes.
Why could this presentation
be interesting to you?
Location data is key today!
What will you know in 30 min?
• Geospatial basics
• Geocode locations and other sources
• How to make geospatial queries
• How to serve geospatial data via APIs
Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street
Map, Spatial queries, Mapnik, Shapely,
ogr2ogr, kml, GIS, Proj, TIGER,
SpatiaLite, Mapbox, leaflet, raster,
shapefiles, geometries, EPSG, WGS 84,
layers, ArcGIS, bbox, distance loopup,
datum, GEOS, geocode, WKT, WKB,
GeoDjango, GPS, projections, tiles
Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street
Map, Spatial queries, Mapnik, Shapely,
ogr2ogr, kml, GIS, Proj, TIGER,
SpatiaLite, Mapbox, leaflet, raster,
shapefiles, geometries, EPSG, WGS 84,
layers, ArcGIS, bbox, distance loopup,
datum, GEOS, geocode, WKT, WKB,
GeoDjango, GPS, projections, tiles
Geospatial Basics
Just a very few …
Coordinate Systems
Source: https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/LocationAwarenessPG/Art/sphere_to_cylinder.jpg
projected
vs.

unprojected 

coordinate systems
SRID
Spatial Reference System Identifier
spatialreference.org
Remember
4326
Projections
Source: http://gdb.voanews.com/9E85F1AA-B4C0-49FA-ADC0-27BA1FD1C426_mw1024_s_n.jpg
Projections
WGS-84 GCJ-02
Same, same, but different …
GIS Data
raster format
vector format
other formats
Storing Geospatial Data
Sources

SQLite: https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/SQLite370.svg/640px-SQLite370.svg.png

Postgres: http://www.tivix.com/uploads/blog_pics/postgresql_1.png
Postgres is a defacto
standard
Source

PostGIS: http://www.moderncourse.com/wp-content/uploads/2015/03/postgis.png
Thanks to PostGIS
PostGIS, what?
Adds geospatial functionality

to Postgres
Source

PostGIS: https://en.wikipedia.org/wiki/PostGIS
Useful tools …
QGIS
Source

QGIS: https://underdark.files.wordpress.com/2012/07/stamen_watercolor1.png?w=700
CartoDB
Source

CartoDB: http://dmdphilly.org/wp-content/uploads/2015/07/cartodb-editor.2876823e.png
Enough basics!



Let’s get started …
Wait, one last
comment …
Consider the 

user’s privacy …
Sources for Geodata
Geocode Addresses
PyDX Presentation about Python, GeoData and Maps
Geocoder
• pip install geocoder
• Package by Denis Carriere
• Supports 20 geocoding services
• Multiple export data structures
• Command line interface
Code Example
#!/usr/bin/python
import geocoder
"""Sample code to demonstrate geocoding"""
address = "70 NW Couch St, Portland, OR 97209"
result = geocoder.google(address)
print ("Your address: %s" % address)
print ("%s, %s" % (result.lng, result.lat))
print ("Location type: %s" % result.accuracy)
Geolocating IP
Addresses
Code Example
#!/usr/bin/python
import geocoder
"""Sample code to demonstrate the ip geocoder
IP address obtained via http://fakeip.org/"""
ip_address = "192.169.226.73"
result = geocoder.ip(ip_address)
print ("Your ip address: %s" % ip_address)
print ("Location address: %s" % result.address)
print ("%s, %s" % (result.lng, result.lat))
Images as

geodata sources
Code Example
#!/usr/bin/python
from pyexif import pyexif
"""Sample code to demonstrate pyexif"""
file_name = “your_image.JPG"
result = pyexif.Exif(file_name)
print ("Your location: %s %s" % result.lat, result.lon)
GPS Latitude 44.100339 degrees
GPS Longitude121.772294 degrees
GPS Altitude 3109.044444 m
GPS Time Stamp 21:49:20.09
GPS Speed 0.26
GPS Img Direction Ref True North
GPS Img Direction 233.1646707
GPS Dest Bearing Ref True North
GPS Dest Bearing 233.1646707
GPS Date Stamp 2015:08:15
Shapefiles
What are Shapefiles?
Source: http://store.mapsofworld.com/image/cache/data/GIS/Oregon%20County-900x700.jpg
Obtain Shapefiles
# Install the gis stack (geos, proj4, postgres, postgis, gdal)
$ brew install postgis gdal
# Clone the Portland Atlas
$ git clone https://github.com/Caged/portland-atlas.git
$ cd portland-atlas/
$ make shp/neighborhoods.shp
# Check structure
$ ogrinfo -al -geom=OFF shp/neighborhoods.shp
# Check projection
$ ogrinfo -so shp/neighborhoods.shp -sql "SELECT * FROM neighborhoods"
Generate Model from Shapefile
$ python manage.py ogrinspect 
sample_geodjango/data/neighborhoods.shp 
PortlandNeighborhood --srid=4326 --mapping
# This is an auto-generated Django model module created by ogrinspect.
from django.contrib.gis.db import models
class PortlandNeighborhood(models.Model):
objectid = models.FloatField()
name = models.CharField(max_length=50)
shape_leng = models.FloatField()
…
geom = models.PolygonField(srid=4326)
objects = models.GeoManager()
# Auto-generated `LayerMapping` dictionary for PortlandNeighborhood model
portlandneighborhood_mapping = {
'objectid' : 'OBJECTID',
'name' : 'NAME',
'shape_leng' : 'SHAPE_LENG',
…
'geom' : 'POLYGON',
}
Load Shapefile into your DB
import os
from django.contrib.gis.utils import LayerMapping
from .models import Neighborhood
neighboorhood_mapping = {
'name': 'NAME',
'poly': 'POLYGON',
}
neighboorhood_shp = os.path.abspath(
os.path.join(
os.path.dirname(__file__),
'../../data', 'neighborhoods.shp'))
def run(verbose=True):
lm = LayerMapping(
Neighborhood,
neighboorhood_shp, neighboorhood_mapping,
transform=False, encoding='iso-8859-1')
lm.save(strict=True, verbose=verbose)
Now, what to do with
the data?
Generate KML files
with Python
How to create kml data
structures?
• Keyhole Markup Languange (KML)
• XML based
• Developed by Google for Google Earth
• Package SimpleKML

https://pypi.python.org/pypi/simplekml/1.2.8
Code Example
""" Example code which will generate a kml file of
Portland's neighborhoods
Execute this script with the runscript command
of the Django extensions
$ python manage.py runscript simplekml_example -v3
"""
import simplekml
from project.models import Neighborhood
def run():
kml = simplekml.Kml()
neighborhoods = Neighborhood.objects.all()
for neighborhood in neighborhoods:
# this is just bare example
# normally use neighborhood.poly.kml to the kml data
kml.newpolygon(
name=neighborhood.name,
outerboundaryis=list(
neighborhood.poly.boundary.coords[0]),
innerboundaryis=list(
neighborhood.poly.boundary.coords[0])
)
kml.save("portland_neighborhoods.kml")
Google Earth
Generate Static Maps
Motionless
• pip install motionless
• Package by Ryan Cox
• Supports street, satellite, and terrain maps
• Based on Google Static Map API
• Watch out for the rate limit
Code Example
#!/usr/bin/python
"""Sample code to demonstrate motionless"""
import requests
from StringIO import StringIO
from PIL import Image
from slugify import slugify
from motionless import DecoratedMap, AddressMarker
address = "70 NW Couch St, Portland, OR 97209"
# generate static map object
dmap = DecoratedMap(maptype='satellite', zoom=18)
dmap.add_marker(AddressMarker(address))
# download the static image
response = requests.get(dmap.generate_url())
if response.status_code == 200:
image = Image.open(StringIO(response.content))
image.save('.'.join([slugify(address), 'png']))
else:
print ("Download error with status code %s",
response.status_code)
Code Example
But what about
database queries?
GeoDjango
GeoDjango
• Created by Justin Bronn
• Part of the Django core
• Supports various spatial databases and libraries
• Let’s you define geospatial attributes in your
models
GeoDjango
• PointField
• LineStringField
• PolygonField
• MultiPointField
• MultiLineStringField
• MultiPolygonField
GeoDjango Queries
GeoDjango

Spatial Lookups
bbcontains, bboverlaps, contained, contains,
contains_properly, coveredby, covers,
crosses, disjoint, equals, exact, same_as,
intersects, overlaps, relate, touches, within,
left, right, overlaps_left, overlaps_right,
o v e r l a p s _ a b o v e , o v e r l a p s _ b e l o w,
strictly_above, strictly_below
GeoDjango

Distance Lookups
• distance_lt
• distance_lte
• distance_gt
• distance_gte
• dwithin
GeoDjango Example
GeoDjango - Example
• Simple GeoModel
• Demonstrate the GeoDjango Admin
• How to save Points
• How to search for Points within a Polygon
• Leaflet Maps
GeoDjango - Example
GeoDjango - Model
class CallLocation(TimeStampedModel):
address = models.CharField(
max_length=255)
…
location = models.PointField()
objects = models.GeoManager()
def save(self, *args, **kwargs):
try:
# get the geocoding results
result = geocoder.google(self.address)
# correct the address spelling
self.address = result.address.encode('utf-8')
self.location = fromstr(
'POINT(%s %s)' % (result.lng, result.lat),
srid=4326)
except (AttributeError, Exception):
print "Oops! Couldn't geocode address because of %s" 
% sys.exc_info()[0]
super(CallLocation, self).save(*args, **kwargs)
GeoDjango - View
class CallLocationCreateView(CreateView):
model = CallLocation
form = CallLocationForm
success_url = reverse_lazy('home')
fields = ['address', 'call_type', 'comment']
def get_context_data(self, *args, **kwargs):
context = super(
CallLocationCreateView, self
).get_context_data(**kwargs)
context['neighborhoods'] = Neighborhood.objects.all()
context['pk'] = int(self.kwargs.get('pk', -1))
if context['pk'] > 0:
neighborhood = get_object_or_404(Neighborhood, pk=context['pk'])
context['object_list'] = CallLocation.objects.filter(
location__within=neighborhood.poly)
else:
context['object_list'] = CallLocation.objects.all()
return context
GeoDjango - Admin
from django.contrib.gis import admin
from .models import CallLocation, CallType, Neighborhood
class GeoLocationAdmin(admin.OSMGeoAdmin):
default_zoom = 11
list_display = (
'call_type', 'address',
'modified',)
class NeighboorhoodAdmin(admin.OSMGeoAdmin):
default_zoom = 11
list_display = ('name',)
admin.site.register(CallLocation, GeoLocationAdmin)
admin.site.register(Neighborhood, NeighboorhoodAdmin)
admin.site.register(CallType)
GeoDjango - Admin
GeoDjango - Template
<script type="text/javascript">
function initialize() {
var map = new L.Map(
"terrain", {
center: new L.LatLng(45.524221, -122.670749),
zoom: 16
}
);
…
var markers = [];
var bbox = [];
{% for call in object_list %}
markers.push(
L.marker(
[ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ]
).addTo(map));
bbox.push(
[ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ]
);
{% endfor %}
map.fitBounds(bbox);
};
…
</script>
GeoDjango - Example
Django REST
Framework
REST+GIS - Example
• Create an API endpoint
• Set up a GeoSerializer
• AngularJS ng-map
• something very practical …
REST+GIS - Example
What needs to change
in your API code?
REST+GIS - Model
Nothing needs to change
REST+GIS - Serializer
from .models import CallLocation
from rest_framework import serializers
from rest_framework_gis import serializers as gis_serializer
class CallSerializer(gis_serializer.GeoModelSerializer):
call_type = serializers.StringRelatedField()
class Meta:
model = CallLocation
fields = ('status_id', 'address', 'location',
'call_type', 'timestamp')
geo_field = 'point'
pip install djangorestframework-gis
REST+GIS - Serializer
{
"status_id": "650065994463838208",
"address": "2200 NE 36th Ave, Portland, OR 97212, USA",
"location": "SRID=4326;POINT (-122.6259952999999996 45.5384747000000019)",
"call_type": "PROPERTY LOST, FOUND, RECOVERED ",
"timestamp": "2015-10-02T21:53:05Z"
}
{
"status_id": "650864005439840258",
"address": "8900 SW 30th Ave, Portland, OR 97219, USA",
"location": {
"type": "Point",
"coordinates": [
-122.7077201,
45.4608926
]
},
"call_type": "THEFT - COLD ",
"timestamp": "2015-10-05T02:44:06Z"
}
Without the djangorestframework-gis
With the djangorestframework-gis
REST+GIS - View
# Django Packages
# 3rd party
from rest_framework import viewsets
# project packages
from .models import CallLocation
from .serializers import CallSerializer
class CallLocationViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows users to be viewed or edited.
"""
serializer_class = CallSerializer
No magic here.
REST+GIS - Front end
$scope.getMarkers = function () {
$scope.state.markers = [];
Call.query().$promise.then(function(data) {
data.forEach(function(call){
$scope.state.markers.push((
[call.location.coordinates[1],
call.location.coordinates[0]]))
});
}, function(errResponse) {
console.log('REST error');
});
};
...
<div class="container-fluid">
<map
zoom="10" center="[45.5200, -122.6819]"
on-center-changed="getMarkers()">
<marker ng-repeat="p in state.markers"
position="{{ p }}"></marker>
</map>
</div>
AngularJS with ngmap
How to limit the

API requests?
Limit requests by the

using bounding box
# views.py
from rest_framework import viewsets
from rest_framework_gis.filters import InBBOXFilter
class CallLocationViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows users to be viewed or edited.
"""
serializer_class = CallSerializer
# bbox defines which column is used to select the objects
returned
# to the frontend
bbox_filter_field = 'location'
# filter settings
filter_backends = (InBBOXFilter, )
# the overlapping allows polygons to be partially be part of the
bbox
bbox_filter_include_overlapping = True # not needed for
PointFields
pip install django-filters
Limit requests by the
using bounding box
// Change AngularJS call
_getPolygonStr = function (map){
var bounds = [];
bounds[0] = map.getBounds().getSouthWest().lng(); // min lon
bounds[1] = map.getBounds().getSouthWest().lat(); // min lat
bounds[2] = map.getBounds().getNorthEast().lng(); // max lon
bounds[3] = map.getBounds().getNorthEast().lat(); // max lat
return bounds.join(",");
};
...
if ($scope.map) {
bounds = _getPolygonStr($scope.map);
};
...
Call.query({in_bbox: bounds}).$promise.then(function(data) {
data.forEach(function(call){
…
});
}
/api/calls/?in_bbox=-123.093887304,45.327228800,-122.269912695,45.71211299
Recap
• SRID, Projections
• How to get geodata (geocoding)
• How to store it
• How to query it
• How to create an API endpoint
Useful Resources
Useful Resources
• Python Geospatial 

Development



Erik Westra, O’Reilly

Useful Resources
• GeoDjango Tutorial 

https://docs.djangoproject.com/en/1.8/ref/contrib/gis/
• Python & GeoData PyCon 2014 (thanks to @pdxmele): 

https://github.com/pdxmele/python-geodata-bffs
• GeoDjango DjangoCon 2014 (thanks to @aleck_landgraf)

https://www.youtube.com/watch?v=mUhinowr3RY
• Creating maps with D3.js (thanks to @oceankidbilly)

http://wrobstory.github.io/2013/06/creating-with-d3-
dymo.html
Useful Resources
• Fake Addresses (thanks to @alphydan): 

https://fakena.me/random-real-address/
• Portland Atlas (thanks to @Caged)

https://github.com/caged/portland-atlas
• Boston GIS Group

http://www.bostongis.com/
Big thanks to
@alphydan
@aleck_landgraf
@srikanth_chikoo

@patrickbeeson
Thank you.
Questions?
Hannes
@hanneshapke
code

bit.ly/pydx-python-geodata

Weitere ähnliche Inhalte

Ähnlich wie PyDX Presentation about Python, GeoData and Maps

GeoDjango & HTML5 Geolocation
GeoDjango & HTML5 GeolocationGeoDjango & HTML5 Geolocation
GeoDjango & HTML5 GeolocationJohn Paulett
 
Django101 geodjango
Django101 geodjangoDjango101 geodjango
Django101 geodjangoCalvin Cheng
 
Saving Gaia with GeoDjango
Saving Gaia with GeoDjangoSaving Gaia with GeoDjango
Saving Gaia with GeoDjangoCalvin Cheng
 
2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
 
2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locatorAlberto Paro
 
Tips And Tricks For Bioinformatics Software Engineering
Tips And Tricks For Bioinformatics Software EngineeringTips And Tricks For Bioinformatics Software Engineering
Tips And Tricks For Bioinformatics Software Engineeringjtdudley
 
OSCON july 2011
OSCON july 2011OSCON july 2011
OSCON july 2011chelm
 
Server side geo_tools_in_drupal_pnw_2012
Server side geo_tools_in_drupal_pnw_2012Server side geo_tools_in_drupal_pnw_2012
Server side geo_tools_in_drupal_pnw_2012Mack Hardy
 
RasterFrames - FOSS4G NA 2018
RasterFrames - FOSS4G NA 2018RasterFrames - FOSS4G NA 2018
RasterFrames - FOSS4G NA 2018Simeon Fitch
 
RasterFrames: Enabling Global-Scale Geospatial Machine Learning
RasterFrames: Enabling Global-Scale Geospatial Machine LearningRasterFrames: Enabling Global-Scale Geospatial Machine Learning
RasterFrames: Enabling Global-Scale Geospatial Machine LearningAstraea, Inc.
 
D3 Mapping Visualization
D3 Mapping VisualizationD3 Mapping Visualization
D3 Mapping VisualizationSudhir Chowbina
 
Getting Started with Geospatial Data in MongoDB
Getting Started with Geospatial Data in MongoDBGetting Started with Geospatial Data in MongoDB
Getting Started with Geospatial Data in MongoDBMongoDB
 
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece CoLab Athens
 
LocationTech Projects
LocationTech ProjectsLocationTech Projects
LocationTech ProjectsJody Garnett
 
R getting spatial
R getting spatialR getting spatial
R getting spatialFAO
 
State of the Art Web Mapping with Open Source
State of the Art Web Mapping with Open SourceState of the Art Web Mapping with Open Source
State of the Art Web Mapping with Open SourceOSCON Byrum
 
Happy Go Programming
Happy Go ProgrammingHappy Go Programming
Happy Go ProgrammingLin Yo-An
 

Ähnlich wie PyDX Presentation about Python, GeoData and Maps (20)

GeoDjango & HTML5 Geolocation
GeoDjango & HTML5 GeolocationGeoDjango & HTML5 Geolocation
GeoDjango & HTML5 Geolocation
 
Django101 geodjango
Django101 geodjangoDjango101 geodjango
Django101 geodjango
 
Saving Gaia with GeoDjango
Saving Gaia with GeoDjangoSaving Gaia with GeoDjango
Saving Gaia with GeoDjango
 
2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator
 
2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator
 
Tips And Tricks For Bioinformatics Software Engineering
Tips And Tricks For Bioinformatics Software EngineeringTips And Tricks For Bioinformatics Software Engineering
Tips And Tricks For Bioinformatics Software Engineering
 
OSCON july 2011
OSCON july 2011OSCON july 2011
OSCON july 2011
 
Elk stack
Elk stackElk stack
Elk stack
 
10. R getting spatial
10.  R getting spatial10.  R getting spatial
10. R getting spatial
 
Server side geo_tools_in_drupal_pnw_2012
Server side geo_tools_in_drupal_pnw_2012Server side geo_tools_in_drupal_pnw_2012
Server side geo_tools_in_drupal_pnw_2012
 
RasterFrames - FOSS4G NA 2018
RasterFrames - FOSS4G NA 2018RasterFrames - FOSS4G NA 2018
RasterFrames - FOSS4G NA 2018
 
RasterFrames: Enabling Global-Scale Geospatial Machine Learning
RasterFrames: Enabling Global-Scale Geospatial Machine LearningRasterFrames: Enabling Global-Scale Geospatial Machine Learning
RasterFrames: Enabling Global-Scale Geospatial Machine Learning
 
D3 Mapping Visualization
D3 Mapping VisualizationD3 Mapping Visualization
D3 Mapping Visualization
 
Getting Started with Geospatial Data in MongoDB
Getting Started with Geospatial Data in MongoDBGetting Started with Geospatial Data in MongoDB
Getting Started with Geospatial Data in MongoDB
 
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece
Ioannis Doxaras on GIS and Gmaps at 1st GTUG meetup Greece
 
Pycon2011
Pycon2011Pycon2011
Pycon2011
 
LocationTech Projects
LocationTech ProjectsLocationTech Projects
LocationTech Projects
 
R getting spatial
R getting spatialR getting spatial
R getting spatial
 
State of the Art Web Mapping with Open Source
State of the Art Web Mapping with Open SourceState of the Art Web Mapping with Open Source
State of the Art Web Mapping with Open Source
 
Happy Go Programming
Happy Go ProgrammingHappy Go Programming
Happy Go Programming
 

Mehr von Hannes Hapke

PDXPortland - Dockerize Django
PDXPortland - Dockerize DjangoPDXPortland - Dockerize Django
PDXPortland - Dockerize DjangoHannes Hapke
 
Introduction to Convolutional Neural Networks
Introduction to Convolutional Neural NetworksIntroduction to Convolutional Neural Networks
Introduction to Convolutional Neural NetworksHannes Hapke
 
Introduction to Neural Networks - Perceptron
Introduction to Neural Networks - PerceptronIntroduction to Neural Networks - Perceptron
Introduction to Neural Networks - PerceptronHannes Hapke
 
Share your code with the Python world by
 creating pip packages
Share your code with the Python world by
 creating pip packagesShare your code with the Python world by
 creating pip packages
Share your code with the Python world by
 creating pip packagesHannes Hapke
 
Create responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSCreate responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSHannes Hapke
 
Python Ecosystem for Beginners - PyCon Uruguay 2013
Python Ecosystem for Beginners - PyCon Uruguay 2013Python Ecosystem for Beginners - PyCon Uruguay 2013
Python Ecosystem for Beginners - PyCon Uruguay 2013Hannes Hapke
 

Mehr von Hannes Hapke (6)

PDXPortland - Dockerize Django
PDXPortland - Dockerize DjangoPDXPortland - Dockerize Django
PDXPortland - Dockerize Django
 
Introduction to Convolutional Neural Networks
Introduction to Convolutional Neural NetworksIntroduction to Convolutional Neural Networks
Introduction to Convolutional Neural Networks
 
Introduction to Neural Networks - Perceptron
Introduction to Neural Networks - PerceptronIntroduction to Neural Networks - Perceptron
Introduction to Neural Networks - Perceptron
 
Share your code with the Python world by
 creating pip packages
Share your code with the Python world by
 creating pip packagesShare your code with the Python world by
 creating pip packages
Share your code with the Python world by
 creating pip packages
 
Create responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJSCreate responsive websites with Django, REST and AngularJS
Create responsive websites with Django, REST and AngularJS
 
Python Ecosystem for Beginners - PyCon Uruguay 2013
Python Ecosystem for Beginners - PyCon Uruguay 2013Python Ecosystem for Beginners - PyCon Uruguay 2013
Python Ecosystem for Beginners - PyCon Uruguay 2013
 

Kürzlich hochgeladen

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 

Kürzlich hochgeladen (17)

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 

PyDX Presentation about Python, GeoData and Maps

  • 1. Python, Geodata and Maps PyDX Conference, Oct 11th 2015 Hannes Hapke @hanneshapke
  • 2. Hello, my name is 
 Hannes.
  • 3. Why could this presentation be interesting to you? Location data is key today!
  • 4. What will you know in 30 min? • Geospatial basics • Geocode locations and other sources • How to make geospatial queries • How to serve geospatial data via APIs
  • 5. Broad Field … SRID, PostGIS, GDAL, tiles, Open Street Map, Spatial queries, Mapnik, Shapely, ogr2ogr, kml, GIS, Proj, TIGER, SpatiaLite, Mapbox, leaflet, raster, shapefiles, geometries, EPSG, WGS 84, layers, ArcGIS, bbox, distance loopup, datum, GEOS, geocode, WKT, WKB, GeoDjango, GPS, projections, tiles
  • 6. Broad Field … SRID, PostGIS, GDAL, tiles, Open Street Map, Spatial queries, Mapnik, Shapely, ogr2ogr, kml, GIS, Proj, TIGER, SpatiaLite, Mapbox, leaflet, raster, shapefiles, geometries, EPSG, WGS 84, layers, ArcGIS, bbox, distance loopup, datum, GEOS, geocode, WKT, WKB, GeoDjango, GPS, projections, tiles
  • 8. Just a very few …
  • 10. SRID Spatial Reference System Identifier spatialreference.org
  • 14. GIS Data raster format vector format other formats
  • 15. Storing Geospatial Data Sources
 SQLite: https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/SQLite370.svg/640px-SQLite370.svg.png
 Postgres: http://www.tivix.com/uploads/blog_pics/postgresql_1.png
  • 16. Postgres is a defacto standard Source
 PostGIS: http://www.moderncourse.com/wp-content/uploads/2015/03/postgis.png Thanks to PostGIS
  • 17. PostGIS, what? Adds geospatial functionality
 to Postgres Source
 PostGIS: https://en.wikipedia.org/wiki/PostGIS
  • 27. Geocoder • pip install geocoder • Package by Denis Carriere • Supports 20 geocoding services • Multiple export data structures • Command line interface
  • 28. Code Example #!/usr/bin/python import geocoder """Sample code to demonstrate geocoding""" address = "70 NW Couch St, Portland, OR 97209" result = geocoder.google(address) print ("Your address: %s" % address) print ("%s, %s" % (result.lng, result.lat)) print ("Location type: %s" % result.accuracy)
  • 30. Code Example #!/usr/bin/python import geocoder """Sample code to demonstrate the ip geocoder IP address obtained via http://fakeip.org/""" ip_address = "192.169.226.73" result = geocoder.ip(ip_address) print ("Your ip address: %s" % ip_address) print ("Location address: %s" % result.address) print ("%s, %s" % (result.lng, result.lat))
  • 32. Code Example #!/usr/bin/python from pyexif import pyexif """Sample code to demonstrate pyexif""" file_name = “your_image.JPG" result = pyexif.Exif(file_name) print ("Your location: %s %s" % result.lat, result.lon) GPS Latitude 44.100339 degrees GPS Longitude121.772294 degrees GPS Altitude 3109.044444 m GPS Time Stamp 21:49:20.09 GPS Speed 0.26 GPS Img Direction Ref True North GPS Img Direction 233.1646707 GPS Dest Bearing Ref True North GPS Dest Bearing 233.1646707 GPS Date Stamp 2015:08:15
  • 34. What are Shapefiles? Source: http://store.mapsofworld.com/image/cache/data/GIS/Oregon%20County-900x700.jpg
  • 35. Obtain Shapefiles # Install the gis stack (geos, proj4, postgres, postgis, gdal) $ brew install postgis gdal # Clone the Portland Atlas $ git clone https://github.com/Caged/portland-atlas.git $ cd portland-atlas/ $ make shp/neighborhoods.shp # Check structure $ ogrinfo -al -geom=OFF shp/neighborhoods.shp # Check projection $ ogrinfo -so shp/neighborhoods.shp -sql "SELECT * FROM neighborhoods"
  • 36. Generate Model from Shapefile $ python manage.py ogrinspect sample_geodjango/data/neighborhoods.shp PortlandNeighborhood --srid=4326 --mapping # This is an auto-generated Django model module created by ogrinspect. from django.contrib.gis.db import models class PortlandNeighborhood(models.Model): objectid = models.FloatField() name = models.CharField(max_length=50) shape_leng = models.FloatField() … geom = models.PolygonField(srid=4326) objects = models.GeoManager() # Auto-generated `LayerMapping` dictionary for PortlandNeighborhood model portlandneighborhood_mapping = { 'objectid' : 'OBJECTID', 'name' : 'NAME', 'shape_leng' : 'SHAPE_LENG', … 'geom' : 'POLYGON', }
  • 37. Load Shapefile into your DB import os from django.contrib.gis.utils import LayerMapping from .models import Neighborhood neighboorhood_mapping = { 'name': 'NAME', 'poly': 'POLYGON', } neighboorhood_shp = os.path.abspath( os.path.join( os.path.dirname(__file__), '../../data', 'neighborhoods.shp')) def run(verbose=True): lm = LayerMapping( Neighborhood, neighboorhood_shp, neighboorhood_mapping, transform=False, encoding='iso-8859-1') lm.save(strict=True, verbose=verbose)
  • 38. Now, what to do with the data?
  • 40. How to create kml data structures? • Keyhole Markup Languange (KML) • XML based • Developed by Google for Google Earth • Package SimpleKML
 https://pypi.python.org/pypi/simplekml/1.2.8
  • 41. Code Example """ Example code which will generate a kml file of Portland's neighborhoods Execute this script with the runscript command of the Django extensions $ python manage.py runscript simplekml_example -v3 """ import simplekml from project.models import Neighborhood def run(): kml = simplekml.Kml() neighborhoods = Neighborhood.objects.all() for neighborhood in neighborhoods: # this is just bare example # normally use neighborhood.poly.kml to the kml data kml.newpolygon( name=neighborhood.name, outerboundaryis=list( neighborhood.poly.boundary.coords[0]), innerboundaryis=list( neighborhood.poly.boundary.coords[0]) ) kml.save("portland_neighborhoods.kml")
  • 44. Motionless • pip install motionless • Package by Ryan Cox • Supports street, satellite, and terrain maps • Based on Google Static Map API • Watch out for the rate limit
  • 45. Code Example #!/usr/bin/python """Sample code to demonstrate motionless""" import requests from StringIO import StringIO from PIL import Image from slugify import slugify from motionless import DecoratedMap, AddressMarker address = "70 NW Couch St, Portland, OR 97209" # generate static map object dmap = DecoratedMap(maptype='satellite', zoom=18) dmap.add_marker(AddressMarker(address)) # download the static image response = requests.get(dmap.generate_url()) if response.status_code == 200: image = Image.open(StringIO(response.content)) image.save('.'.join([slugify(address), 'png'])) else: print ("Download error with status code %s", response.status_code)
  • 49. GeoDjango • Created by Justin Bronn • Part of the Django core • Supports various spatial databases and libraries • Let’s you define geospatial attributes in your models
  • 50. GeoDjango • PointField • LineStringField • PolygonField • MultiPointField • MultiLineStringField • MultiPolygonField
  • 52. GeoDjango
 Spatial Lookups bbcontains, bboverlaps, contained, contains, contains_properly, coveredby, covers, crosses, disjoint, equals, exact, same_as, intersects, overlaps, relate, touches, within, left, right, overlaps_left, overlaps_right, o v e r l a p s _ a b o v e , o v e r l a p s _ b e l o w, strictly_above, strictly_below
  • 53. GeoDjango
 Distance Lookups • distance_lt • distance_lte • distance_gt • distance_gte • dwithin
  • 55. GeoDjango - Example • Simple GeoModel • Demonstrate the GeoDjango Admin • How to save Points • How to search for Points within a Polygon • Leaflet Maps
  • 57. GeoDjango - Model class CallLocation(TimeStampedModel): address = models.CharField( max_length=255) … location = models.PointField() objects = models.GeoManager() def save(self, *args, **kwargs): try: # get the geocoding results result = geocoder.google(self.address) # correct the address spelling self.address = result.address.encode('utf-8') self.location = fromstr( 'POINT(%s %s)' % (result.lng, result.lat), srid=4326) except (AttributeError, Exception): print "Oops! Couldn't geocode address because of %s" % sys.exc_info()[0] super(CallLocation, self).save(*args, **kwargs)
  • 58. GeoDjango - View class CallLocationCreateView(CreateView): model = CallLocation form = CallLocationForm success_url = reverse_lazy('home') fields = ['address', 'call_type', 'comment'] def get_context_data(self, *args, **kwargs): context = super( CallLocationCreateView, self ).get_context_data(**kwargs) context['neighborhoods'] = Neighborhood.objects.all() context['pk'] = int(self.kwargs.get('pk', -1)) if context['pk'] > 0: neighborhood = get_object_or_404(Neighborhood, pk=context['pk']) context['object_list'] = CallLocation.objects.filter( location__within=neighborhood.poly) else: context['object_list'] = CallLocation.objects.all() return context
  • 59. GeoDjango - Admin from django.contrib.gis import admin from .models import CallLocation, CallType, Neighborhood class GeoLocationAdmin(admin.OSMGeoAdmin): default_zoom = 11 list_display = ( 'call_type', 'address', 'modified',) class NeighboorhoodAdmin(admin.OSMGeoAdmin): default_zoom = 11 list_display = ('name',) admin.site.register(CallLocation, GeoLocationAdmin) admin.site.register(Neighborhood, NeighboorhoodAdmin) admin.site.register(CallType)
  • 61. GeoDjango - Template <script type="text/javascript"> function initialize() { var map = new L.Map( "terrain", { center: new L.LatLng(45.524221, -122.670749), zoom: 16 } ); … var markers = []; var bbox = []; {% for call in object_list %} markers.push( L.marker( [ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ] ).addTo(map)); bbox.push( [ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ] ); {% endfor %} map.fitBounds(bbox); }; … </script>
  • 64. REST+GIS - Example • Create an API endpoint • Set up a GeoSerializer • AngularJS ng-map • something very practical …
  • 66. What needs to change in your API code?
  • 67. REST+GIS - Model Nothing needs to change
  • 68. REST+GIS - Serializer from .models import CallLocation from rest_framework import serializers from rest_framework_gis import serializers as gis_serializer class CallSerializer(gis_serializer.GeoModelSerializer): call_type = serializers.StringRelatedField() class Meta: model = CallLocation fields = ('status_id', 'address', 'location', 'call_type', 'timestamp') geo_field = 'point' pip install djangorestframework-gis
  • 69. REST+GIS - Serializer { "status_id": "650065994463838208", "address": "2200 NE 36th Ave, Portland, OR 97212, USA", "location": "SRID=4326;POINT (-122.6259952999999996 45.5384747000000019)", "call_type": "PROPERTY LOST, FOUND, RECOVERED ", "timestamp": "2015-10-02T21:53:05Z" } { "status_id": "650864005439840258", "address": "8900 SW 30th Ave, Portland, OR 97219, USA", "location": { "type": "Point", "coordinates": [ -122.7077201, 45.4608926 ] }, "call_type": "THEFT - COLD ", "timestamp": "2015-10-05T02:44:06Z" } Without the djangorestframework-gis With the djangorestframework-gis
  • 70. REST+GIS - View # Django Packages # 3rd party from rest_framework import viewsets # project packages from .models import CallLocation from .serializers import CallSerializer class CallLocationViewSet(viewsets.ModelViewSet): """ API endpoint that allows users to be viewed or edited. """ serializer_class = CallSerializer No magic here.
  • 71. REST+GIS - Front end $scope.getMarkers = function () { $scope.state.markers = []; Call.query().$promise.then(function(data) { data.forEach(function(call){ $scope.state.markers.push(( [call.location.coordinates[1], call.location.coordinates[0]])) }); }, function(errResponse) { console.log('REST error'); }); }; ... <div class="container-fluid"> <map zoom="10" center="[45.5200, -122.6819]" on-center-changed="getMarkers()"> <marker ng-repeat="p in state.markers" position="{{ p }}"></marker> </map> </div> AngularJS with ngmap
  • 72. How to limit the
 API requests?
  • 73. Limit requests by the
 using bounding box # views.py from rest_framework import viewsets from rest_framework_gis.filters import InBBOXFilter class CallLocationViewSet(viewsets.ModelViewSet): """ API endpoint that allows users to be viewed or edited. """ serializer_class = CallSerializer # bbox defines which column is used to select the objects returned # to the frontend bbox_filter_field = 'location' # filter settings filter_backends = (InBBOXFilter, ) # the overlapping allows polygons to be partially be part of the bbox bbox_filter_include_overlapping = True # not needed for PointFields pip install django-filters
  • 74. Limit requests by the using bounding box // Change AngularJS call _getPolygonStr = function (map){ var bounds = []; bounds[0] = map.getBounds().getSouthWest().lng(); // min lon bounds[1] = map.getBounds().getSouthWest().lat(); // min lat bounds[2] = map.getBounds().getNorthEast().lng(); // max lon bounds[3] = map.getBounds().getNorthEast().lat(); // max lat return bounds.join(","); }; ... if ($scope.map) { bounds = _getPolygonStr($scope.map); }; ... Call.query({in_bbox: bounds}).$promise.then(function(data) { data.forEach(function(call){ … }); } /api/calls/?in_bbox=-123.093887304,45.327228800,-122.269912695,45.71211299
  • 75. Recap • SRID, Projections • How to get geodata (geocoding) • How to store it • How to query it • How to create an API endpoint
  • 77. Useful Resources • Python Geospatial 
 Development
 
 Erik Westra, O’Reilly

  • 78. Useful Resources • GeoDjango Tutorial 
 https://docs.djangoproject.com/en/1.8/ref/contrib/gis/ • Python & GeoData PyCon 2014 (thanks to @pdxmele): 
 https://github.com/pdxmele/python-geodata-bffs • GeoDjango DjangoCon 2014 (thanks to @aleck_landgraf)
 https://www.youtube.com/watch?v=mUhinowr3RY • Creating maps with D3.js (thanks to @oceankidbilly)
 http://wrobstory.github.io/2013/06/creating-with-d3- dymo.html
  • 79. Useful Resources • Fake Addresses (thanks to @alphydan): 
 https://fakena.me/random-real-address/ • Portland Atlas (thanks to @Caged)
 https://github.com/caged/portland-atlas • Boston GIS Group
 http://www.bostongis.com/