IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018. A quick talk on how to ingest IoT sensor data, camera images and run deep learning prebuilt models on edge devices including Raspberry Pis and NVidia Jetson TX1s. From there data is processed and orchestrated with Apache NiFi to send to various Big Data backends.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
IoT Edge Processing with
Apache NiFi and MiniFi and
Apache MXNet for IoT NY 2018
Timothy Spann, Senior Solutions Engineer
Hortonworks @PaaSDev

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved.
Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved.
What do we want to do?
• MiniFi ingests camera images and
sensor data
• MiniFi executes Apache MXNet at
the edge
• Run Apache MXNet Inception to
recognize objects in image
• Apache NiFi stores images,
metadata and enriched data in
Hadoop

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved.
Aggregate all data from sensors, drones, cameras, logs, geo-location
devices, images, machines and social feeds.
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Files, Apache Kafka, Apache HBase, Apache Hive, HDFS, Slack
and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, image recognition and object detection with TensorFlow and
Apache MXNet.
Curate: Gain Insights

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved.
{
"imagefilename" : "/opt/demo/images/2018-04-
17_1127.jpg",
"yaw" : 100.0,
"host" : "sensehatmovidius",
"top3" : "n06874185 traffic light, traffic signal,
stoplight",
"top5" : "n03773504 missile",
"humidity" : 31.2,
"uuid" : "mxnet_uuid_json_20180417152727.json",
"ipaddress" : 192.168.1.104,
"top2" : "n04286575 spotlight, spot",
"top3pct" : "6.199999898672104",
"top2pct" : "10.199999809265137",
"cputemp2" : 56.92,
"z" : 1.0,
"diskfree" : "4152.5 MB",
"top1pct" : "13.79999965429306",
"currenttime" : "2018-04-17 15:27:37",
"label2" : "n04592741 wing",
"pitch" : 360.0,
"pressure" : 1026.2,
"roll" : 1.0,
"label1" : "n04286575 spotlight, spot",
"top5pct" : "4.30000014603138",
"label4" : "n06874185 traffic light, traffic signal, stoplight",
"y" : 0.0,
"label3" : "n04009552 projector",
"cputemp" : 58,
"top1" : "n02930766 cab, hack, taxi, taxicab",
"top4pct" : "5.000000074505806",
"tempf" : 75.81,
"memory" : 56.5,
"top4" : "n03345487 fire engine, fire truck",
"starttime" : "2018-04-17 15:27:25",
"runtime" : "12",
"label5" : "n09229709 bubble",
"temp" : 35.45,
"x" : 0.0
}
Example Data

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved.
IoT For Enterprise
Multiple devices, protocols, frameworks, languages, data types, sensors and networks
Protocols
• MQTT
• HTTPS / SSL (REST/JSON)
• OPC UA
• CoAP
• AMQP
• TCP/IP
• UDP
• Zigbee
• BLE
• LoRaWAN
• JSON
• XML
• CSV
• Raw Text
• Images (JPEG, PNG)
• Raw Data Streams
• Videos
• Log Files
• Geo Data
• Audio
Data Types Sensors
• Cameras
• Temperature/Humidity
• IR
• Proximity
• Motion Sensors
• GPS
• LIDAR
• Gas
• RFID
Protocols
• NVidia Jetson TX1
• Raspberry Pi
• Arduino
• TS-7800 V2
• ESP8266
• DragonBoard 410c
• BeagleBone Black
• NanoPi
• Google AIY
• Tinkerboard
Devices

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved.
IIoT Processing
Data Acquisition
Edge Processing
Real Time Stream Analytics
Rapid Application Development
IoT
ANALYTICS
CLOUD
Acquire Move
Route
&
Filter
Deliver Parse Enrich Analyze
Aggregate
Model

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved.
Run everywhere
Orchestration
Queueing
Simple Event Processing
Deep Learning Framework
Enabling Record Processing
Schema Management
Open Source Components
Edge Ingest
Data Buffering
Device Integration

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved.
HDF Data-In-Motion Platform – with HDF 3.1

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved.
Edge Intelligence with Apache MiNiFi
à Guaranteed delivery
à Data buffering
‒ Backpressure
‒ Pressure release
à Prioritized queuing
à Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
à Data provenance
à Recovery / recording a rolling log
of fine-grained history
à Designed for extension
Different from Apache NiFi
à Design and Deploy
à Warm re-deploys
Key Features

12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Java Agent C++ Agent
Multi Cloud HDP Clusters
HBase Druid
YARN
HDFS
Hive
Weather
Location
Sensors

13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
http://mxnet.incubator.apache.org/
• Cloud ready
• Experienced team (XGBoost)
• AWS, Microsoft, NVIDIA, Baidu, Intel backing
• Apache Incubator Project
• Run distributed on YARN
• In my early tests, faster than TensorFlow.
• Runs on Raspberry PI, NVidia Jetson TX1
and other constrained devices
• Great documentation
• Gluon
• Great Python Interaction
• Model Server Available
• ONNX Support
• Now in Version 1.1!
• Great Model Zoo
https://mxnet.incubator.apache.org/how_to/cloud.html
https://github.com/apache/incubator-mxnet/tree/1.1.0/example

14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Pre-Built Models
• CaffeNet
• SqueezeNet v1.1
• Inception v3
• Single Shot Detection (SSD)
• VGG19
• ResidualNet 152
• LSTM
http://mxnet.incubator.apache.org/model_zoo/index.html
https://github.com/dmlc/mxnet-model-gallery

15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Raspberry Pi Setup
pip install --upgrade pip
pip install scikit-image
git clone https://github.com/tspannhw/mxnet_rpi.git
sudo apt-get update -y
sudo apt-get install python-pip python-opencv python-scipy python-picamera -y
sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 liblapack* libblas* libopencv*
git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet --branch 1.1.0
cd incubator-mxnet
export USE_OPENCV = 0
make
pip install mxnet==1.1.0
https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html

16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Raspberry Pi Setup
curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11;
rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-
US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-
alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L
tar -xvzf inception-bn.tar.gz
cp Inception-BN-0126.params Inception-BN-0000.params

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MXNet Running on IoT Devices
https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html
https://github.com/tspannhw/mxnet_rpi
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-
running-apac.html

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
GPU Powered Nodes with NVidia Jetson TX1 Running Apache MXNet
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-apac.html

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved.
Using Apache MXNet on Devices with Sensors and Intel Movidius
https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on-the.html
https://community.hortonworks.com/articles/155475/powering-apache-minifi-flows-with-a-movidius-neura.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow with MiniFi
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved.
Human Integration
https://community.hortonworks.com/articles/121916/controlling-big-data-flows-with-gestures-minifi-ni.html
• Voice
• Touch
• Gestures
https://community.hortonworks.com/articles/167196/ingesting-data-from-the-matrix-creator-with-minifi.html
https://community.hortonworks.com/articles/155519/voice-controlled-data-flows-with-google-aiy-voice.html

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved.
Geolocation
https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html
https://community.hortonworks.com/articles/155606/iot-ingesting-gps-data-from-odroid-xu4-devices-wit.html
https://community.hortonworks.com/articles/177137/ingesting-flight-data-ads-b-usb-receiver-with-apac.html
https://community.hortonworks.com/articles/178196/integrating-lucene-geo-gazetteer-for-geo-parsing-w.html

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved.
Weather Integration
https://community.hortonworks.com/articles/101904/part-2-iot-augmenting-gps-data-with-weather.html

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved.
Image Analytics
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved.
Bluetooth Low Energy (BLE)
https://community.hortonworks.com/articles/108947/minifi-for-ble-bluetooth-low-energy-beacon-data-in.html
Beacons
IBeacon
NFC
https://community.hortonworks.com/articles/99861/ingesting-ibeacon-data-via-ble-to-mqtt-wifi-gatewa.html

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved.
Sensors via Python API
{"tempf": 75.14, "temp": 35.08,
"pitch": 1.0,
"diskfree": "1211.8 MB", "yaw": 55.0,
"cputemp": 52.08,
"ts": "2017-06-16 17:39:08",
"humidity": 41.5, "pressure": 0.0,
"memory": 23.0, "y": 0.0, "x": -1.0, "z": 1.0,
"roll": 1.0}
https://community.hortonworks.com/articles/108966/minifi-for-sensor-data-ingest-from-devices.html
https://community.hortonworks.com/articles/110475/ingesting-sensor-data-from-raspberry-pis-running-r.html
https://community.hortonworks.com/articles/155326/monitoring-energy-usage-utilizing-apache-nifi-pyth.html

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved.
Text Processing
https://community.hortonworks.com/articles/178510/integration-apache-opennlp-184-into-apache-nifi-15.html
• Dates
• Locations
• Names
• Money
• Organizations
https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach.html
Sentiment AnalysisNamed Entity Extraction
https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html
https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html
https://community.hortonworks.com/articles/81222/adding-stanford-corenlp-to-big-data-pipelines-apac.html
https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved.
MQTT
https://community.hortonworks.com/articles/167199/sending-messages-and-displaying-them-on-an-oled-sc.html
https://community.hortonworks.com/articles/183474/iot-using-minifi-java-agent-to-send-mqtt-messages.html
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html
https://community.hortonworks.com/articles/72420/ingesting
-remote-sensor-feeds-into-apache-phoenix.html

29. 29 © Hortonworks Inc. 2011–2018. All rights reserved.
Questions?

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved.
Contact
https://github.com/tspannhw/ApacheDeepLearning101
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
@PaasDev

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved.
Community Engagement
Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved
4,000+
Registered Users
10,000+
Answers
15,000+
Technical Assets
One Website!