SlideShare ist ein Scribd-Unternehmen logo
1 von 34
Downloaden Sie, um offline zu lesen
Apache Metron
Meetup & Code Lab
George Vetticaden
Principal Architect @ Hortonworks
Apache Metron Committer
James Sirota
Engineering Lead & Chief Data Scientist @ Hortonworks
Apache Metron Committer
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda
Metron
Page4
The Good Guys
Security
Practitioner
I have too many tools I need to learn
I don’t have a centralized view of my data
My tools are too expensive
I can’t find enough talent
I can’t keep relying on static rules
I need to discover bad stuff quicker
Most of my alerts are false positives
I have too many manual tasks
SOC
Manager
Threat landscape too dynamic
More assets/users to manage
Attack surface increases
Legacy techniques don’t work anymore
Metron will make it easier and faster to find
the real issues I need to act on
Metron is a more cost effective way for my team
to deal with the fast moving threat landscape
Page5
The Bad Guys
Advanced
Persistent
Threat
Script
Kiddie
My techniques are predictable and known
My attack vectors are also known
You are not the only person I’ve attacked
I brag about what I did or will do
I set off a large number of alerts
I fumble around a lot
I am very unique in a way I do things
I live on your network for about 300 days
I know what I am after and I look for it, slowly
Your rules will not detect me, I am too smart
I impersonate a legitimate user, but I don’t act like one
Metron can take everything that is known
about me and check for it in real time
Metron can model historical behavior of whoever I am
impersonating and flag me as I try to deviate
Page6
Problems With Existing Tools
Security
Information
Management
System
I am prohibitively expensive
I have vendor lock-in
I can’t deal with big data
I am not open
I am not extensible enough
Legacy
Point
Tools
I was built for 1995
I am super specialized
I don’t scale horizontally
I have a proprietary format
You need a PhD to operate me
Behavioral
Analytics
Tools
I am mostly vapor ware
I was built by a small startup
I was modeled after a data set from 1999
I spam you with false positives
Page7
Apache Metron Vision
“Apache Metron is a Security Data
Analytics Platform (SDAP). As a
next generation security analytics
framework, it is designed to
consume and monitor network
traffic and machine data within an
enterprise. Apache Metron is
extensible and is designed to work
at a massive scale. It is not a SIEM
but rather the next evolution of a
SIEM.”
Apache Metron provides the following capabilities:
 Extensible spouts and parsers for attaching Apache
Metron to monitor any telemetry source
 Extensible enrichment framework for any telemetry
stream
 Hadoop-backed storage for telemetry stream with a
customizable retention time
 Automated real-time index for telemetry streams
enabling real-time search
 Telemetry correlation and SQL query capability for data
stored in Hadoop backed by Hive
 ODBC/JDBC compatibility and integration with existing
analytics tools
Challenges that Apache Metron Solves
60%: Percent of breaches that
happened in minutes
8 months: Average time an
advanced security breach goes
unnoticed
$400 million in estimated
financial loss in 2015
70%-90%: Percentage of
malware in breach unique to
organization
2015 Verizon Data Breach Investigations Report
• Too expensive to keep data for enough time to
understand history
• Not enough of the right data to provide
context
• Too expensive to collect all the desired data to
understand context
• Not sure if can detect a targeted event.
• Too many events to review in timely manner
• Not enough staff to review events in a timely
manner
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda
Real-time Processing Engine
PCAP
NETFLOW
DPI
IDS
AV
EMAIL
FIREWALL
HOST LOGS
PARSE
NORMALIZE
TAG
VALIDATE
PROCESS
USER
ASSET
GEO
WHOIS
CONN
ENRICH
STIX
Flat Files
Aggregators
Model As A
Service
Cloud
Services
LABEL
PCAP
Store
ALERT
PERSIST
Alert
Security Data
Vault
Apache Metron Logical Architecture
Network
Tap
Custom Metron UI/Portals
Real-Time
Search
Interactive
Dashboards
Data
Modelling
Integration
Layer
PCAP
Replay
Security
Layer
Data & Integration Services
Apache Metron
Page11
Sensor
A
Sensor
B
Sensor
N
Topic A
Topic B
Topic (N)
Apache
Kafka
PCAP
PCAP
Probe
Physical Architecture
Normalizing
Topology A
Normalizing
Topology B
Normalizing
Topology N
Apache
Storm
Native Format
Native Format
Native Format
PCAP on HDFS Metron PCAP
Service
PCAP
Topology
Enrich
Normalized
Metron
Format Enrichment/
Threat Intel
Topology
Out to Index + HDFS
Page12
Topic A
Normalizing
Topology A
Sensor
A
Native Format
Apache
Kafka
Apache
Storm
Kafka
Spout
Parser
Kafka
Bolt
Enriched
Metron JSON
Parsing/Normalization Topology
Key Points:
• Each New Telemetry Data Source will have its own Parser Topology
• Two types of Parsers available: Grok and Java
Page13
2 Types of Parsers
Parser Type Description Telemetry Type
Grok • A grok is a collection of named regular expressions.
• Provides a declarative way to write new parsers
without any code
• A parser takes an input, which is usually a byte
array coming from the Kafka Spout, and turns it into
a Metron JSON Object.
• The Grok parser does this by utilizing the Grok
library inside of the Parser Kafka Bolt Adapter.
• Use this parser when
telemetry is simple to parse
or low in volume
Java • Java based approach to writing a custom parsers • Use this parser when
telemetry is complex to
parse or high volume
Page14
Metron JSON Object
• Numerous sensors log in different formats. The parser should normalize at least the
following subset of fields to the following Metron JSON naming conventions:
Page15
Enrich
ment
Bolt(a)
Enrich
ment
Bolt(n)
Threat
Intel
Joiner
Message
Splitter:
Enrichment
Enrich
ment
Joiner
Message
Splitter:
Threat Intel
Model
Bolt
(n)
Threat
Intel
Bolt
(n)
Metron
Enrichment
Loader
Framework
Metron Threat
Loader
Framework
Data
Store
Fast
Cach
e
Fast
Cach
e
Fast
Cach
e
Fast
Cach
e
Data
Store
Enrichment
Topology
Apache
Kafka
Enriched
Writer
Bolt
= Message Stream
Apache Storm
= Enrichment Stream
Enrichment Topology
Page16
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda
Page17
Personas
Page18
Metron’s Key Functional Themes
Platform
Work done to harden the platform for performance, scale, extensibility
and maintainability. This also includes capabilities around
provisioning, managing and monitoring the application.
Set of Data Sources that Metron provides capabilities to stream,
ingest and parse into the platform.
A set of Storm Topologies to perform various actions in real-time
including: normalization of telemetry data, enrichment, cross
reference with threat intel feeds, alerting, indexing, and persisting into
Historical stores
Data Collection
Data Processing
UI Set of portal, dashboard and user interfaces for the different
personas.
Page19
Target Personas and Themes for Apache Metron 0.1
T
e
c
h
P
r
e
v
i
e
w
1
-
I
n
t
r
o
Theme: Platform Theme: Data Collection
Theme: Data Processing Theme: UI
Security Platform
Engineer
Security Platform
Engineer
Security Platform
Engineer
SOC Investigator Security Platform
Engineer SOC Investigator
Forensic Investigator SOC Investigator
SOC Analyst SOC
Manager
Page20
• Fully automated vagrant install of Metron on a single VM
• Fully automated install of Metron on multi-node HDP cluster via Ansible scripts, Ambari
blueprints and APIs including:
• Multi-node Elastic Search Cluster
• Metron-UI Web Application
• Deployment of the Metron Storm Topology
• Deployment of telemetry sensors: PCAP, Bro, YAF(Netflow), Snort
• OpenSOC redesign (new topology structure, extensible enrichments, threat intel, data
loads, configs, ease of adding new topologies)
Platform
Data Collection
• Ingestion of the following data sources: PCAP via pycapa or C++ DPDK probe, Bro,
Netflow via YAF, Snort
• Parsers for the following data sources: PCAP, Bro, Netflow & Snort
Data Processing
• Support for the following enrichment services: Geo, WhoIs, Host
• Threat Intelligence Message enrichment - Enrich messages with fields that mat the
threat intelligence data in HBase
• Support for the following persistence services: HDFS, HBase and Elastic Search
• Indexing events and Alerts into Elastic Search cluster
• Support for Soltra(CIF) Threat Aggregator Services via STIX and Taxii Feed
• Ability to replay PCAP files for Testing
UI
• Metron Investigator UI to search across indexed events and alerts for SOC Analyst &
Investigators
• Histogram Panels for each of the data sources (YAF, Bro, Snort)
• Table Views for Alerts (YAF, Bro, Snort)
• Customize new panels with different data sources and different panel types.
Key Features of Apache Metron 0.1
Page21
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda
Page22
Why Metron? SOC Analyst Perspective
Looking through
alerts
25%
Collecting contextual
data
25%
Formulating a
Hypothesis
5%
Investigate
20%
Remediate
15%
Update Workflow
5%
Wrte Report
5%
ANALYST WORKFLOW • Alerts Relevancy Engine
• Smarter ML alerts
• Centralized Alerts Console
• Enriched with threat intel data
• Fully enriched messages
• Single pane of glass UI
• Centralized real-time search
• All logs in one place
• Granular access to PCAP
• Replay old PCAP against new signatures
• Tag behavior for modelling by data scientists
• Raw messages used as evidentiary store
• Mine investigation history
• Asset inventory as an enrichment
• User identity as an enrichment
• Workflow engine
• Ticket clustering
Everything you need to know in one place
Page23
Why Metron? Data Scientist Perspective
Formulating a
Hypothesis
5%
Finding Data
20%
Cleaning Data
20%
Munging Data
20%
Visualizing Data
20%
Modelling Data
10%
Validating Model
5%
DATA SCIENCE WORKFLOW
• All my data is in the same place
• Data exposed through a variety of APIs
• Standard Access Control Policies
• Quickly see what I have
• Metron normalizes objects
• Partial schema validation on ingest
• Tagging on ingest
• Automatic data enrichment
• Automatic application of class labels
• Common Metron Objects
• Massively parallel computation framework
• Reusable Zeppelin Dashboards
• Real-time search + UI
• Integration with Python/R
• Integration with analytics tools
Reducing time from hypothesis to model
Page24
Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda
Page25
Use Case Setup
• Scenario
• Customer Foo has installed Metron TP1 and they are using the out of the box data sources (PCAP,
YAF/Netflow, Snort and Bro). They love Metron!
• But now they want to add new data source the the platform: squid proxy logs.
• Customer Foo’s requirements are the following
1. Need to ingest the proxy events from Squid logs in real-time
2. The proxy logs has to be parsed into a standardized JSON structure that Metron can understand
3. In real-time, the squid proxy event needs to be enriched with domain/whois information (domain,
cert, country, company)
4. In real-time, the domain of the proxy event must be checked against for threat intel feeds
5. If there is a threat intel hit, an alert needs to be raised
6. The end user must be able to see the new telemetry events and the alerts from the new data
source
Page26
Squid & its Telemetry Event
• What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and
improves response times by caching and reusing frequently-requested web pages
• What does a Squid Access Log look like?
• When you make an outbound http connection to https://www.cnn.com, the following entry gets added to a file
called access.log:
Unix Epoch Time
IP of host where connection was made.
The domain name
of the outbound connection
1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
Page27
What Metron does to the Squid Telemetry Event in Real-time
Convert from
Unix Epoch to
Timestamp
Use Metron’s asset enrichment to enrich
that IP (hostname, type of device)
Use Metron’s WhoIs enrichment
To look up domain name information (e.g:
Use the Metron’s Threat Intel Services
to cross-reference the IP with threat intel
feed to see if there is a hit
1461576382.642 161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
Index the event into Elastic
and persist into HDFS
(Security Data Vault)
Page28
Real-time Processing Engine
Squid Logs
PARSE
NORMALIZE
TAG
VALIDATE
PROCESS
USER
ASSET
GEO
WHOIS
CONN
ENRICH
STIX
Flat Files
Aggregators
Model As A
Service
Cloud
Services
LABEL
PCAP
Store
ALERT
PERSIST
Alert
Security Data
Vault
Real-Time
Search
Interactive
Dashboards
Data
Modelling
Integration
Layer
PCAP
Replay
Security
Layer
Data & Integration Services
Tracing the Squid Event across the Platform
Custom Metron UI/Portals
Page29
Step 1: Telemetry Ingest (Tracing an Event)
1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
Page30
Step 2 – Process/Parse (Tracing an Event)
Page31
Step 3 – Enrich (Tracing an Event)
Page32
Enriching Data Architecture
Metron Enrichment
Store
(HBase/)
Enrichment Loader
Framework
Bulk Load Polling
Enrichment Source
Storm
Bolt
Cache
Metron Streaming Messages Enriched Metron Streaming Messages
Page33
Step 4 – Label/Threat Intel (Tracing an Event)
Threat Intel Store
(HBase)
Threat Intel Loader
Framework
Bulk Load Polling
Storm
Bolt
Cache
Metron Streaming Messages
(Enriched)
Enriched Metron Streaming Messages
(Enriched) + Threat Intel Hits
Threat Intel Feed Source
(Optional) Threat
Intel Aggregator
Page34
High level Steps – How to Add the New Telemetry
1. Create new Kafka topic for the new telemetry source called “squid”
2. Create and validate a grok statement file that parses the squid event log into a format that Metron can understand
3. Store that grok statement in HDFS
4. Create a new flux configuration for the new Squid parser Storm Topology.
5. Update Zookeeper with configuration to mark what fields in the telemetry to enrich and what fields to cross-
reference with threat intel feeds.
6. Move the flux configuration to the host where you will deploy the topology.
7. Deploy the new squid Storm parser topology using the new flux configuration
8. Load WhoIs enrichment data and configure enrichment mapping
9. Load Threat Intel data and configure threat intel matching mapping
10. Use Apache Nifi to capture the squid events and push them into Metron
11. Create a new Panel in Kibana and see the telemetry events
Key Points
Easy Extensibility – The ability to add new data source without writing any code and in an easy mann
Repeatable Pattern - The following represents a repeatable pattern that you can apply to most data s

Weitere ähnliche Inhalte

Was ist angesagt?

Apply big data and data lake for processing security data collections
Apply big data and  data lake for processing security data collectionsApply big data and  data lake for processing security data collections
Apply big data and data lake for processing security data collectionsGregory Shlyuger
 
A streaming architecture for Cyber Security - Apache Metron
A streaming architecture for Cyber Security - Apache MetronA streaming architecture for Cyber Security - Apache Metron
A streaming architecture for Cyber Security - Apache MetronSimon Elliston Ball
 
Open Security Operations Center - OpenSOC
Open Security Operations Center - OpenSOCOpen Security Operations Center - OpenSOC
Open Security Operations Center - OpenSOCSheetal Dolas
 
Treat Detection using Hadoop
Treat Detection using HadoopTreat Detection using Hadoop
Treat Detection using HadoopDataWorks Summit
 
Apache Metron in the Real World
Apache Metron in the Real WorldApache Metron in the Real World
Apache Metron in the Real WorldDave Russell
 
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...DataWorks Summit
 
Performing Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopPerforming Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopDataWorks Summit
 
Fighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache SpotFighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache Spotmarkgrover
 
Security From The Big Data and Analytics Perspective
Security From The Big Data and Analytics PerspectiveSecurity From The Big Data and Analytics Perspective
Security From The Big Data and Analytics PerspectiveAll Things Open
 
Designing and Implementing your IOT Solutions with Open Source
Designing and Implementing your IOT Solutions with Open SourceDesigning and Implementing your IOT Solutions with Open Source
Designing and Implementing your IOT Solutions with Open SourceDataWorks Summit/Hadoop Summit
 
A Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOpsA Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOpsBigPanda
 
Hadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionHadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionMapR Technologies
 
Apache Eagle: Architecture Evolvement and New Features
Apache Eagle: Architecture Evolvement and New FeaturesApache Eagle: Architecture Evolvement and New Features
Apache Eagle: Architecture Evolvement and New FeaturesHao Chen
 
QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoDamien Dallimore
 
Sumo Logic: Optimizing Scheduled Searches
Sumo Logic: Optimizing Scheduled SearchesSumo Logic: Optimizing Scheduled Searches
Sumo Logic: Optimizing Scheduled SearchesSumo Logic
 
mcubed london - data science at the edge
mcubed london - data science at the edgemcubed london - data science at the edge
mcubed london - data science at the edgeSimon Elliston Ball
 
Data Onboarding Breakout Session
Data Onboarding Breakout SessionData Onboarding Breakout Session
Data Onboarding Breakout SessionSplunk
 
Burning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle:  Security Analytics in ActionBurning Down the Haystack to Find the Needle:  Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in ActionJosh Sokol
 

Was ist angesagt? (20)

Apply big data and data lake for processing security data collections
Apply big data and  data lake for processing security data collectionsApply big data and  data lake for processing security data collections
Apply big data and data lake for processing security data collections
 
Apache Spot
Apache SpotApache Spot
Apache Spot
 
A streaming architecture for Cyber Security - Apache Metron
A streaming architecture for Cyber Security - Apache MetronA streaming architecture for Cyber Security - Apache Metron
A streaming architecture for Cyber Security - Apache Metron
 
Solving Cyber at Scale
Solving Cyber at ScaleSolving Cyber at Scale
Solving Cyber at Scale
 
Open Security Operations Center - OpenSOC
Open Security Operations Center - OpenSOCOpen Security Operations Center - OpenSOC
Open Security Operations Center - OpenSOC
 
Treat Detection using Hadoop
Treat Detection using HadoopTreat Detection using Hadoop
Treat Detection using Hadoop
 
Apache Metron in the Real World
Apache Metron in the Real WorldApache Metron in the Real World
Apache Metron in the Real World
 
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
Bringing it All Together: Apache Metron (Incubating) as a Case Study of a Mod...
 
Performing Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopPerforming Network & Security Analytics with Hadoop
Performing Network & Security Analytics with Hadoop
 
Fighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache SpotFighting cybersecurity threats with Apache Spot
Fighting cybersecurity threats with Apache Spot
 
Security From The Big Data and Analytics Perspective
Security From The Big Data and Analytics PerspectiveSecurity From The Big Data and Analytics Perspective
Security From The Big Data and Analytics Perspective
 
Designing and Implementing your IOT Solutions with Open Source
Designing and Implementing your IOT Solutions with Open SourceDesigning and Implementing your IOT Solutions with Open Source
Designing and Implementing your IOT Solutions with Open Source
 
A Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOpsA Practical Guide to Anomaly Detection for DevOps
A Practical Guide to Anomaly Detection for DevOps
 
Hadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionHadoop / Spark on Malware Expression
Hadoop / Spark on Malware Expression
 
Apache Eagle: Architecture Evolvement and New Features
Apache Eagle: Architecture Evolvement and New FeaturesApache Eagle: Architecture Evolvement and New Features
Apache Eagle: Architecture Evolvement and New Features
 
QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
 
Sumo Logic: Optimizing Scheduled Searches
Sumo Logic: Optimizing Scheduled SearchesSumo Logic: Optimizing Scheduled Searches
Sumo Logic: Optimizing Scheduled Searches
 
mcubed london - data science at the edge
mcubed london - data science at the edgemcubed london - data science at the edge
mcubed london - data science at the edge
 
Data Onboarding Breakout Session
Data Onboarding Breakout SessionData Onboarding Breakout Session
Data Onboarding Breakout Session
 
Burning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle:  Security Analytics in ActionBurning Down the Haystack to Find the Needle:  Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in Action
 

Andere mochten auch

2016 Cybersecurity Analytics State of the Union
2016 Cybersecurity Analytics State of the Union2016 Cybersecurity Analytics State of the Union
2016 Cybersecurity Analytics State of the UnionCloudera, Inc.
 
War on stealth cyber attacks phishing docusign apache metron
War on stealth cyber attacks phishing docusign apache metronWar on stealth cyber attacks phishing docusign apache metron
War on stealth cyber attacks phishing docusign apache metrongvetticaden
 
Intro to Spark with Zeppelin
Intro to Spark with ZeppelinIntro to Spark with Zeppelin
Intro to Spark with ZeppelinHortonworks
 
Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4Hortonworks
 
Clairvoyant Squirrel: Large Scale Malicious Domain Classification
Clairvoyant Squirrel: Large Scale Malicious Domain ClassificationClairvoyant Squirrel: Large Scale Malicious Domain Classification
Clairvoyant Squirrel: Large Scale Malicious Domain ClassificationJason Trost
 
Royal Statistical Society: Does the provision of workplace pensions affect ot...
Royal Statistical Society: Does the provision of workplace pensions affect ot...Royal Statistical Society: Does the provision of workplace pensions affect ot...
Royal Statistical Society: Does the provision of workplace pensions affect ot...Bernardo F. Nunes
 
Machine learning in Web proxy caching
Machine learning in Web proxy cachingMachine learning in Web proxy caching
Machine learning in Web proxy cachingNimishan Sivaraj
 
유전 알고리즘으로 패킷 필터링 규칙 만들기
유전 알고리즘으로 패킷 필터링 규칙 만들기유전 알고리즘으로 패킷 필터링 규칙 만들기
유전 알고리즘으로 패킷 필터링 규칙 만들기Hyunwoo Kim
 
Apache NiFi 1.0 in Nutshell
Apache NiFi 1.0 in NutshellApache NiFi 1.0 in Nutshell
Apache NiFi 1.0 in NutshellKoji Kawamura
 
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...Cloudera, Inc.
 
Blue team reboot - HackFest
Blue team reboot - HackFest Blue team reboot - HackFest
Blue team reboot - HackFest Haydn Johnson
 
R-CISC Summit 2016 Borderless Threat Intelligence
R-CISC Summit 2016 Borderless Threat IntelligenceR-CISC Summit 2016 Borderless Threat Intelligence
R-CISC Summit 2016 Borderless Threat IntelligenceJason Trost
 
Hortonworks Data In Motion Webinar Series Pt. 2
Hortonworks Data In Motion Webinar Series Pt. 2Hortonworks Data In Motion Webinar Series Pt. 2
Hortonworks Data In Motion Webinar Series Pt. 2Hortonworks
 
Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1Hortonworks
 
Seeing Wetiko: Tracking the Spread of Memes on Social Media
Seeing Wetiko: Tracking the Spread of Memes on Social MediaSeeing Wetiko: Tracking the Spread of Memes on Social Media
Seeing Wetiko: Tracking the Spread of Memes on Social MediaJoe Brewer
 
Porter 5 forces model
Porter 5 forces modelPorter 5 forces model
Porter 5 forces modelRaj vardhan
 

Andere mochten auch (18)

Tracing your security telemetry with Apache Metron
Tracing your security telemetry with Apache MetronTracing your security telemetry with Apache Metron
Tracing your security telemetry with Apache Metron
 
2016 Cybersecurity Analytics State of the Union
2016 Cybersecurity Analytics State of the Union2016 Cybersecurity Analytics State of the Union
2016 Cybersecurity Analytics State of the Union
 
War on stealth cyber attacks phishing docusign apache metron
War on stealth cyber attacks phishing docusign apache metronWar on stealth cyber attacks phishing docusign apache metron
War on stealth cyber attacks phishing docusign apache metron
 
Intro to Spark with Zeppelin
Intro to Spark with ZeppelinIntro to Spark with Zeppelin
Intro to Spark with Zeppelin
 
Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4Hortonworks Data In Motion Series Part 4
Hortonworks Data In Motion Series Part 4
 
Clairvoyant Squirrel: Large Scale Malicious Domain Classification
Clairvoyant Squirrel: Large Scale Malicious Domain ClassificationClairvoyant Squirrel: Large Scale Malicious Domain Classification
Clairvoyant Squirrel: Large Scale Malicious Domain Classification
 
Royal Statistical Society: Does the provision of workplace pensions affect ot...
Royal Statistical Society: Does the provision of workplace pensions affect ot...Royal Statistical Society: Does the provision of workplace pensions affect ot...
Royal Statistical Society: Does the provision of workplace pensions affect ot...
 
Machine learning in Web proxy caching
Machine learning in Web proxy cachingMachine learning in Web proxy caching
Machine learning in Web proxy caching
 
유전 알고리즘으로 패킷 필터링 규칙 만들기
유전 알고리즘으로 패킷 필터링 규칙 만들기유전 알고리즘으로 패킷 필터링 규칙 만들기
유전 알고리즘으로 패킷 필터링 규칙 만들기
 
Apache NiFi 1.0 in Nutshell
Apache NiFi 1.0 in NutshellApache NiFi 1.0 in Nutshell
Apache NiFi 1.0 in Nutshell
 
Porter 5 Forces
Porter 5 ForcesPorter 5 Forces
Porter 5 Forces
 
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...
Delivering User Behavior Analytics at Apache Hadoop Scale : A new perspective...
 
Blue team reboot - HackFest
Blue team reboot - HackFest Blue team reboot - HackFest
Blue team reboot - HackFest
 
R-CISC Summit 2016 Borderless Threat Intelligence
R-CISC Summit 2016 Borderless Threat IntelligenceR-CISC Summit 2016 Borderless Threat Intelligence
R-CISC Summit 2016 Borderless Threat Intelligence
 
Hortonworks Data In Motion Webinar Series Pt. 2
Hortonworks Data In Motion Webinar Series Pt. 2Hortonworks Data In Motion Webinar Series Pt. 2
Hortonworks Data In Motion Webinar Series Pt. 2
 
Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1Hortonworks Data in Motion Webinar Series - Part 1
Hortonworks Data in Motion Webinar Series - Part 1
 
Seeing Wetiko: Tracking the Spread of Memes on Social Media
Seeing Wetiko: Tracking the Spread of Memes on Social MediaSeeing Wetiko: Tracking the Spread of Memes on Social Media
Seeing Wetiko: Tracking the Spread of Memes on Social Media
 
Porter 5 forces model
Porter 5 forces modelPorter 5 forces model
Porter 5 forces model
 

Ähnlich wie Apache Metron Meetup May 4, 2016 - Big data cybersecurity

Metasploit (Module-1) - Getting Started With Metasploit
Metasploit (Module-1) - Getting Started With MetasploitMetasploit (Module-1) - Getting Started With Metasploit
Metasploit (Module-1) - Getting Started With MetasploitAnurag Srivastava
 
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDN
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDNOliver Schuermann - Integrated Software in Networking - the Mystery of SDN
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDNcentralohioissa
 
Nt2580 Unit 7 Chapter 12
Nt2580 Unit 7 Chapter 12Nt2580 Unit 7 Chapter 12
Nt2580 Unit 7 Chapter 12Laura Arrigo
 
Malware Classification and Analysis
Malware Classification and AnalysisMalware Classification and Analysis
Malware Classification and AnalysisPrashant Chopra
 
Novetta Cyber Analytics
Novetta Cyber AnalyticsNovetta Cyber Analytics
Novetta Cyber AnalyticsNovetta
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxGrace Jansen
 
CyberCrime in the Cloud and How to defend Yourself
CyberCrime in the Cloud and How to defend Yourself CyberCrime in the Cloud and How to defend Yourself
CyberCrime in the Cloud and How to defend Yourself Alert Logic
 
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...CloudVillage
 
Vulnerability Penetration Test
Vulnerability Penetration TestVulnerability Penetration Test
Vulnerability Penetration TestTanya Williams
 
Key Open Standards for inter-operable IoT systems
Key Open Standards for inter-operable IoT systemsKey Open Standards for inter-operable IoT systems
Key Open Standards for inter-operable IoT systemsPratul Sharma
 
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2securityxploded
 
Building a Real-Time Security Application Using Log Data and Machine Learning...
Building a Real-Time Security Application Using Log Data and Machine Learning...Building a Real-Time Security Application Using Log Data and Machine Learning...
Building a Real-Time Security Application Using Log Data and Machine Learning...Sri Ambati
 
The Internet Of Things Can Be Developed And Refined
The Internet Of Things Can Be Developed And RefinedThe Internet Of Things Can Be Developed And Refined
The Internet Of Things Can Be Developed And RefinedStephanie Roberts
 
20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security BrokersRobin Vermeirsch
 
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptx
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptxINTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptx
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptxSuhailShaik16
 

Ähnlich wie Apache Metron Meetup May 4, 2016 - Big data cybersecurity (20)

Metasploit (Module-1) - Getting Started With Metasploit
Metasploit (Module-1) - Getting Started With MetasploitMetasploit (Module-1) - Getting Started With Metasploit
Metasploit (Module-1) - Getting Started With Metasploit
 
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDN
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDNOliver Schuermann - Integrated Software in Networking - the Mystery of SDN
Oliver Schuermann - Integrated Software in Networking - the Mystery of SDN
 
Nt2580 Unit 7 Chapter 12
Nt2580 Unit 7 Chapter 12Nt2580 Unit 7 Chapter 12
Nt2580 Unit 7 Chapter 12
 
PET Scan Analysis
PET Scan AnalysisPET Scan Analysis
PET Scan Analysis
 
Malware Classification and Analysis
Malware Classification and AnalysisMalware Classification and Analysis
Malware Classification and Analysis
 
Novetta Cyber Analytics
Novetta Cyber AnalyticsNovetta Cyber Analytics
Novetta Cyber Analytics
 
ThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptxThroughTheLookingGlass_EffectiveObservability.pptx
ThroughTheLookingGlass_EffectiveObservability.pptx
 
CyberCrime in the Cloud and How to defend Yourself
CyberCrime in the Cloud and How to defend Yourself CyberCrime in the Cloud and How to defend Yourself
CyberCrime in the Cloud and How to defend Yourself
 
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...
ATT&CKing the Sentinel – deploying a threat hunting capability on Azure Senti...
 
Iot Devices Essay
Iot Devices EssayIot Devices Essay
Iot Devices Essay
 
Vulnerability Penetration Test
Vulnerability Penetration TestVulnerability Penetration Test
Vulnerability Penetration Test
 
Key Open Standards for inter-operable IoT systems
Key Open Standards for inter-operable IoT systemsKey Open Standards for inter-operable IoT systems
Key Open Standards for inter-operable IoT systems
 
BLOCKHUNTER.pptx
BLOCKHUNTER.pptxBLOCKHUNTER.pptx
BLOCKHUNTER.pptx
 
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2
Advanced Malware Analysis Training Session 3 - Botnet Analysis Part 2
 
Building a Real-Time Security Application Using Log Data and Machine Learning...
Building a Real-Time Security Application Using Log Data and Machine Learning...Building a Real-Time Security Application Using Log Data and Machine Learning...
Building a Real-Time Security Application Using Log Data and Machine Learning...
 
S4x20 Forescout Presentation
S4x20 Forescout Presentation S4x20 Forescout Presentation
S4x20 Forescout Presentation
 
The Internet Of Things Can Be Developed And Refined
The Internet Of Things Can Be Developed And RefinedThe Internet Of Things Can Be Developed And Refined
The Internet Of Things Can Be Developed And Refined
 
20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers
 
SOC-as-a-Service - comSpark 2019
SOC-as-a-Service - comSpark 2019SOC-as-a-Service - comSpark 2019
SOC-as-a-Service - comSpark 2019
 
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptx
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptxINTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptx
INTERNSHIPREVIEW-ISHAQ (1) [Recovered].pptx
 

Mehr von Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyHortonworks
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakHortonworks
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsHortonworks
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysHortonworks
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's NewHortonworks
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerHortonworks
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsHortonworks
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeHortonworks
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidHortonworks
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleHortonworks
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATAHortonworks
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Hortonworks
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseHortonworks
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseHortonworks
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationHortonworks
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementHortonworks
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHortonworks
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCHortonworks
 

Mehr von Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's New
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
 

Kürzlich hochgeladen

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Kürzlich hochgeladen (20)

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Apache Metron Meetup May 4, 2016 - Big data cybersecurity

  • 1. Apache Metron Meetup & Code Lab George Vetticaden Principal Architect @ Hortonworks Apache Metron Committer James Sirota Engineering Lead & Chief Data Scientist @ Hortonworks Apache Metron Committer
  • 2. Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
  • 4. Page4 The Good Guys Security Practitioner I have too many tools I need to learn I don’t have a centralized view of my data My tools are too expensive I can’t find enough talent I can’t keep relying on static rules I need to discover bad stuff quicker Most of my alerts are false positives I have too many manual tasks SOC Manager Threat landscape too dynamic More assets/users to manage Attack surface increases Legacy techniques don’t work anymore Metron will make it easier and faster to find the real issues I need to act on Metron is a more cost effective way for my team to deal with the fast moving threat landscape
  • 5. Page5 The Bad Guys Advanced Persistent Threat Script Kiddie My techniques are predictable and known My attack vectors are also known You are not the only person I’ve attacked I brag about what I did or will do I set off a large number of alerts I fumble around a lot I am very unique in a way I do things I live on your network for about 300 days I know what I am after and I look for it, slowly Your rules will not detect me, I am too smart I impersonate a legitimate user, but I don’t act like one Metron can take everything that is known about me and check for it in real time Metron can model historical behavior of whoever I am impersonating and flag me as I try to deviate
  • 6. Page6 Problems With Existing Tools Security Information Management System I am prohibitively expensive I have vendor lock-in I can’t deal with big data I am not open I am not extensible enough Legacy Point Tools I was built for 1995 I am super specialized I don’t scale horizontally I have a proprietary format You need a PhD to operate me Behavioral Analytics Tools I am mostly vapor ware I was built by a small startup I was modeled after a data set from 1999 I spam you with false positives
  • 7. Page7 Apache Metron Vision “Apache Metron is a Security Data Analytics Platform (SDAP). As a next generation security analytics framework, it is designed to consume and monitor network traffic and machine data within an enterprise. Apache Metron is extensible and is designed to work at a massive scale. It is not a SIEM but rather the next evolution of a SIEM.” Apache Metron provides the following capabilities:  Extensible spouts and parsers for attaching Apache Metron to monitor any telemetry source  Extensible enrichment framework for any telemetry stream  Hadoop-backed storage for telemetry stream with a customizable retention time  Automated real-time index for telemetry streams enabling real-time search  Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive  ODBC/JDBC compatibility and integration with existing analytics tools
  • 8. Challenges that Apache Metron Solves 60%: Percent of breaches that happened in minutes 8 months: Average time an advanced security breach goes unnoticed $400 million in estimated financial loss in 2015 70%-90%: Percentage of malware in breach unique to organization 2015 Verizon Data Breach Investigations Report • Too expensive to keep data for enough time to understand history • Not enough of the right data to provide context • Too expensive to collect all the desired data to understand context • Not sure if can detect a targeted event. • Too many events to review in timely manner • Not enough staff to review events in a timely manner
  • 9. Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
  • 10. Real-time Processing Engine PCAP NETFLOW DPI IDS AV EMAIL FIREWALL HOST LOGS PARSE NORMALIZE TAG VALIDATE PROCESS USER ASSET GEO WHOIS CONN ENRICH STIX Flat Files Aggregators Model As A Service Cloud Services LABEL PCAP Store ALERT PERSIST Alert Security Data Vault Apache Metron Logical Architecture Network Tap Custom Metron UI/Portals Real-Time Search Interactive Dashboards Data Modelling Integration Layer PCAP Replay Security Layer Data & Integration Services Apache Metron
  • 11. Page11 Sensor A Sensor B Sensor N Topic A Topic B Topic (N) Apache Kafka PCAP PCAP Probe Physical Architecture Normalizing Topology A Normalizing Topology B Normalizing Topology N Apache Storm Native Format Native Format Native Format PCAP on HDFS Metron PCAP Service PCAP Topology Enrich Normalized Metron Format Enrichment/ Threat Intel Topology Out to Index + HDFS
  • 12. Page12 Topic A Normalizing Topology A Sensor A Native Format Apache Kafka Apache Storm Kafka Spout Parser Kafka Bolt Enriched Metron JSON Parsing/Normalization Topology Key Points: • Each New Telemetry Data Source will have its own Parser Topology • Two types of Parsers available: Grok and Java
  • 13. Page13 2 Types of Parsers Parser Type Description Telemetry Type Grok • A grok is a collection of named regular expressions. • Provides a declarative way to write new parsers without any code • A parser takes an input, which is usually a byte array coming from the Kafka Spout, and turns it into a Metron JSON Object. • The Grok parser does this by utilizing the Grok library inside of the Parser Kafka Bolt Adapter. • Use this parser when telemetry is simple to parse or low in volume Java • Java based approach to writing a custom parsers • Use this parser when telemetry is complex to parse or high volume
  • 14. Page14 Metron JSON Object • Numerous sensors log in different formats. The parser should normalize at least the following subset of fields to the following Metron JSON naming conventions:
  • 16. Page16 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
  • 18. Page18 Metron’s Key Functional Themes Platform Work done to harden the platform for performance, scale, extensibility and maintainability. This also includes capabilities around provisioning, managing and monitoring the application. Set of Data Sources that Metron provides capabilities to stream, ingest and parse into the platform. A set of Storm Topologies to perform various actions in real-time including: normalization of telemetry data, enrichment, cross reference with threat intel feeds, alerting, indexing, and persisting into Historical stores Data Collection Data Processing UI Set of portal, dashboard and user interfaces for the different personas.
  • 19. Page19 Target Personas and Themes for Apache Metron 0.1 T e c h P r e v i e w 1 - I n t r o Theme: Platform Theme: Data Collection Theme: Data Processing Theme: UI Security Platform Engineer Security Platform Engineer Security Platform Engineer SOC Investigator Security Platform Engineer SOC Investigator Forensic Investigator SOC Investigator SOC Analyst SOC Manager
  • 20. Page20 • Fully automated vagrant install of Metron on a single VM • Fully automated install of Metron on multi-node HDP cluster via Ansible scripts, Ambari blueprints and APIs including: • Multi-node Elastic Search Cluster • Metron-UI Web Application • Deployment of the Metron Storm Topology • Deployment of telemetry sensors: PCAP, Bro, YAF(Netflow), Snort • OpenSOC redesign (new topology structure, extensible enrichments, threat intel, data loads, configs, ease of adding new topologies) Platform Data Collection • Ingestion of the following data sources: PCAP via pycapa or C++ DPDK probe, Bro, Netflow via YAF, Snort • Parsers for the following data sources: PCAP, Bro, Netflow & Snort Data Processing • Support for the following enrichment services: Geo, WhoIs, Host • Threat Intelligence Message enrichment - Enrich messages with fields that mat the threat intelligence data in HBase • Support for the following persistence services: HDFS, HBase and Elastic Search • Indexing events and Alerts into Elastic Search cluster • Support for Soltra(CIF) Threat Aggregator Services via STIX and Taxii Feed • Ability to replay PCAP files for Testing UI • Metron Investigator UI to search across indexed events and alerts for SOC Analyst & Investigators • Histogram Panels for each of the data sources (YAF, Bro, Snort) • Table Views for Alerts (YAF, Bro, Snort) • Customize new panels with different data sources and different panel types. Key Features of Apache Metron 0.1
  • 21. Page21 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
  • 22. Page22 Why Metron? SOC Analyst Perspective Looking through alerts 25% Collecting contextual data 25% Formulating a Hypothesis 5% Investigate 20% Remediate 15% Update Workflow 5% Wrte Report 5% ANALYST WORKFLOW • Alerts Relevancy Engine • Smarter ML alerts • Centralized Alerts Console • Enriched with threat intel data • Fully enriched messages • Single pane of glass UI • Centralized real-time search • All logs in one place • Granular access to PCAP • Replay old PCAP against new signatures • Tag behavior for modelling by data scientists • Raw messages used as evidentiary store • Mine investigation history • Asset inventory as an enrichment • User identity as an enrichment • Workflow engine • Ticket clustering Everything you need to know in one place
  • 23. Page23 Why Metron? Data Scientist Perspective Formulating a Hypothesis 5% Finding Data 20% Cleaning Data 20% Munging Data 20% Visualizing Data 20% Modelling Data 10% Validating Model 5% DATA SCIENCE WORKFLOW • All my data is in the same place • Data exposed through a variety of APIs • Standard Access Control Policies • Quickly see what I have • Metron normalizes objects • Partial schema validation on ingest • Tagging on ingest • Automatic data enrichment • Automatic application of class labels • Common Metron Objects • Massively parallel computation framework • Reusable Zeppelin Dashboards • Real-time search + UI • Integration with Python/R • Integration with analytics tools Reducing time from hypothesis to model
  • 24. Page24 Part 1 – Overview of Apache Metron • Challenges with Today’s Security Tools to Combat Cyber Attacks • Introduction to Apache Metron • Metron Architecture • Personas and Core Themes • Why Apache Metron? Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron • Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform • Get your Metron vagrant VM started • Use Case 1: Adding a net new telemetry data source to Metron • Use Case 2: Enriching Telemetry Data • Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds • Use Case 4: Setting up your IDE and writing Tests Agenda
  • 25. Page25 Use Case Setup • Scenario • Customer Foo has installed Metron TP1 and they are using the out of the box data sources (PCAP, YAF/Netflow, Snort and Bro). They love Metron! • But now they want to add new data source the the platform: squid proxy logs. • Customer Foo’s requirements are the following 1. Need to ingest the proxy events from Squid logs in real-time 2. The proxy logs has to be parsed into a standardized JSON structure that Metron can understand 3. In real-time, the squid proxy event needs to be enriched with domain/whois information (domain, cert, country, company) 4. In real-time, the domain of the proxy event must be checked against for threat intel feeds 5. If there is a threat intel hit, an alert needs to be raised 6. The end user must be able to see the new telemetry events and the alerts from the new data source
  • 26. Page26 Squid & its Telemetry Event • What is Squid? • Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages • What does a Squid Access Log look like? • When you make an outbound http connection to https://www.cnn.com, the following entry gets added to a file called access.log: Unix Epoch Time IP of host where connection was made. The domain name of the outbound connection 1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
  • 27. Page27 What Metron does to the Squid Telemetry Event in Real-time Convert from Unix Epoch to Timestamp Use Metron’s asset enrichment to enrich that IP (hostname, type of device) Use Metron’s WhoIs enrichment To look up domain name information (e.g: Use the Metron’s Threat Intel Services to cross-reference the IP with threat intel feed to see if there is a hit 1461576382.642 161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html Index the event into Elastic and persist into HDFS (Security Data Vault)
  • 28. Page28 Real-time Processing Engine Squid Logs PARSE NORMALIZE TAG VALIDATE PROCESS USER ASSET GEO WHOIS CONN ENRICH STIX Flat Files Aggregators Model As A Service Cloud Services LABEL PCAP Store ALERT PERSIST Alert Security Data Vault Real-Time Search Interactive Dashboards Data Modelling Integration Layer PCAP Replay Security Layer Data & Integration Services Tracing the Squid Event across the Platform Custom Metron UI/Portals
  • 29. Page29 Step 1: Telemetry Ingest (Tracing an Event) 1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
  • 30. Page30 Step 2 – Process/Parse (Tracing an Event)
  • 31. Page31 Step 3 – Enrich (Tracing an Event)
  • 32. Page32 Enriching Data Architecture Metron Enrichment Store (HBase/) Enrichment Loader Framework Bulk Load Polling Enrichment Source Storm Bolt Cache Metron Streaming Messages Enriched Metron Streaming Messages
  • 33. Page33 Step 4 – Label/Threat Intel (Tracing an Event) Threat Intel Store (HBase) Threat Intel Loader Framework Bulk Load Polling Storm Bolt Cache Metron Streaming Messages (Enriched) Enriched Metron Streaming Messages (Enriched) + Threat Intel Hits Threat Intel Feed Source (Optional) Threat Intel Aggregator
  • 34. Page34 High level Steps – How to Add the New Telemetry 1. Create new Kafka topic for the new telemetry source called “squid” 2. Create and validate a grok statement file that parses the squid event log into a format that Metron can understand 3. Store that grok statement in HDFS 4. Create a new flux configuration for the new Squid parser Storm Topology. 5. Update Zookeeper with configuration to mark what fields in the telemetry to enrich and what fields to cross- reference with threat intel feeds. 6. Move the flux configuration to the host where you will deploy the topology. 7. Deploy the new squid Storm parser topology using the new flux configuration 8. Load WhoIs enrichment data and configure enrichment mapping 9. Load Threat Intel data and configure threat intel matching mapping 10. Use Apache Nifi to capture the squid events and push them into Metron 11. Create a new Panel in Kibana and see the telemetry events Key Points Easy Extensibility – The ability to add new data source without writing any code and in an easy mann Repeatable Pattern - The following represents a repeatable pattern that you can apply to most data s