SlideShare ist ein Scribd-Unternehmen logo
1 von 32
Downloaden Sie, um offline zu lesen
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Practical Kerberos
with Apache HBase
Josh Elser
HBaseCon East
2016/09/26
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Engineer at Hortonworks, Member of the Apache Software Foundation
Top-Level Projects
• Apache Accumulo®
• Apache CalciteTM
• Apache CommonsTM
• Apache HBase®
• Apache PhoenixTM
ASF Incubator
• Apache FluoTM
• Apache GossipTM
• Apache PirkTM
• Apache RyaTM
• Apache SliderTM
These names are trademarks or registered trademarks
of the Apache Software Foundation.
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
… but today we’re talking about Kerberos!
- “The Madness beyond the Gate” [1]
- An exploration in black magic and voodoo
- The word most accompanied with expletives
1: https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/kerberos_the_madness.html
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What this talk won’t be...
3dom via https://www.flickr.com/photos/steve_l/6042206137/in/album-72157629289333057/, CC-BY-NC
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Kerberos
⬢ “Kerberos is a network authentication protocol. It is designed to provide strong
authentication for client/server applications by using secret-key cryptography” [1]
⬢ MIT Kerberos is one implementation
– Heimdal is another
– We’re talking about MIT Kerberos
⬢ Authentication over a computer network
– Not authorization
– No data privacy
1: http://web.mit.edu/kerberos/
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Kerberos
⬢ Key Distribution Center (KDC)
– Centralized server which grants Kerberos “tickets”
– The “trusted third party” of the security model
⬢ Users are defined by a ”principal”
– primary[/instance]@REALM
– A human: elserj@HORTONWORKS.COM
– A service: hbase/regionserver1.hbase.hwx.com@HORTONWORKS.COM
– elserj@HORTONWORKS.COM is unique with elserj/login.hbase.hwx.com@HORTONWORKS.COM
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Kerberos
⬢Principals are identified by a secret shared with the KDC
– A normal password
– A keytab file (non-plaintext “password”, suitable for non-interactive logins)
⬢ Kerberos Ticket obtained from the KDC by using your secret
– Tickets expire
– Tickets are renewable*
Client Server
KDC
Password/Keytab Keytab
Authenticated RPC
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Interacting with Kerberos
⬢ kadmin (or kadmin.local)
– Command-line interface for administrators to create, modify, delete principals.
⬢ kinit
– A command-line tool to obtain a ticket for a principal
– Places the ticket in a file on disk in a well-known location called a “ticket cache”
• Default location on Linux: /tmp/krb5cc_$(id –u `whoami`)
– The ticket cache is read-write protected for the user only (e.g. chmod 600)
– Can obtain a ticket for any principal using a password or keytab
– Ticket caches can hold multiple tickets
⬢ klist
– Lists the contents of the current user’s ticket cache
– Can list the keys in a keytab file
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Benefits of Kerberos
⬢ Building a secure, network-based authentication system is very hard
⬢ Functions on non-trusted networks
– Security for multi-tenant systems, protect against malicious and non-malicious users
⬢ Leveraged across the Apache Hadoop “Stack”
⬢ Widely integrated externally
– Operating systems and programming languages
⬢ Can integrate with Active Directory
Apache Hadoop is a registered trademark of the Apache Software Foundation
1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Promises
It’s simple, you just get your Kerberos ticket, use HBase and it knows who you are!
[elserj@localhost] $ kinit elserj
Password for elserj@HORTONWORKS.COM:
[elserj@localhost] $ hbase com.hortonworks.hbase.MyMapReduceJob
/user/elserj/my-big-data.txt
…
Success!
[elserj@localhost] $
1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reality
[elserj@localhost] $ kinit elserj
Password for elserj@HORTONWORKS.COM:
[elserj@localhost] $ hbase com.hortonworks.hbase.MyMapReduceJob /big-
data.txt
...
2016-09-26 14:03:11,549 FATAL [main] ipc.AbstractRpcClient
(RpcClientImpl.java:run(709)) – SASL authentication failed. The most
likely cause is missing or invalid credentials. Consider ‘kinit’.
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
[elserj@localhost] $
(╯°□°)╯︵ ┻━┻
1
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ok, let’s figure out what went wrong?
What should I search for?
RPC
SASL
GSSAPI
JGSSUGI
JAAS
KDC
JCE
Token
Ticket
Voldemort
“Bars near me
open now”
Cthulhu
Kerberos
1
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
How JVM-based applications can obtain Kerberos tickets
⬢ Extract a ticket from the local ticket cache for a principal
– hbase shell or hdfs dfs –ls /
⬢ UserGroupInformation Hadoop API (UGI)
– UserGroupInformation.loginUserFromKeytab(String, String)
– UserGroupInformation.loginUserFromKeytabAndReturnUGI(String, String)
⬢ javax.security.auth.Subject with Krb5LoginModule
– The APIs which UserGroupInformation uses under the covers
⬢ Automatic login via JAAS
– “Java Authentication and Authorization Service”, implementation of PAM (RFC 86.0)
– Configuration file, specified via Java system properties.
– Each “block” uses an identifier to denote login details for a specific system
1
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Service Logins
⬢ HBase services are daemons; they always use a keytab to login
⬢ Principal and keytab are specified in hbase-site.xml for each service
⬢ A JAAS configuration file is also provided for Apache ZooKeeper client authentication
– Necessary for authenticated ZooKeeper access (HBase-only ACLs)
⬢ HBase services automatically perform logins/renewals as necessary
– Anyone who tells you that they need to ”kinit for HBase to work” doesn’t know what they’re
talking about.
Apache ZooKeeper is a trademark of the Apache Software Foundation
1
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
HBase Clients
⬢ HBase clients will use a variety of mechanism for authentication
– Interactive use: ticket-cache
– Automated tasks/Daemons: UGI with keytab
⬢ Reminder: Kerberos tickets expire
– Clients must implement renewal logic
– UGI provides an API to do this
⬢ Typically, UGI is the way to go
–Concise and well-understood
1
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
On using UserGroupInformation correctly
⬢ We mentioned two different method calls earlier for logins
– void loginUserFromKeytab(String, String)
– UserGroupInformation loginUserFromKeytabAndReturnUGI(String, String)
⬢ loginUserFromKeytab is “global”
– Syntactic-sugar to make your life easier
– Works great when the application only acts as one user
⬢ loginUserFromKeytabAndReturnUGI is “localized”
– Requires invoking “doAs(...)”
– Allows for concurrent execution as different users in one JVM
1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enter SASL: authentication framework over a transport
⬢ SASL is a framework for building RPC systems with authentication
⬢ “Simple Authentication and Security Layer” RFC-4422
– “A framework for authentication and data security in Internet protocols” [1]
– “decouples authentication mechanisms from application protocols”[1]
• Generic Security Services Application Program Interface (GSSAPI) speaks Kerberos
• DIGEST-MD5 an HTTP Digest authentication-like method (delegation tokens)
– Data security aka Quality of Protection (QoP)
• auth: Authentication only (default)
• auth-int: Previous, and integrity check of message content
• auth-conf: Previous, and encryption of message content
[1] https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer
1
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Trust on an untrusted network
⬢ A Kerberos ticket implies a valid identity, not necessarily the identity you wanted
⬢ Kerberos relies on accurate/consistent DNS as the basis for a secure RPC model
– Secure your DNS as much as your KDC
⬢ Recall the service principal from earlier
– hbase/regionserver1.hbase.hwx.com@HORTONWORKS.COM
⬢ The instance must be a fully-qualified domain name
⬢ Clients need to know primary and instance must match DNS
– “Caused by: KrbException: Identifier doesn't match expected value (906)”
– “error Message is Server not found in Kerberos database”
1
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Trust on an untrusted network
Client Trusted ServiceGood
DNS
Rogue Service
Bad
DNS
service/svc1.hwx.com@HORTONWORKS.COMservice/svc18.hwx.com@HORTONWORKS.COM
Sends RPC “service” at
svc1.hwx.com
Without enforcement of DNS naming via SASL, a
client could be maliciously sent to a rogue service
without the client realizing it happened.
2
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Harping on DNS
⬢ DNS must be correct, consistent, and secure
⬢ Hostnames are advertised for discovery
– Also benefits multi-homed networks
⬢ Forward and Reverse DNS mappings must be accurate on every node
– `nslookup regionserver1.hbase.hwx.com` returns 10.0.0.1
– `nslookup 10.0.0.1` returns regionserver1.hbase.hwx.com
⬢ Check /etc/resolv.conf for quick troubleshooting
2
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Recap: Kerberos authentication for HBase RPCs
⬢ Client and Server both obtain Kerberos ticket
– Password or Keytab via UGI/JAAS/Ticket-Cache
– Tickets must be renewed before they expire
⬢ SASL is the framework which HBase leverages for authenticated RPCs
– GSSAPI as the SASL mechanism which can “speak” Kerberos
– QoP defines the security of the RPC data (minimum of authentication)
⬢ Fully-qualified hostnames everywhere
– Forward and reverse DNS must be consistent across all clients and servers
2
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
The edge cases
⬢ Exceptions to how authentication works
– YARN jobs
– HBase REST and Thrift services
⬢ Not the traditional client/server model Kerberos was designed to fit
– 100-1000’s of tasks concurrently requiring a ticket
– Talk to HBase as a user without having that user’s credentials
⬢ Two approaches introduced to address these problems
2
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Delegation Tokens
⬢ Earlier mentioned, SASL supports a variety of mechanisms
– DIGEST-MD5 allows a digest-token style authentication scheme
⬢ Delegation token is a temporary ”password” which can authenticate a user
– Slight compromise of security for performance
⬢ Circumvents authentication to the KDC, instead handled by HDFS or HBase
⬢ Automatically obtained during job submission and added to the job cache
– We must rely on YARN to do the right thing
If you thought Kerberos documentation for Hadoop/HBase was sparse…
2
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Delegation Tokens
Client HBase Master
KDCPassword/Keytab Keytab
Obtain DT
YARN
Containers
HBase
RegionServers
YARN
ResourceManager
Client Ticket
and DT YARN Ticket
and DT
DT
2
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Proxy Users
⬢ A proxy is some intermediate service that provides access to a backend service
– HBase Thrift and REST services
⬢ Each of these services have its own Kerberos principal and keytab used to communicate
with HBase
⬢ These services are accessing HBase on behalf of another user.
– The ticket is for the service, but we want it to appear as if it is elserj@HORTONWORKS.COM
⬢ ProxyUsers refer to a set of configuration values in Hadoop (core-site.xml)
– hadoop.proxyuser.SERVICE.{hosts,groups,users}
⬢ Configuration-based approach to allow services to “pretend” to be a user without
actually having that user’s credentials
2
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Proxy Users
Client
KDC
Password/Keytab
HBaseProxy Server
Client Ticket
Server Ticket
(Client principal)
Keytab
Keytab
Proxy Servers: HBase REST, HBase Thrift, Phoenix Query Server, etc
2
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kerberos authentication for HTTP-based services (SPNEGO)
⬢ The need to protect services using HTTP
–Don’t want to reuse SASL
⬢ Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) RFC-4178
– The Negotiate HTTP header
– Built into cURL (--negotiate), most Java-based HTTP libraries, and web-browsers
⬢ Web-browsers often need special configuration to properly authenticate.
– Firefox: network.negotiate-auth.delegation-uris, network.negotiate-auth.trusted-uris
– Chrome: --auth-server-whitelist="*.domain" --auth-negotiate-delegate-whitelist="*.domain"
2
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Troubleshooting: Prerequisites
⬢ Ensure a recent version of your JVM and Hadoop
– Bugs exist in UserGroupInformation for certain JVMs (vendor+version)
⬢ Ensure that the unlimited strength Java Cryptographic Extensions (JCE) are installed on
all nodes in the cluster
– And that clients/servers are using that JVM installation!
– Required for AES-256 encryption type on Kerberos keys (which you will likely get by default)
⬢ Ensure that you have DEBUG logging enabled for HBase services
– Potentially, org.apache.hadoop.hbase.ipc=DEBUG is sufficient
⬢ Set the sun.security.krb5.debug system property to true in your application
– Or sun.security.spnego.debug for debugging SPNEGO
2
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Troubleshooting: Tips
⬢ Remember that DNS is the cornerstone
– When reading logs, make sure that you see the expected fully-qualified domain names
– Do not assume that DNS is correct: verify it.
⬢ Determine if an RPC issue is authentication or authorization
– If you see an HBase-level error, it is likely an authorization issue
– If you only see transport/connection-setup errors, it is likely an authentication issue
⬢ Remember that tickets expire
– Cross-reference ticket lifetimes with application logs
⬢ Read the logs. Actually read them.
– A vast majority of errors can be solved with appropriate logging JVM-debugging
3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reference Material
⬢ “Hadoop and Kerberos: The Madness beyond the Gate”
– https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/index.html
⬢ Oracle documentation
– http://docs.oracle.com/javase/7/docs/technotes/guides/security/jaas/tutorials/GeneralAcnOnly.html
– https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5
LoginModule.html
⬢ MIT Kerberos documentation
– http://web.mit.edu/kerberos/
⬢ “Explain like I’m 5: Kerberos” (great low-level Kerberos write-up)
– http://www.roguelynn.com/words/explain-like-im-5-kerberos/
⬢ KDiag: “Kerberos diagnostics for Hadoop”
–Apache Hadoop >=2.8 or https://github.com/steveloughran/kdiag
3
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Developing with Kerberos
⬢ Apache Directory’s Kerby project
– Great for Kerberos authentication without Hadoop in the picture
– http://directory.apache.org/kerby/downloads.html
⬢ Apache Hadoop’s MiniKDC
– Built on top of Apache Directory
– https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-common-project/hadoop-
minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java
⬢ Support in HDFS, YARN, and HBase MiniCluster classes too
No excuse to not write tests!
Apache Directory is a trademark of the Apache Software Foundation
3
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thanks!
Email: elserj@apache.org
Twitter: @josh_elser
3dom via https://www.flickr.com/photos/steve_l/6674480535/in/album-72157629289333057/, CC-BY-NC
Thanks to those who gave feedback along the way: Brandon Wilson, Bryan Bende,
Michael Stack, Randy Gelhausen, Steve Loughran.

Weitere ähnliche Inhalte

Was ist angesagt?

Postman & API Testing by Amber Race
Postman & API Testing by Amber RacePostman & API Testing by Amber Race
Postman & API Testing by Amber RacePostman
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
 
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Shirshanka Das
 
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized WorldPostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized WorldJignesh Shah
 
A glimpse of cassandra 4.0 features netflix
A glimpse of cassandra 4.0 features   netflixA glimpse of cassandra 4.0 features   netflix
A glimpse of cassandra 4.0 features netflixVinay Kumar Chella
 
Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)Amazon Web Services
 
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...HostedbyConfluent
 
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxHadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxVinay Shukla
 
Introduction to REST - API
Introduction to REST - APIIntroduction to REST - API
Introduction to REST - APIChetan Gadodia
 
Apache Cassandra at the Geek2Geek Berlin
Apache Cassandra at the Geek2Geek BerlinApache Cassandra at the Geek2Geek Berlin
Apache Cassandra at the Geek2Geek BerlinChristian Johannsen
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and LinkerdService Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and LinkerdKai Wähner
 
Programming in Spark using PySpark
Programming in Spark using PySpark      Programming in Spark using PySpark
Programming in Spark using PySpark Mostafa
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang
 
Introduction to APIs (Application Programming Interface)
Introduction to APIs (Application Programming Interface) Introduction to APIs (Application Programming Interface)
Introduction to APIs (Application Programming Interface) Vibhawa Nirmal
 
Vectorized Query Execution in Apache Spark at Facebook
Vectorized Query Execution in Apache Spark at FacebookVectorized Query Execution in Apache Spark at Facebook
Vectorized Query Execution in Apache Spark at FacebookDatabricks
 
An Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a ServiceAn Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent
 
Source Code Analysis with SAST
Source Code Analysis with SASTSource Code Analysis with SAST
Source Code Analysis with SASTBlueinfy Solutions
 

Was ist angesagt? (20)

Postman & API Testing by Amber Race
Postman & API Testing by Amber RacePostman & API Testing by Amber Race
Postman & API Testing by Amber Race
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
REST API
REST APIREST API
REST API
 
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...
 
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized WorldPostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized World
 
A glimpse of cassandra 4.0 features netflix
A glimpse of cassandra 4.0 features   netflixA glimpse of cassandra 4.0 features   netflix
A glimpse of cassandra 4.0 features netflix
 
Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)Amazon Relational Database Service (Amazon RDS)
Amazon Relational Database Service (Amazon RDS)
 
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
 
Hadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache KnoxHadoop Security Today & Tomorrow with Apache Knox
Hadoop Security Today & Tomorrow with Apache Knox
 
Introduction to REST - API
Introduction to REST - APIIntroduction to REST - API
Introduction to REST - API
 
Apache Cassandra at the Geek2Geek Berlin
Apache Cassandra at the Geek2Geek BerlinApache Cassandra at the Geek2Geek Berlin
Apache Cassandra at the Geek2Geek Berlin
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and LinkerdService Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
 
Programming in Spark using PySpark
Programming in Spark using PySpark      Programming in Spark using PySpark
Programming in Spark using PySpark
 
Intro to Amazon ECS
Intro to Amazon ECSIntro to Amazon ECS
Intro to Amazon ECS
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka Streams
 
Introduction to APIs (Application Programming Interface)
Introduction to APIs (Application Programming Interface) Introduction to APIs (Application Programming Interface)
Introduction to APIs (Application Programming Interface)
 
Vectorized Query Execution in Apache Spark at Facebook
Vectorized Query Execution in Apache Spark at FacebookVectorized Query Execution in Apache Spark at Facebook
Vectorized Query Execution in Apache Spark at Facebook
 
An Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a ServiceAn Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a Service
 
Source Code Analysis with SAST
Source Code Analysis with SASTSource Code Analysis with SAST
Source Code Analysis with SAST
 

Ähnlich wie Practical Kerberos with Apache HBase

HBaseConEast2016: Practical Kerberos with Apache HBase
HBaseConEast2016: Practical Kerberos with Apache HBaseHBaseConEast2016: Practical Kerberos with Apache HBase
HBaseConEast2016: Practical Kerberos with Apache HBaseMichael Stack
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit
 
De-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServerDe-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServerJosh Elser
 
Apache Phoenix Query Server PhoenixCon2016
Apache Phoenix Query Server PhoenixCon2016Apache Phoenix Query Server PhoenixCon2016
Apache Phoenix Query Server PhoenixCon2016Josh Elser
 
Hadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the GateHadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the GateSteve Loughran
 
Apache Phoenix Query Server
Apache Phoenix Query ServerApache Phoenix Query Server
Apache Phoenix Query ServerJosh Elser
 
Apache Spark and Object Stores
Apache Spark and Object StoresApache Spark and Object Stores
Apache Spark and Object StoresSteve Loughran
 
Securing Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo VanzinSecuring Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo VanzinSpark Summit
 
Securing Your Apache Spark Applications
Securing Your Apache Spark ApplicationsSecuring Your Apache Spark Applications
Securing Your Apache Spark ApplicationsCloudera, Inc.
 
Introduction and HDInsight best practices
Introduction and HDInsight best practicesIntroduction and HDInsight best practices
Introduction and HDInsight best practicesAshish Thapliyal
 
Curb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure ClusterCurb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure Clusterahortonworks
 
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...Carolyn Duby
 
Visualizing Kafka Security
Visualizing Kafka SecurityVisualizing Kafka Security
Visualizing Kafka SecurityDataWorks Summit
 
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionHadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
 

Ähnlich wie Practical Kerberos with Apache HBase (20)

HBaseConEast2016: Practical Kerberos with Apache HBase
HBaseConEast2016: Practical Kerberos with Apache HBaseHBaseConEast2016: Practical Kerberos with Apache HBase
HBaseConEast2016: Practical Kerberos with Apache HBase
 
Practical Kerberos
Practical KerberosPractical Kerberos
Practical Kerberos
 
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
 
Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!
Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!
Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!
 
Curb your insecurity with HDP
Curb your insecurity with HDPCurb your insecurity with HDP
Curb your insecurity with HDP
 
De-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServerDe-Mystifying the Apache Phoenix QueryServer
De-Mystifying the Apache Phoenix QueryServer
 
Apache Phoenix Query Server PhoenixCon2016
Apache Phoenix Query Server PhoenixCon2016Apache Phoenix Query Server PhoenixCon2016
Apache Phoenix Query Server PhoenixCon2016
 
Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionRunning Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
 
Hadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the GateHadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the Gate
 
Apache Phoenix Query Server
Apache Phoenix Query ServerApache Phoenix Query Server
Apache Phoenix Query Server
 
Apache Spark and Object Stores
Apache Spark and Object StoresApache Spark and Object Stores
Apache Spark and Object Stores
 
Securing Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo VanzinSecuring Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo Vanzin
 
Securing Your Apache Spark Applications
Securing Your Apache Spark ApplicationsSecuring Your Apache Spark Applications
Securing Your Apache Spark Applications
 
Introduction and HDInsight best practices
Introduction and HDInsight best practicesIntroduction and HDInsight best practices
Introduction and HDInsight best practices
 
Securing Spark Applications
Securing Spark ApplicationsSecuring Spark Applications
Securing Spark Applications
 
Curb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure ClusterCurb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure Cluster
 
Apache Ranger
Apache RangerApache Ranger
Apache Ranger
 
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...
Boston Future of Data Meetup: May 2017: Spark Introduction with Credit Card F...
 
Visualizing Kafka Security
Visualizing Kafka SecurityVisualizing Kafka Security
Visualizing Kafka Security
 
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionHadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
 

Mehr von Josh Elser

Apache HBase Internals you hoped you Never Needed to Understand
Apache HBase Internals you hoped you Never Needed to UnderstandApache HBase Internals you hoped you Never Needed to Understand
Apache HBase Internals you hoped you Never Needed to UnderstandJosh Elser
 
Effective Testing of Apache Accumulo Iterators
Effective Testing of Apache Accumulo IteratorsEffective Testing of Apache Accumulo Iterators
Effective Testing of Apache Accumulo IteratorsJosh Elser
 
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data WarehouseApache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data WarehouseJosh Elser
 
Apache Accumulo 1.8.0 Overview
Apache Accumulo 1.8.0 OverviewApache Accumulo 1.8.0 Overview
Apache Accumulo 1.8.0 OverviewJosh Elser
 
Calcite meetup-2016-04-20
Calcite meetup-2016-04-20Calcite meetup-2016-04-20
Calcite meetup-2016-04-20Josh Elser
 
Designing and Testing Accumulo Iterators
Designing and Testing Accumulo IteratorsDesigning and Testing Accumulo Iterators
Designing and Testing Accumulo IteratorsJosh Elser
 
Alternatives to Apache Accumulo’s Java API
Alternatives to Apache Accumulo’s Java APIAlternatives to Apache Accumulo’s Java API
Alternatives to Apache Accumulo’s Java APIJosh Elser
 
Data-Center Replication with Apache Accumulo
Data-Center Replication with Apache AccumuloData-Center Replication with Apache Accumulo
Data-Center Replication with Apache AccumuloJosh Elser
 
RPInventory 2-25-2010
RPInventory 2-25-2010RPInventory 2-25-2010
RPInventory 2-25-2010Josh Elser
 

Mehr von Josh Elser (9)

Apache HBase Internals you hoped you Never Needed to Understand
Apache HBase Internals you hoped you Never Needed to UnderstandApache HBase Internals you hoped you Never Needed to Understand
Apache HBase Internals you hoped you Never Needed to Understand
 
Effective Testing of Apache Accumulo Iterators
Effective Testing of Apache Accumulo IteratorsEffective Testing of Apache Accumulo Iterators
Effective Testing of Apache Accumulo Iterators
 
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data WarehouseApache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
 
Apache Accumulo 1.8.0 Overview
Apache Accumulo 1.8.0 OverviewApache Accumulo 1.8.0 Overview
Apache Accumulo 1.8.0 Overview
 
Calcite meetup-2016-04-20
Calcite meetup-2016-04-20Calcite meetup-2016-04-20
Calcite meetup-2016-04-20
 
Designing and Testing Accumulo Iterators
Designing and Testing Accumulo IteratorsDesigning and Testing Accumulo Iterators
Designing and Testing Accumulo Iterators
 
Alternatives to Apache Accumulo’s Java API
Alternatives to Apache Accumulo’s Java APIAlternatives to Apache Accumulo’s Java API
Alternatives to Apache Accumulo’s Java API
 
Data-Center Replication with Apache Accumulo
Data-Center Replication with Apache AccumuloData-Center Replication with Apache Accumulo
Data-Center Replication with Apache Accumulo
 
RPInventory 2-25-2010
RPInventory 2-25-2010RPInventory 2-25-2010
RPInventory 2-25-2010
 

Kürzlich hochgeladen

Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfBrain Inventory
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 

Kürzlich hochgeladen (20)

Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Why Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdfWhy Choose Brain Inventory For Ecommerce Development.pdf
Why Choose Brain Inventory For Ecommerce Development.pdf
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 

Practical Kerberos with Apache HBase

  • 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Practical Kerberos with Apache HBase Josh Elser HBaseCon East 2016/09/26
  • 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Engineer at Hortonworks, Member of the Apache Software Foundation Top-Level Projects • Apache Accumulo® • Apache CalciteTM • Apache CommonsTM • Apache HBase® • Apache PhoenixTM ASF Incubator • Apache FluoTM • Apache GossipTM • Apache PirkTM • Apache RyaTM • Apache SliderTM These names are trademarks or registered trademarks of the Apache Software Foundation.
  • 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved … but today we’re talking about Kerberos! - “The Madness beyond the Gate” [1] - An exploration in black magic and voodoo - The word most accompanied with expletives 1: https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/kerberos_the_madness.html
  • 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What this talk won’t be... 3dom via https://www.flickr.com/photos/steve_l/6042206137/in/album-72157629289333057/, CC-BY-NC
  • 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Introduction to Kerberos ⬢ “Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography” [1] ⬢ MIT Kerberos is one implementation – Heimdal is another – We’re talking about MIT Kerberos ⬢ Authentication over a computer network – Not authorization – No data privacy 1: http://web.mit.edu/kerberos/
  • 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Introduction to Kerberos ⬢ Key Distribution Center (KDC) – Centralized server which grants Kerberos “tickets” – The “trusted third party” of the security model ⬢ Users are defined by a ”principal” – primary[/instance]@REALM – A human: elserj@HORTONWORKS.COM – A service: hbase/regionserver1.hbase.hwx.com@HORTONWORKS.COM – elserj@HORTONWORKS.COM is unique with elserj/login.hbase.hwx.com@HORTONWORKS.COM
  • 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Introduction to Kerberos ⬢Principals are identified by a secret shared with the KDC – A normal password – A keytab file (non-plaintext “password”, suitable for non-interactive logins) ⬢ Kerberos Ticket obtained from the KDC by using your secret – Tickets expire – Tickets are renewable* Client Server KDC Password/Keytab Keytab Authenticated RPC
  • 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interacting with Kerberos ⬢ kadmin (or kadmin.local) – Command-line interface for administrators to create, modify, delete principals. ⬢ kinit – A command-line tool to obtain a ticket for a principal – Places the ticket in a file on disk in a well-known location called a “ticket cache” • Default location on Linux: /tmp/krb5cc_$(id –u `whoami`) – The ticket cache is read-write protected for the user only (e.g. chmod 600) – Can obtain a ticket for any principal using a password or keytab – Ticket caches can hold multiple tickets ⬢ klist – Lists the contents of the current user’s ticket cache – Can list the keys in a keytab file
  • 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Benefits of Kerberos ⬢ Building a secure, network-based authentication system is very hard ⬢ Functions on non-trusted networks – Security for multi-tenant systems, protect against malicious and non-malicious users ⬢ Leveraged across the Apache Hadoop “Stack” ⬢ Widely integrated externally – Operating systems and programming languages ⬢ Can integrate with Active Directory Apache Hadoop is a registered trademark of the Apache Software Foundation
  • 10. 1 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Promises It’s simple, you just get your Kerberos ticket, use HBase and it knows who you are! [elserj@localhost] $ kinit elserj Password for elserj@HORTONWORKS.COM: [elserj@localhost] $ hbase com.hortonworks.hbase.MyMapReduceJob /user/elserj/my-big-data.txt … Success! [elserj@localhost] $
  • 11. 1 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reality [elserj@localhost] $ kinit elserj Password for elserj@HORTONWORKS.COM: [elserj@localhost] $ hbase com.hortonworks.hbase.MyMapReduceJob /big- data.txt ... 2016-09-26 14:03:11,549 FATAL [main] ipc.AbstractRpcClient (RpcClientImpl.java:run(709)) – SASL authentication failed. The most likely cause is missing or invalid credentials. Consider ‘kinit’. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [elserj@localhost] $ (╯°□°)╯︵ ┻━┻
  • 12. 1 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ok, let’s figure out what went wrong? What should I search for? RPC SASL GSSAPI JGSSUGI JAAS KDC JCE Token Ticket Voldemort “Bars near me open now” Cthulhu Kerberos
  • 13. 1 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved How JVM-based applications can obtain Kerberos tickets ⬢ Extract a ticket from the local ticket cache for a principal – hbase shell or hdfs dfs –ls / ⬢ UserGroupInformation Hadoop API (UGI) – UserGroupInformation.loginUserFromKeytab(String, String) – UserGroupInformation.loginUserFromKeytabAndReturnUGI(String, String) ⬢ javax.security.auth.Subject with Krb5LoginModule – The APIs which UserGroupInformation uses under the covers ⬢ Automatic login via JAAS – “Java Authentication and Authorization Service”, implementation of PAM (RFC 86.0) – Configuration file, specified via Java system properties. – Each “block” uses an identifier to denote login details for a specific system
  • 14. 1 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HBase Service Logins ⬢ HBase services are daemons; they always use a keytab to login ⬢ Principal and keytab are specified in hbase-site.xml for each service ⬢ A JAAS configuration file is also provided for Apache ZooKeeper client authentication – Necessary for authenticated ZooKeeper access (HBase-only ACLs) ⬢ HBase services automatically perform logins/renewals as necessary – Anyone who tells you that they need to ”kinit for HBase to work” doesn’t know what they’re talking about. Apache ZooKeeper is a trademark of the Apache Software Foundation
  • 15. 1 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HBase Clients ⬢ HBase clients will use a variety of mechanism for authentication – Interactive use: ticket-cache – Automated tasks/Daemons: UGI with keytab ⬢ Reminder: Kerberos tickets expire – Clients must implement renewal logic – UGI provides an API to do this ⬢ Typically, UGI is the way to go –Concise and well-understood
  • 16. 1 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved On using UserGroupInformation correctly ⬢ We mentioned two different method calls earlier for logins – void loginUserFromKeytab(String, String) – UserGroupInformation loginUserFromKeytabAndReturnUGI(String, String) ⬢ loginUserFromKeytab is “global” – Syntactic-sugar to make your life easier – Works great when the application only acts as one user ⬢ loginUserFromKeytabAndReturnUGI is “localized” – Requires invoking “doAs(...)” – Allows for concurrent execution as different users in one JVM
  • 17. 1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enter SASL: authentication framework over a transport ⬢ SASL is a framework for building RPC systems with authentication ⬢ “Simple Authentication and Security Layer” RFC-4422 – “A framework for authentication and data security in Internet protocols” [1] – “decouples authentication mechanisms from application protocols”[1] • Generic Security Services Application Program Interface (GSSAPI) speaks Kerberos • DIGEST-MD5 an HTTP Digest authentication-like method (delegation tokens) – Data security aka Quality of Protection (QoP) • auth: Authentication only (default) • auth-int: Previous, and integrity check of message content • auth-conf: Previous, and encryption of message content [1] https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer
  • 18. 1 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Trust on an untrusted network ⬢ A Kerberos ticket implies a valid identity, not necessarily the identity you wanted ⬢ Kerberos relies on accurate/consistent DNS as the basis for a secure RPC model – Secure your DNS as much as your KDC ⬢ Recall the service principal from earlier – hbase/regionserver1.hbase.hwx.com@HORTONWORKS.COM ⬢ The instance must be a fully-qualified domain name ⬢ Clients need to know primary and instance must match DNS – “Caused by: KrbException: Identifier doesn't match expected value (906)” – “error Message is Server not found in Kerberos database”
  • 19. 1 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Trust on an untrusted network Client Trusted ServiceGood DNS Rogue Service Bad DNS service/svc1.hwx.com@HORTONWORKS.COMservice/svc18.hwx.com@HORTONWORKS.COM Sends RPC “service” at svc1.hwx.com Without enforcement of DNS naming via SASL, a client could be maliciously sent to a rogue service without the client realizing it happened.
  • 20. 2 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Harping on DNS ⬢ DNS must be correct, consistent, and secure ⬢ Hostnames are advertised for discovery – Also benefits multi-homed networks ⬢ Forward and Reverse DNS mappings must be accurate on every node – `nslookup regionserver1.hbase.hwx.com` returns 10.0.0.1 – `nslookup 10.0.0.1` returns regionserver1.hbase.hwx.com ⬢ Check /etc/resolv.conf for quick troubleshooting
  • 21. 2 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Recap: Kerberos authentication for HBase RPCs ⬢ Client and Server both obtain Kerberos ticket – Password or Keytab via UGI/JAAS/Ticket-Cache – Tickets must be renewed before they expire ⬢ SASL is the framework which HBase leverages for authenticated RPCs – GSSAPI as the SASL mechanism which can “speak” Kerberos – QoP defines the security of the RPC data (minimum of authentication) ⬢ Fully-qualified hostnames everywhere – Forward and reverse DNS must be consistent across all clients and servers
  • 22. 2 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The edge cases ⬢ Exceptions to how authentication works – YARN jobs – HBase REST and Thrift services ⬢ Not the traditional client/server model Kerberos was designed to fit – 100-1000’s of tasks concurrently requiring a ticket – Talk to HBase as a user without having that user’s credentials ⬢ Two approaches introduced to address these problems
  • 23. 2 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Delegation Tokens ⬢ Earlier mentioned, SASL supports a variety of mechanisms – DIGEST-MD5 allows a digest-token style authentication scheme ⬢ Delegation token is a temporary ”password” which can authenticate a user – Slight compromise of security for performance ⬢ Circumvents authentication to the KDC, instead handled by HDFS or HBase ⬢ Automatically obtained during job submission and added to the job cache – We must rely on YARN to do the right thing If you thought Kerberos documentation for Hadoop/HBase was sparse…
  • 24. 2 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Delegation Tokens Client HBase Master KDCPassword/Keytab Keytab Obtain DT YARN Containers HBase RegionServers YARN ResourceManager Client Ticket and DT YARN Ticket and DT DT
  • 25. 2 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Proxy Users ⬢ A proxy is some intermediate service that provides access to a backend service – HBase Thrift and REST services ⬢ Each of these services have its own Kerberos principal and keytab used to communicate with HBase ⬢ These services are accessing HBase on behalf of another user. – The ticket is for the service, but we want it to appear as if it is elserj@HORTONWORKS.COM ⬢ ProxyUsers refer to a set of configuration values in Hadoop (core-site.xml) – hadoop.proxyuser.SERVICE.{hosts,groups,users} ⬢ Configuration-based approach to allow services to “pretend” to be a user without actually having that user’s credentials
  • 26. 2 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Proxy Users Client KDC Password/Keytab HBaseProxy Server Client Ticket Server Ticket (Client principal) Keytab Keytab Proxy Servers: HBase REST, HBase Thrift, Phoenix Query Server, etc
  • 27. 2 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kerberos authentication for HTTP-based services (SPNEGO) ⬢ The need to protect services using HTTP –Don’t want to reuse SASL ⬢ Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) RFC-4178 – The Negotiate HTTP header – Built into cURL (--negotiate), most Java-based HTTP libraries, and web-browsers ⬢ Web-browsers often need special configuration to properly authenticate. – Firefox: network.negotiate-auth.delegation-uris, network.negotiate-auth.trusted-uris – Chrome: --auth-server-whitelist="*.domain" --auth-negotiate-delegate-whitelist="*.domain"
  • 28. 2 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Troubleshooting: Prerequisites ⬢ Ensure a recent version of your JVM and Hadoop – Bugs exist in UserGroupInformation for certain JVMs (vendor+version) ⬢ Ensure that the unlimited strength Java Cryptographic Extensions (JCE) are installed on all nodes in the cluster – And that clients/servers are using that JVM installation! – Required for AES-256 encryption type on Kerberos keys (which you will likely get by default) ⬢ Ensure that you have DEBUG logging enabled for HBase services – Potentially, org.apache.hadoop.hbase.ipc=DEBUG is sufficient ⬢ Set the sun.security.krb5.debug system property to true in your application – Or sun.security.spnego.debug for debugging SPNEGO
  • 29. 2 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Troubleshooting: Tips ⬢ Remember that DNS is the cornerstone – When reading logs, make sure that you see the expected fully-qualified domain names – Do not assume that DNS is correct: verify it. ⬢ Determine if an RPC issue is authentication or authorization – If you see an HBase-level error, it is likely an authorization issue – If you only see transport/connection-setup errors, it is likely an authentication issue ⬢ Remember that tickets expire – Cross-reference ticket lifetimes with application logs ⬢ Read the logs. Actually read them. – A vast majority of errors can be solved with appropriate logging JVM-debugging
  • 30. 3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reference Material ⬢ “Hadoop and Kerberos: The Madness beyond the Gate” – https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/index.html ⬢ Oracle documentation – http://docs.oracle.com/javase/7/docs/technotes/guides/security/jaas/tutorials/GeneralAcnOnly.html – https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5 LoginModule.html ⬢ MIT Kerberos documentation – http://web.mit.edu/kerberos/ ⬢ “Explain like I’m 5: Kerberos” (great low-level Kerberos write-up) – http://www.roguelynn.com/words/explain-like-im-5-kerberos/ ⬢ KDiag: “Kerberos diagnostics for Hadoop” –Apache Hadoop >=2.8 or https://github.com/steveloughran/kdiag
  • 31. 3 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Developing with Kerberos ⬢ Apache Directory’s Kerby project – Great for Kerberos authentication without Hadoop in the picture – http://directory.apache.org/kerby/downloads.html ⬢ Apache Hadoop’s MiniKDC – Built on top of Apache Directory – https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-common-project/hadoop- minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java ⬢ Support in HDFS, YARN, and HBase MiniCluster classes too No excuse to not write tests! Apache Directory is a trademark of the Apache Software Foundation
  • 32. 3 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved3 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thanks! Email: elserj@apache.org Twitter: @josh_elser 3dom via https://www.flickr.com/photos/steve_l/6674480535/in/album-72157629289333057/, CC-BY-NC Thanks to those who gave feedback along the way: Brandon Wilson, Bryan Bende, Michael Stack, Randy Gelhausen, Steve Loughran.