Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
August 26, 2021
Event-driven APIs &
Apache Kafka
Hugo Guerrero
APIs & Messaging Developer Advocate
Red Hat
Hugo Guerrero
Mexican @ MA, USA
APIs & Event-Driven
Specialist | Open Source
Advocate | History, Travel &
Food Enthusiast
...
Event-driven APIs &
Modern Applications
apidays
Hong
Kong
What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
4
@hguerreroo
Distributed Decoupled Highly Connected
apidays
Hong
...
What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
5
@hguerreroo
Distributed Decoupled Highly Connected
Microservices...
What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
6
@hguerreroo
Distributed Decoupled Highly Connected
Microservices...
The Traditional Workflow
EVENT-DRIVEN APIS & APACHE KAFKA
7
Project Planning
The business experts spend time writing up a
...
Kafkaesque
EVENT-DRIVEN APIS & APACHE KAFKA
8
Photo credit: dbeck03 on VisualHunt
ENDPOINT DOCUMENTATION ?
DATA FORMAT AND...
Kafkaesque
EVENT-DRIVEN APIS & APACHE KAFKA
9
Photo credit: dbeck03 on VisualHunt
INFRASTRUCTURE ACCESS ?
PUBLISHERS & CON...
Event-Driven APIs and
Contract-First Workflows
apidays
Hong
Kong
EVENT-DRIVEN APIS & APACHE KAFKA
11
Why Contract-First?
@hguerreroo
▸ Because you want to allow people to work independent...
How does it work on REST APIs?
EVENT-DRIVEN APIS & APACHE KAFKA
12
Source:
https://swagger.io/docs/specification/about/
ht...
Event-driven Secret Weapon
EVENT-DRIVEN APIS & APACHE KAFKA
13
An open source initiative that seeks to improve
the current...
AsyncAPI
EVENT-DRIVEN APIS & APACHE KAFKA
14
asyncapi.org
apidays
Hong
Kong
Protocol agnostic:
▸ AMQP
▸ MQTT
▸ NATS
▸ HTTP...
Unified API contracts
EVENT-DRIVEN APIS & APACHE KAFKA
15
@hguerreroo
POST /register
{
"fullName":"John Doe",
"email":"joh...
AsyncAPI Anatomy
EVENT-DRIVEN APIS & APACHE KAFKA
16
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Inf...
AsyncAPI Anatomy
EVENT-DRIVEN APIS & APACHE KAFKA
17
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Inf...
Schemas as Event
Contracts
apidays
Hong
Kong
EVENT-DRIVEN APIS & APACHE KAFKA
19
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |  
| |...
EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
20
Why is it important to use API contracts in Apache Kafk...
EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
21
Why is it important to use API contracts in Apache Kafk...
EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
22
Why is it important to use API contracts in Apache Kafk...
EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
23
Why is it important to use API contracts in Apache Kafk...
What issues needs to be addressed in a Kafka
world?
▸ Kafka is not aware of data types or schema
formats.
▸ API schemas ar...
A Registry for Event
Schemas
apidays
Hong
Kong
Using the Registry for Schema Management
EVENT-DRIVEN APIS & APACHE KAFKA
26
Producer Consumer
Deserializer
Serializer
Reg...
EVENT-DRIVEN APIS & APACHE KAFKA
What are the required
capabilities of the registry?
27
The registry needs to address the
...
EVENT-DRIVEN APIS & APACHE KAFKA
Registry
28
Use Cases
Source:
https://github.com/Apicurio/apicurio-registry
API specifica...
▸ Implementation preview of the CNCF
Schema Registry API.
▸ Part of the CloudEvents Working Group
efforts for standardizat...
▸ Getting Started
● https://developers.redhat.com/blog/2019/12/16/getting-started-with-red-hat-integration-service-registr...
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Hugo Guerrero
Twitter...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apache Kafka by Hugo Guerrero, Red Hat

Download to read offline

apidays LIVE Hong Kong 2021 - API Ecosystem & Data Interchange
August 25 & 26, 2021

Event-driven APIs & Schema governance for Apache Kafka
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apache Kafka by Hugo Guerrero, Red Hat

  1. 1. August 26, 2021 Event-driven APIs & Apache Kafka Hugo Guerrero APIs & Messaging Developer Advocate Red Hat
  2. 2. Hugo Guerrero Mexican @ MA, USA APIs & Event-Driven Specialist | Open Source Advocate | History, Travel & Food Enthusiast @hguerreroo apidays Hong Kong Agenda Event-driven APIs Event-driven development Kafkaesque Challenges Contract-First The REST of the World AsyncAPI Spec Event Schemas Schemas as Contracts Registry for Schemas
  3. 3. Event-driven APIs & Modern Applications apidays Hong Kong
  4. 4. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 4 @hguerreroo Distributed Decoupled Highly Connected apidays Hong Kong
  5. 5. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 5 @hguerreroo Distributed Decoupled Highly Connected Microservices Request / Response Events apidays Hong Kong
  6. 6. What is the situation? EVENT-DRIVEN APIS & APACHE KAFKA 6 @hguerreroo Distributed Decoupled Highly Connected Microservices Request / Response Events apidays Hong Kong Events
  7. 7. The Traditional Workflow EVENT-DRIVEN APIS & APACHE KAFKA 7 Project Planning The business experts spend time writing up a project plan for a series of services and potentially a user interface Send to Developers The various development teams attempt to interpret the project plan and implement their code so that they are compatible Iterate and Integrate The developer teams iteratively try to resolve integration issues and spend loads of time ensuring compatibility apidays Hong Kong
  8. 8. Kafkaesque EVENT-DRIVEN APIS & APACHE KAFKA 8 Photo credit: dbeck03 on VisualHunt ENDPOINT DOCUMENTATION ? DATA FORMAT AND VALIDATION ? MIS-UNDERSTANDING
  9. 9. Kafkaesque EVENT-DRIVEN APIS & APACHE KAFKA 9 Photo credit: dbeck03 on VisualHunt INFRASTRUCTURE ACCESS ? PUBLISHERS & CONSUMERS COMPLIANCE ? EVENTS AVAILABILITY ? SLOW TIME TO MARKET, POOR QUALITY
  10. 10. Event-Driven APIs and Contract-First Workflows apidays Hong Kong
  11. 11. EVENT-DRIVEN APIS & APACHE KAFKA 11 Why Contract-First? @hguerreroo ▸ Because you want to allow people to work independently ▸ Because you want to ensure consistency ▸ Because you need strong guarantees about service contracts ▸ Because you, your team, your customers, and your partners can collaborate ▸ Because you can save time by using code generators and testing tooling apidays Hong Kong
  12. 12. How does it work on REST APIs? EVENT-DRIVEN APIS & APACHE KAFKA 12 Source: https://swagger.io/docs/specification/about/ https://www.apicur.io/ https://openapi-generator.tech/ https://microcks.io/ Build an API Specification Using a tool like OpenAPI, write an API specification FIRST Publish the API Specification Using a tool like Swagger or Apicur.io, publish the API specification where others have access and can collaborate Generate Code, Mocks and Testing Using a tool like OpenAPI Generator to create the API stubs for both client and server applications. You can also generate “mock” services with Microcks. apidays Hong Kong
  13. 13. Event-driven Secret Weapon EVENT-DRIVEN APIS & APACHE KAFKA 13 An open source initiative that seeks to improve the current state of Event-Driven Architectures. Created to describe Event-driven APIs using MQTT, AMQP, Apache Kafka, ... “Sister” specification to OpenAPI: same base syntax, use JSON Schema underhood. Allows definition of examples AsyncAPI https://www.asyncapi.com/ apidays Hong Kong
  14. 14. AsyncAPI EVENT-DRIVEN APIS & APACHE KAFKA 14 asyncapi.org apidays Hong Kong Protocol agnostic: ▸ AMQP ▸ MQTT ▸ NATS ▸ HTTP ▸ WebSockets ▸ Kafka ▸ STOMP ▸ JMS Open Source Tooling: ▸ Online playground (playground.asyncapi.org) ▸ Code generators ▸ Documentation generators ▸ Editors ▸ Mocking ▸ Testing
  15. 15. Unified API contracts EVENT-DRIVEN APIS & APACHE KAFKA 15 @hguerreroo POST /register { "fullName":"John Doe", "email":"john@example.io", "age":36 } TOPIC user-registration { "id":"38102178-dbe5-4e81-aa65-8f6df1be4d44", "fullName":"John Doe", "email":"john@example.io", "age":36, "sendAt":"1603786974956", } OpenAPI 3 Contract AsyncAPI 2 Contract apidays Hong Kong
  16. 16. AsyncAPI Anatomy EVENT-DRIVEN APIS & APACHE KAFKA 16 https://www.asyncapi.com/docs/getting-started/coming-from-openapi Info: metadata and documentation for this asynchronous API: version, title, description, contact and licensing terms Servers: list of brokers with binding details (URLs and security protocols) Channel Item: destination that will be used by the API to publish or consume events. Destination may be mapped to a topic, queue, address name depending on the protocol binding. Operation: whether the application using the AsyncAPI should subscribe or produce messages. Holds its own descriptive metadata Message: the content-type and structure of message headers and payload. May reference schemas of different types (AsyncAPI, JSON, Avro). May contain examples! Components: Common definitions (schemas, traits, messages) like in OpenAPI v3 apidays Hong Kong
  17. 17. AsyncAPI Anatomy EVENT-DRIVEN APIS & APACHE KAFKA 17 https://www.asyncapi.com/docs/getting-started/coming-from-openapi Info: metadata and documentation for this asynchronous API: version, title, description, contact and licensing terms Servers: list of brokers with binding details (URLs and security protocols) Channel Item: destination that will be used by the API to publish or consume events. Destination may be mapped to a topic, queue, address name depending on the protocol binding. Operation: whether the application using the AsyncAPI should subscribe or produce messages. Holds its own descriptive metadata Message: the content-type and structure of message headers and payload. May reference schemas of different types (AsyncAPI, JSON, Avro). May include examples! Components: Common definitions (schemas, traits, messages) like in OpenAPI v3 apidays Hong Kong
  18. 18. Schemas as Event Contracts apidays Hong Kong
  19. 19. EVENT-DRIVEN APIS & APACHE KAFKA 19 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Send Data Retrieve Data Consumer apidays Hong Kong @hguerreroo Contract-First API Development
  20. 20. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 20 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer apidays Hong Kong @hguerreroo
  21. 21. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 21 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
  22. 22. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 22 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
  23. 23. EVENT-DRIVEN APIS & APACHE KAFKA Contract-First API Development 23 Why is it important to use API contracts in Apache Kafka? Producer | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Send Data Retrieve Data Consumer Consumer Consumer v2 v2 v2 v1 apidays Hong Kong @hguerreroo
  24. 24. What issues needs to be addressed in a Kafka world? ▸ Kafka is not aware of data types or schema formats. ▸ API schemas are subject to change. ▸ Central registry where the data schemas are stored and accessible. EVENT-DRIVEN APIS & APACHE KAFKA 24 Photo credit: Harald Groven on VisualHunt
  25. 25. A Registry for Event Schemas apidays Hong Kong
  26. 26. Using the Registry for Schema Management EVENT-DRIVEN APIS & APACHE KAFKA 26 Producer Consumer Deserializer Serializer Registry | | | | | | |   | | | | | | |   | | | | | | | Apache Kafka Get or Register Schema by Id Retrieve Schema by Id Topic B (JSON ) Topic C (Protobuf) Topic A (Avro) Send Serialized Data Retrieve Serialized Data apidays Hong Kong @hguerreroo
  27. 27. EVENT-DRIVEN APIS & APACHE KAFKA What are the required capabilities of the registry? 27 The registry needs to address the following 3 pillars for a successful management @hguerreroo You need to be able to manage all the different types of artifacts, be able to browse, download and document in single place. Artifact Management Support for different types of data structures, like Apache Avro, Google Protocol Buffers, or JSON Schemas. Data Formats Artifacts follow content evolution and are controlled by enabling content rules, including Validity and Compatibility. Versioning apidays Hong Kong
  28. 28. EVENT-DRIVEN APIS & APACHE KAFKA Registry 28 Use Cases Source: https://github.com/Apicurio/apicurio-registry API specification registry for API consumers Shared Data Types API Designs Shared data types (schemas) across API and event driven architectures Schema Registry Schema registry for Kafka serializers/deserializers apidays Hong Kong
  29. 29. ▸ Implementation preview of the CNCF Schema Registry API. ▸ Part of the CloudEvents Working Group efforts for standardization and interoperability. EVENT-DRIVEN APIS & APACHE KAFKA CNCF’s Schema Registry API 29 CNCF Schema Registry API support Source: https://github.com/cloudevents/spec/blob/master/schemaregistry/schemaregistry.md apidays Hong Kong
  30. 30. ▸ Getting Started ● https://developers.redhat.com/blog/2019/12/16/getting-started-with-red-hat-integration-service-registry ● https://developers.redhat.com/blog/2020/12/09/new-features-and-storage-options-in-red-hat-integration-service-registry-1 -1-ga ▸ Install and configure Service Registry for AMQ Streams ● Installing Service Registry from the OpenShift OperatorHub ● Configuring Service Registry with AMQ Streams storage on OpenShift ▸ Validating Schemas using Kafka Client Serializer/Deserializer ● Validating schemas using Kafka client serializers/deserializers ● Strategies to look up a schema ▸ Service Registry Java Client ● Managing Service Registry content using the Java client EVENT-DRIVEN APIS & APACHE KAFKA Useful Bookmarks 30 apidays Hong Kong @hguerreroo
  31. 31. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Hugo Guerrero Twitter: @hguerreroo https://www.youtube.com/user/hguerreroo https://developers.redhat.com/author/hugo-guerrero Thank You

apidays LIVE Hong Kong 2021 - API Ecosystem & Data Interchange August 25 & 26, 2021 Event-driven APIs & Schema governance for Apache Kafka Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat

Views

Total views

406

On Slideshare

0

From embeds

0

Number of embeds

5

Actions

Downloads

9

Shares

0

Comments

0

Likes

0

×