apidays LIVE Hong Kong 2021 - API Ecosystem & Data Interchange August 25 & 26, 2021 Event-driven APIs & Schema governance for Apache Kafka Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

1. August 26, 2021
Event-driven APIs &
Apache Kafka
Hugo Guerrero
APIs & Messaging Developer Advocate
Red Hat

2. Hugo Guerrero
Mexican @ MA, USA
APIs & Event-Driven
Specialist | Open Source
Advocate | History, Travel &
Food Enthusiast
@hguerreroo
apidays
Hong
Kong
Agenda
Event-driven APIs
Event-driven development
Kafkaesque Challenges
Contract-First
The REST of the World
AsyncAPI Spec
Event Schemas
Schemas as Contracts
Registry for Schemas

3. Event-driven APIs &
Modern Applications
apidays
Hong
Kong

4. What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
4
@hguerreroo
Distributed Decoupled Highly Connected
apidays
Hong
Kong

5. What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
5
@hguerreroo
Distributed Decoupled Highly Connected
Microservices
Request /
Response
Events
apidays
Hong
Kong

6. What is the situation?
EVENT-DRIVEN APIS & APACHE KAFKA
6
@hguerreroo
Distributed Decoupled Highly Connected
Microservices
Request /
Response
Events
apidays
Hong
Kong
Events

7. The Traditional Workflow
EVENT-DRIVEN APIS & APACHE KAFKA
7
Project Planning
The business experts spend time writing up a
project plan for a series of services and
potentially a user interface
Send to Developers
The various development teams attempt to
interpret the project plan and implement their
code so that they are compatible
Iterate and Integrate
The developer teams iteratively try to resolve
integration issues and spend loads of time
ensuring compatibility
apidays
Hong
Kong

8. Kafkaesque
EVENT-DRIVEN APIS & APACHE KAFKA
8
Photo credit: dbeck03 on VisualHunt
ENDPOINT DOCUMENTATION ?
DATA FORMAT AND
VALIDATION ?
MIS-UNDERSTANDING

9. Kafkaesque
EVENT-DRIVEN APIS & APACHE KAFKA
9
Photo credit: dbeck03 on VisualHunt
INFRASTRUCTURE ACCESS ?
PUBLISHERS & CONSUMERS
COMPLIANCE ?
EVENTS AVAILABILITY ?
SLOW TIME TO MARKET,
POOR QUALITY

10. Event-Driven APIs and
Contract-First Workflows
apidays
Hong
Kong

11. EVENT-DRIVEN APIS & APACHE KAFKA
11
Why Contract-First?
@hguerreroo
▸ Because you want to allow people to work independently
▸ Because you want to ensure consistency
▸ Because you need strong guarantees about service contracts
▸ Because you, your team, your customers, and your partners can collaborate
▸ Because you can save time by using code generators and testing tooling
apidays
Hong
Kong

12. How does it work on REST APIs?
EVENT-DRIVEN APIS & APACHE KAFKA
12
Source:
https://swagger.io/docs/specification/about/
https://www.apicur.io/
https://openapi-generator.tech/
https://microcks.io/
Build an API Specification
Using a tool like OpenAPI,
write an API specification FIRST
Publish the API Specification
Using a tool like Swagger or Apicur.io,
publish the API specification where others
have access and can collaborate
Generate Code, Mocks and Testing
Using a tool like OpenAPI Generator to create
the API stubs for both client and server
applications. You can also generate “mock”
services with Microcks.
apidays
Hong
Kong

13. Event-driven Secret Weapon
EVENT-DRIVEN APIS & APACHE KAFKA
13
An open source initiative that seeks to improve
the current state of Event-Driven Architectures.
Created to describe Event-driven APIs using
MQTT, AMQP, Apache Kafka, ...
“Sister” specification to OpenAPI: same base
syntax, use JSON Schema underhood.
Allows definition of examples
AsyncAPI
https://www.asyncapi.com/
apidays
Hong
Kong

14. AsyncAPI
EVENT-DRIVEN APIS & APACHE KAFKA
14
asyncapi.org
apidays
Hong
Kong
Protocol agnostic:
▸ AMQP
▸ MQTT
▸ NATS
▸ HTTP
▸ WebSockets
▸ Kafka
▸ STOMP
▸ JMS
Open Source Tooling:
▸ Online playground
(playground.asyncapi.org)
▸ Code generators
▸ Documentation generators
▸ Editors
▸ Mocking
▸ Testing

15. Unified API contracts
EVENT-DRIVEN APIS & APACHE KAFKA
15
@hguerreroo
POST /register
{
"fullName":"John Doe",
"email":"john@example.io",
"age":36
}
TOPIC user-registration
{
"id":"38102178-dbe5-4e81-aa65-8f6df1be4d44",
"fullName":"John Doe",
"email":"john@example.io",
"age":36,
"sendAt":"1603786974956",
}
OpenAPI 3
Contract
AsyncAPI 2
Contract
apidays
Hong
Kong

16. AsyncAPI Anatomy
EVENT-DRIVEN APIS & APACHE KAFKA
16
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Info: metadata and documentation for this asynchronous API: version, title,
description, contact and licensing terms
Servers: list of brokers with binding details (URLs and security protocols)
Channel Item: destination that will be used by the API to publish or
consume events. Destination may be mapped to a topic, queue, address
name depending on the protocol binding.
Operation: whether the application using the AsyncAPI should subscribe
or produce messages. Holds its own descriptive metadata
Message: the content-type and structure of message headers and
payload. May reference schemas of different types (AsyncAPI, JSON,
Avro). May contain examples!
Components: Common definitions (schemas, traits, messages) like in
OpenAPI v3
apidays
Hong
Kong

17. AsyncAPI Anatomy
EVENT-DRIVEN APIS & APACHE KAFKA
17
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Info: metadata and documentation for this asynchronous API: version, title,
description, contact and licensing terms
Servers: list of brokers with binding details (URLs and security protocols)
Channel Item: destination that will be used by the API to publish or
consume events. Destination may be mapped to a topic, queue, address
name depending on the protocol binding.
Operation: whether the application using the AsyncAPI should subscribe
or produce messages. Holds its own descriptive metadata
Message: the content-type and structure of message headers and
payload. May reference schemas of different types (AsyncAPI, JSON,
Avro). May include examples!
Components: Common definitions (schemas, traits, messages) like in
OpenAPI v3
apidays
Hong
Kong

18. Schemas as Event
Contracts
apidays
Hong
Kong

19. EVENT-DRIVEN APIS & APACHE KAFKA
19
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
apidays
Hong
Kong
@hguerreroo
Contract-First API Development

20. EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
20
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
apidays
Hong
Kong
@hguerreroo

21. EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
21
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1
apidays
Hong
Kong
@hguerreroo

22. EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
22
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1
apidays
Hong
Kong
@hguerreroo

23. EVENT-DRIVEN APIS & APACHE KAFKA
Contract-First API Development
23
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1
apidays
Hong
Kong
@hguerreroo

24. What issues needs to be addressed in a Kafka
world?
▸ Kafka is not aware of data types or schema
formats.
▸ API schemas are subject to change.
▸ Central registry where the data schemas are
stored and accessible.
EVENT-DRIVEN APIS & APACHE KAFKA
24
Photo credit: Harald Groven on VisualHunt

25. A Registry for Event
Schemas
apidays
Hong
Kong

26. Using the Registry for Schema Management
EVENT-DRIVEN APIS & APACHE KAFKA
26
Producer Consumer
Deserializer
Serializer
Registry
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Get or
Register
Schema
by Id
Retrieve
Schema
by Id
Topic B
(JSON
)
Topic C
(Protobuf)
Topic A
(Avro)
Send
Serialized
Data
Retrieve
Serialized
Data
apidays
Hong
Kong
@hguerreroo

27. EVENT-DRIVEN APIS & APACHE KAFKA
What are the required
capabilities of the registry?
27
The registry needs to address the
following 3 pillars for a successful
management
@hguerreroo
You need to be able to manage all the different types
of artifacts, be able to browse, download and
document in single place.
Artifact Management
Support for different types of data structures, like
Apache Avro, Google Protocol Buffers, or JSON
Schemas.
Data Formats
Artifacts follow content evolution and are controlled
by enabling content rules, including Validity and
Compatibility.
Versioning
apidays
Hong
Kong

28. EVENT-DRIVEN APIS & APACHE KAFKA
Registry
28
Use Cases
Source:
https://github.com/Apicurio/apicurio-registry
API specification registry for
API consumers
Shared Data Types
API Designs
Shared data types (schemas)
across
API and event driven architectures
Schema Registry
Schema registry for Kafka
serializers/deserializers
apidays
Hong
Kong

29. ▸ Implementation preview of the CNCF
Schema Registry API.
▸ Part of the CloudEvents Working Group
efforts for standardization and
interoperability.
EVENT-DRIVEN APIS & APACHE KAFKA
CNCF’s Schema Registry API
29
CNCF Schema Registry API support
Source:
https://github.com/cloudevents/spec/blob/master/schemaregistry/schemaregistry.md
apidays
Hong
Kong

30. ▸ Getting Started
● https://developers.redhat.com/blog/2019/12/16/getting-started-with-red-hat-integration-service-registry
● https://developers.redhat.com/blog/2020/12/09/new-features-and-storage-options-in-red-hat-integration-service-registry-1
-1-ga
▸ Install and configure Service Registry for AMQ Streams
● Installing Service Registry from the OpenShift OperatorHub
● Configuring Service Registry with AMQ Streams storage on OpenShift
▸ Validating Schemas using Kafka Client Serializer/Deserializer
● Validating schemas using Kafka client serializers/deserializers
● Strategies to look up a schema
▸ Service Registry Java Client
● Managing Service Registry content using the Java client
EVENT-DRIVEN APIS & APACHE KAFKA
Useful Bookmarks
30
apidays
Hong
Kong
@hguerreroo

31. linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Hugo Guerrero
Twitter: @hguerreroo
https://www.youtube.com/user/hguerreroo
https://developers.redhat.com/author/hugo-guerrero
Thank You