Mingyu Bao presented on the application of Apache Pulsar in Tencent's billing and advertising systems. Tencent developed their own distributed transaction framework (TDXA) and message queue (TDMQ) based on Pulsar to ensure high transaction consistency and availability across regions. TDMQ supports asynchronous data transmission between systems, exception handling, and high data consistency. Pulsar is also used in Tencent's advertising system to support real-time and offline data processing with low latency, handle massively delayed messages through wheel timers and delayed topics, and currently processes over 100 billion messages per day across 70+ clusters with maximum throughput of 166w+ requests per second.

1. Pulsar Virtual Summit Europe 2021
Application of Apache pulsar
in Tencent Billing and
Tencent Advertising
Mingyu Bao
Senior Software Engineer,Tencent

2. Pulsar Virtual Summit Europe 2021
Self-Introduction
MingYu Bao.
Senior Software Engineer at Tencent.
Professional in distributed financial systems.
Focused on distributed message queue
system and transaction engine architecture,
providing billing services of high stability for
Tencent's business.
Open-source enthusiast.
Contributed of Apache Pulsar community.

3. Pulsar Virtual Summit Europe 2021
Midas
米大师
Standard Edition
Games,Video,Club etc
30+ Billions of Accounts
Billions of Transactions
Enterprise Editon
WeChat Ads
Cloud Up to 1,000,000 TPS
OverSeas Editon
• 100+ Payment Channels
• 180+ Countries
• Full link Risk Control
Midas platform
Industry Edition
Education，Travel，Traffic etc

4. Pulsar Virtual Summit Europe 2021
Midas Architecture

5. Pulsar Virtual Summit Europe 2021
 High consistency of transaction, ensuring the consistency of payments and goods
delivery
 Failure and timeout: A single payment of Midas often involves many internal and external systems. This leads to
longer call chains and more exceptions, especially in network timeouts (e.g. overseas payment services), DB
manipulation, goods delivery Failures.
 High data reliability crossing regions
 Oceans of timed processing
 subscription & recurring billing, in-time account reconciliation
 High Performance
 High Scalability
 automatically scales elastically according to the business scope
Pain Points of Billing platform

6. Pulsar Virtual Summit Europe 2021
Our Solution
Producer
Consumer
http
proxy
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster-1
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster-1
Consumer
Business side Op side
Second meter monitoring
In/Out message tracing
Flow control
High data reliability
Core side
Import/export tool

7. Pulsar Virtual Summit Europe 2021
Our Solution
TDXA+TDMQ
TDXA, a framework solution for transaction control. Ensuring high consistency of transactions, high fault tolerance, high scalability.
TDMQ, based on Pulsar，distributed Message Queue. EnablesTDXA to deal with messaging with high consistency and availability, and makes
convergence of various exceptions with the state table.

8. Pulsar Virtual Summit Europe 2021
Deployment of Tencent billing system
…
TDSQL
…
TDSQL
TDXA TDXA TDXA
App App App
TDMQ TDMQ
Wechat
Pay
Wechat
Pay
bank
Shenzhen Shanghai Hongkong
TDSQL
TDMQ’s Achievements:
1. Asynchronous data
transmission
between sytems
2. System Exception
handling
3. High data
consistency
3
1
2

9. Pulsar Virtual Summit Europe 2021
TDsort
Function
broker
Application of Pulsar in Apache InLong
SDK Agent http
Bus/DataProxy
CKafka Hippo TubeMQ Pulsar
Sort
Real-time/offli
ne calculation
SDK
Aggregate
Cache
Consume
Data access universal kit – Apache InLong
DBsync
Collection
Apache InLong

10. Pulsar Virtual Summit Europe 2021
Pulsar in Tencent Advertising
Advertising pool
Apache InLong
DBsync Pulsar
Sync consumer
Push consumer
Public
BM CRM TCC Etc.
Subscribe
Wide table
Click house
Auth ES/Provider ES
Auth Service
Sync Pub Service
Adv play
Adv sort
Adv Search
Adv index
 Real time/offline
 Low latency
 More topics
 Massively delayed messages

11. Pulsar Virtual Summit Europe 2021
Real-time and offline data processing
binlog
TDSQL
…… 5 4 3 2 1
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster for real time consumer
Broker-1 Broker-2
Bookie-1 Bookie-2
Cluster for offline consumer
GEO
DBSync

12. Pulsar Virtual Summit Europe 2021
Massively Delayed Messages in Adv
Produce delayed messages to some certain time
forever()
 Massively delayed message
 Unfixed delay time
1-1 1-3 5-1 1-4 1-5 5-2 1-2 5-3
WheelTi
mer/Sub
-1
consumer

13. Pulsar Virtual Summit Europe 2021
Massively Delayed Messages in Adv
Ledger-3 Ledger-5 Ledger-8
producer
Day index
Hour index
Minute index
WheelTim
er
1
2
3
4
5
6
7
…
consumer
Ledger-9 Ledger-11 Ledger-15
Origin topic
Delay topic
Origin
topic

14. Pulsar Virtual Summit Europe 2021
100 Billion + msg/day
70+ Clusters
166w+ Max Qps
Current status of internal operations

15. Pulsar Virtual Summit Europe 2021
Thanks