Query logging with proxy sql

1. Query logging with
ProxySQL

2. Summary
2
This document explains how to use proxysql to leave query logs that users execute
directly by connecting to db.
This document includes the following things:
- How to install proxysql
- How to set up proxysql for query logging
- How to convert binary format query log to text format in proxysql
- How to install ELK stack and set up

3. Architecture
3
BIN TXT
Users
Admin
……
ProxySQL
DB Servers
FileBeat
Logstash
Elasticsearch Kibana

4. Set up ProxySQL 4
# rpm -ivh proxysql-2.0.4-1-centos67.x86_64.rpm
# service proxysql start
# mysql -u admin -padmin -h 127.0.0.1 -P6032 --prompt='Admin> '
Install & Start & Connect proxysql
https://github.com/sysown/proxysql/releases/tag/v2.0.4

5. 5
Set up ProxySQL
-- Adding User
insert into mysql_users(username, password, transaction_persistent)
values('test_user','test1234', 0);
load mysql users to runtime;
save mysql users to disk;
-- Adding Server Information
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.01',0,3306,'customerdb');
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.02',1,3306,'productdb');
insert into mysql_servers(hostname,hostgroup_id,port,comment)
values('10.xx.xx.03',2,3306,'orderdb');
load mysql servers to runtime;
save mysql servers to disk;
Set up configuration

6. 6
Set up ProxySQL
-- Set enable query logging
set mysql-eventslog_filename='queries.log';
set mysql-eventslog_filesize=1048576;
load mysql variables to runtime;
save mysql variables to disk;
insert into mysql_query_rules (rule_id, active, match_digest, log,apply)
values (1,1,'.',1,0);
insert into mysql_query_rules
(rule_id, active, flagIN, match_pattern, re_modifiers, replace_pattern, apply)
values (2,1,0,'n','GLOBAL',' ',0);
load mysql query rules to runtime;
save mysql query rules to disk;
Set up configuration

7. 7
Set up ProxySQL
-- Add query rules for query routing
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* customerdb',0,1);
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* productdb',1,1);
insert into mysql_query_rules(active,match_pattern, destination_hostgroup,apply)
values(1,'^SELECT .* orderdb',2,1);
load mysql query rules to runtime;
save mysql query rules to disk;
Set up configuration

8. 8
Set up ProxySQL
-- You can check to connect db
# mysql -utest_user -p'test1234' -h10.xx.xx.001 -P6033 -e"SELECT * from customerdb.member limit 10"
# mysql -utest_user -p'test1234' -h10.xx.xx.001 -P6033 -e"SELECT * from orderdb.order limit 10"
-- You can find a query log file(queries.log.0000000x)
# cd /var/lib/proxysql
# ls -al
total 7876
drwxr-xr-x 4 proxysql proxysql 4096 Jun 20 14:47 .
drwxr-xr-x. 27 root root 4096 Jun 19 18:26 ..
-rw------- 1 proxysql proxysql 2177193 Jun 20 14:47 proxysql.log
-rw-r--r-- 1 proxysql proxysql 6 Jun 20 11:18 proxysql.pid
-rw------- 1 proxysql proxysql 2781184 Jun 20 14:47 proxysql_stats.db
-rw------- 1 proxysql proxysql 89474 Jun 20 14:19 queries.log.00000001
Check query logging

9. 9
Convert query log
ProxySQL leaves a query log in binary format.
We need to convert it to text format.
ProxySQL supports tool that convert binary format file to text
format.(eventslog_reader_sample)
# git clone https://github.com/sysown/proxysql.git
# cd proxysql/tools
# ll
total 128
-rwxr-xr-x 1 root root 2896 Jun 11 10:28 check_variables.pl
-rw-r--r-- 1 root root 4527 Jun 11 10:28 eventslog_reader_sample.cpp
-rw-r--r-- 1 root root 122 Jun 11 10:28 Makefile
-rwxr-xr-x 1 root root 16869 Jun 11 10:28 proxysql_galera_checker.sh
-rwxr-xr-x 1 root root 3574 Jun 11 10:28 proxysql_galera_writer.pl

10. 10
Convert query log
You can compile eventslog_reader_sample.cpp file.
And you can convert binary query log file to text format.
# make
# ll
total 128
-rwxr-xr-x 1 root root 2896 Jun 11 10:28 check_variables.pl
-rwxr-xr-x 1 root root 87168 Jun 11 10:41 eventslog_reader_sample
-rw-r--r-- 1 root root 4527 Jun 11 10:28 eventslog_reader_sample.cpp
-rw-r--r-- 1 root root 122 Jun 11 10:28 Makefile
-rwxr-xr-x 1 root root 16869 Jun 11 10:28 proxysql_galera_checker.sh
-rwxr-xr-x 1 root root 3574 Jun 11 10:28 proxysql_galera_writer.pl
# ./eventslog_reader_sample /var/lib/proxysql/queries.log.00000001
or
# ./eventslog_reader_sample /var/lib/proxysql/queries.log.00000001 > result.txt

11. 11
Convert query log
You can see two types of query logs, binary and text.
# tail -n 10 queries.log.00000001
��:�L14SHOW SESSION VARIABLES LIKE 'lower_case_table_names'w�� test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306�KSw/���RSELECT current_user()x��
test_userinformation_schema10.xxx.xxx.xx:1475310.xx.xx.xx:3306��cw�X��Efw�X��
��l��SET CHARACTER SET utf8g�� test_userinformation_schema10.xxx.xxx.xxx:14753�����������pw�X���pw�X����8���Q0.xxx.x:14753x��
test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306�K~w�X���w�X���$�f��SET SQL_SAFE_UPDATES=1o��
test_userinformation_schema10.xxx.xxx.x:14753����������‫ۏ‬w�X��‫ۏ‬w�X��yS��g�H�0.xxx.x:14753ION_ID()���
test_userinformation_schema10.xxx.xxx.x:1475310.xx.xx.xx:3306���w�X��h�w�X��Y�Z�~a��%SHOWorderdb10.xxx.xxx.x:14753����������ӹw�X��ӹw�X��0�o�/0.xxx.x:14753=1
# tail -n 10 queries_result.log.00000001
ProxySQL LOG QUERY: thread_id="30" username="test_user" schemaname="information_schema" client="10.xxx.xxx.x:63285" HID=0 server="10.xx.xx.xxx:3306" starttime="2019-06-20 17:35:20.108808" endtime="2019-06-20
17:35:20.109038" duration=230us digest="0xD8CCF296B01AC933"
select *
from orderdb.order
limit 105
ProxySQL LOG QUERY: thread_id="30" username="test_user" schemaname="information_schema" client="10.xxx.xxx.x:63285" HID=0 server="10.xx.xx.xxx:3306" starttime="2019-06-20 17:35:21.300584" endtime="2019-06-20
17:35:21.300738" duration=154us digest="0xD8CCF296B01AC933"
select *
from customerdb.custom
limit 100

12. 12
Make cron job for converting query log file
Query log file will be created sequentially, we need a script for converting binary query
log file to text format file whenever new query log file is created.
BIN TXTqueries.log.00000001 ./result/queries_result.log.00000001
./eventslog_reader_sample queries.log.00000001 > ./result/queries_result.log.00000001
BIN TXTqueries.log.00000002 ./result/queries_result.log.00000002
BIN TXTqueries.log.00000003 ./result/queries_result.log.00000003

13. 13
How user uses with workbench tool
The user who uses to connect directly db can use like this with workbench.
User has to use table name with database name(schema name) in query.
(ex. orderdb.order)

14. 14
Set up FileBeat
Download
# tar -xzvf filebeat-7.1.1-linux-x86_64.tar.gz -C /opt/
# vi filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/proxysql/result/queries_result.log.*
multiline.pattern: '^ProxySQL'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["localhost:5044"]
https://www.elastic.co/kr/downloads/beats/filebeat

15. 15
Set up Logstash
Download
# tar -xzvf logstash-7.1.1.tar.gz -C /opt/
# cd config
# vi logstash.conf
input {
beats {
port => 5044
}
}
---continue behind
https://www.elastic.co/kr/downloads/logstash

16. 16
Set up Logstash
# vi logstash.conf
filter {
mutate {
gsub => ["message",""",""]
}
mutate {
gsub => ["message",". ",".0"]
}
grok {
match => {"message" => "ProxySQL LOG QUERY: thread_id=%{GREEDYDATA:thread_id} username=%{GREEDYDATA:username}
schemaname=%{GREEDYDATA:schemaname} client=%{GREEDYDATA:client} HID=%{GREEDYDATA:HID} server=%{GREEDYDATA:server}
starttime=%{TIMESTAMP_ISO8601:starttime} endtime=%{TIMESTAMP_ISO8601:endtime} duration=%{GREEDYDATA:duration}
digest=%{DATA:digest}n%{GREEDYDATA:query}"}
}
}
---continue behind

17. 17
Set up Logstash
# vi logstash.conf
output {
elasticsearch {
hosts => ["http://[elasticsearch_server_ip]:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

18. 18
Set up Elasticsearch
Download
# tar -xzvf elasticsearch-7.1.1-linux-x86_64.tar.gz -C /opt/
# cd config
# vi elasticsearch.yml
cluster.name: db-admin
node.name: first-node
path.data: /elasticsearch/data
path.logs: /elasticsearch/log
bootstrap.system_call_filter: false
network.host: [elasticsearch_server_ip]
http.port: 9200
cluster.initial_master_nodes: "first-node"
https://www.elastic.co/kr/downloads/elasticsearch

19. 19
Set up Kibana
Download
# tar -xzvf kibana-7.1.1-linux-x86_64.tar.gz -C /opt/
# cd config
# vi kibana.yml
server.host: "[kibana_server_ip]"
elasticsearch.hosts: ["http://[elasticsearch_server_ip]:9200"]
https://www.elastic.co/kr/downloads/kibana

20. 20
-- Start Filebeat
# cd /opt/filebeat-7.1.1-linux-x86_64
# ./filebeat -e -c filebeat.yml
-- Start Logstash
# cd /opt/logstash-7.1.1
# bin/logstash -f config/logstash.conf
-- Start Elasticsearch
# cd /opt/elasticsearch-7.1.1
# bin/elasticsearch
-- Start Kibana
# cd /opt/kibana-7.1.1-linux-x86_64
# bin/kibana
Start ELK stack

21. Kibana Web 21
Confirm the query logs in kibana web page
http://kibana_server_ip:5601

22. Reference 22
https://github.com/sysown/proxysql/wiki/Query-Logging
https://github.com/sysown/proxysql/issues/871

23. Thank You