迁移 MySQL 数据到 OceanBase 集群
👉 本文共约2694个字，系统预计阅读时间或需11分钟。
- OceanBase Docker安装体验：https://www.xmmup.com/oceanbase-dockeranzhuangtiyan.html
- 手动部署 OceanBase 单副本集群：https://www.xmmup.com/shoudongbushu-oceanbase-danfubenjiqun.html
- 手动部署 OceanBase 三副本集群（在同一个节点）：https://www.xmmup.com/shoudongbushu-oceanbase-sanfubenjiqunzaitongyigejiedian.html
- 使用OBD部署一个OceanBase 三副本集群并使用Promethues监控OB（在不同节点）：https://www.xmmup.com/shiyongobdbushuyigeoceanbase-sanfubenjiqunbingshiyongpromethuesjiankongobzaibutongjiedian.html
说明
相关连接：
https://open.oceanbase.com/blog/10900159
https://open.oceanbase.com/answer/detail?id=20400030
练习目的
本次练习目的掌握从 MySQL 向 OceanBase 迁移数据的基本方法：mysqldump、datax 、canal 等。
练习条件
- 有 服务器，内存资源至少 12G*1台 ，部署有 OceanBase 集群。
- 有 mysql 数据库，有少量业务表和数据（数据量不要求很多）。没有业务表和数据的就推荐使用 sysbench、 tpcc 或者 tpch 的业务数据。
练习内容
请记录并分享下列内容：
- （必选）使用 mysqldump 将 mysql的表结构和数据同步到 OceanBase 的MySQL 租户中。
- （必选）使用 datax 配置至少一个表的 MySQL 到 OceanBase 的 MySQL 租户的离线同步。
- （可选）使用 datax 配置至少一个表的 OceanBase 到 CSV 以及 CSV 到 OceanBase 的离线同步。
- （可选）使用 canal 配置 MySQL 的增量同步到 OceanBase 的 MySQL 租户。
使用 mysqldump 将 mysql的表结构和数据同步到 OceanBase 的MySQL 租户中
源库：MySQL 5.7.36，IP地址：172.17.0.3，数据库：sbtest，10张表，每张表10万行数据
目标库：OB 3.1.1，IP地址：172.17.0.2，数据库：sbtest，租户：obmysql
使用sysbench给mysql环境生成数据
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | [root@docker35 ~]# mysql -uroot -plhr -h172.17.0.3 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.36-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> create database sbtest; Query OK, 1 row affected (0.06 sec) Bye [root@docker35 ~]# sysbench /usr/share/sysbench/oltp_common.lua --time=300 --mysql-host=172.17.0.3 --mysql-port=3306 --mysql-user=root --mysql-password=lhr --mysql-db=sbtest --table-size=100000 --tables=10 --threads=16 --events=999999999 prepare sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest8'... Creating table 'sbtest1'... Creating table 'sbtest7'... Creating table 'sbtest6'... Creating table 'sbtest5'... Creating table 'sbtest3'... Creating table 'sbtest9'... Creating table 'sbtest4'... Creating table 'sbtest10'... Creating table 'sbtest2'... Inserting 100000 records into 'sbtest7' Inserting 100000 records into 'sbtest2' Inserting 100000 records into 'sbtest3' Inserting 100000 records into 'sbtest9' Inserting 100000 records into 'sbtest5' Inserting 100000 records into 'sbtest6' Inserting 100000 records into 'sbtest10' Inserting 100000 records into 'sbtest1' Inserting 100000 records into 'sbtest4' Inserting 100000 records into 'sbtest8' Creating a secondary index on 'sbtest2'... Creating a secondary index on 'sbtest7'... Creating a secondary index on 'sbtest3'... Creating a secondary index on 'sbtest9'... Creating a secondary index on 'sbtest10'... Creating a secondary index on 'sbtest5'... Creating a secondary index on 'sbtest6'... Creating a secondary index on 'sbtest8'... Creating a secondary index on 'sbtest1'... Creating a secondary index on 'sbtest4'... [root@docker35 ~]# mysql -uroot -plhr -h172.17.0.3 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 36 Server version: 5.7.36-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use sbtest; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +------------------+ | Tables_in_sbtest | +------------------+ | sbtest1 | | sbtest10 | | sbtest2 | | sbtest3 | | sbtest4 | | sbtest5 | | sbtest6 | | sbtest7 | | sbtest8 | | sbtest9 | +------------------+ 10 rows in set (0.00 sec) mysql> select count(*) from sbtest1; +----------+ | count(*) | +----------+ | 100000 | +----------+ 1 row in set (0.04 sec) mysql> SELECT TABLE_NAME,DATA_LENGTH,INDEX_LENGTH,(DATA_LENGTH+INDEX_LENGTH) as length,TABLE_ROWS,concat(round((DATA_LENGTH+INDEX_LENGTH)/1024/1024,2), 'MB') as total_size -> FROM information_schema.TABLES -> WHERE TABLE_SCHEMA='sbtest' -> order by length desc -> ; +------------+-------------+--------------+----------+------------+------------+ | TABLE_NAME | DATA_LENGTH | INDEX_LENGTH | length | TABLE_ROWS | total_size | +------------+-------------+--------------+----------+------------+------------+ | sbtest6 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest5 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest4 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest3 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest2 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest10 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest1 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest9 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest8 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest7 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | +------------+-------------+--------------+----------+------------+------------+ 10 rows in set (0.01 sec) |
使用mysqldump导出
1 2 3 4 5 6 7 8 9 | [root@docker35 soft]# mysqldump -uroot -plhr -h172.17.0.3 -P3306 --single-transaction --hex-blob --routines --events --triggers --master-data=2 --set-gtid-purged=OFF --databases sbtest --default-character-set=utf8 --max_allowed_packet=512M > sbtest.sql mysqldump: [Warning] Using a password on the command line interface can be insecure. [root@docker35 soft]# ll -h total 315M -rw-r--r-- 1 root root 7.9M Nov 5 09:10 obproxy-3.2.0-1.el7.x86_64.rpm -rw-r--r-- 1 root root 47M Dec 31 17:57 oceanbase-ce-3.1.2-10000392021123010.el7.x86_64.rpm -rw-r--r-- 1 root root 156K Dec 31 19:40 oceanbase-ce-libs-3.1.2-10000392021123010.el7.x86_64.rpm -rw-r--r-- 1 root root 72M Dec 18 06:25 prometheus-2.32.1.linux-amd64.tar.gz -rw-r--r-- 1 root root 189M Jan 10 15:38 sbtest.sql |
使用source命令导入数据
OB的环境参考：https://www.xmmup.com/oceanbase-dockeranzhuangtiyan.html
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | [admin@0209306f10bf tmp]$ obclient -h 127.1 -uroot@obmysql#obce-single -P2883 -p -c -A test Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 10 Server version: 5.6.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:52:05) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [test]> show databases; +--------------------+ | Database | +--------------------+ | oceanbase | | information_schema | | mysql | | test | | lhrdb | +--------------------+ 5 rows in set (0.005 sec) MySQL [test]> source /tmp/sbtest.sql Query OK, 0 rows affected (0.001 sec) ERROR 1064 (42000) at line 2 in file: '/tmp/sbtest.sql': You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near '--' at line 1 Query OK, 0 rows affected (0.001 sec) Query OK, 5258 rows affected (0.478 sec) Records: 5258 Duplicates: 0 Warnings: 0 ..... Query OK, 0 rows affected (0.001 sec) Query OK, 5258 rows affected (0.456 sec) Records: 5258 Duplicates: 0 Warnings: 0 Query OK, 5258 rows affected (0.478 sec) Records: 5258 Duplicates: 0 Warnings: 0 MySQL [sbtest]> MySQL [sbtest]> MySQL [sbtest]> show databases; +--------------------+ | Database | +--------------------+ | oceanbase | | information_schema | | mysql | | test | | lhrdb | | sbtest | +--------------------+ 6 rows in set (0.006 sec) MySQL [sbtest]> use sbtest; Database changed MySQL [sbtest]> show tables; +------------------+ | Tables_in_sbtest | +------------------+ | sbtest1 | | sbtest10 | | sbtest2 | | sbtest3 | | sbtest4 | | sbtest5 | | sbtest6 | | sbtest7 | | sbtest8 | | sbtest9 | +------------------+ 10 rows in set (0.004 sec) MySQL [sbtest]> select count(1) from sbtest1; +----------+ | count(1) | +----------+ | 100000 | +----------+ 1 row in set (0.375 sec) |
使用datax (离线)从 MySQL 同步表数据到 OceanBase
使用 datax 配置至少一个表的 MySQL 到 OceanBase 的 MySQL 租户的离线同步。
部署DATAX软件
有关dataX更多内容请参考：https://www.xmmup.com/alikaiyuanetlgongjuzhidataxhedatax-webjieshao.html
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | -- 大小：820MB wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz tar -xf datax.tar.gz -C /usr/local/ cd /usr/local/datax # 删除datax中的隐藏文件 find /usr/local/datax/plugin -name ".*" | xargs rm -f yum install -y python2 java -- 生成模板文件 python2 /usr/local/datax/bin/datax.py -r mysqlreader -w oceanbasev10writer > /usr/local/datax/job/my2ob.json # 根据实际环境修改 vi /usr/local/datax/job/my2ob.json |
修改同步作业文件
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | cat > /usr/local/datax/job/my2ob.json <<"EOF" { "job": { "setting": { "speed": { "channel": 2, }, "errorLimit": { "record": 10 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "root", "password": "lhr", "splitPk": "id", "column": ["*"], "connection": [ { "jdbcUrl": [ "jdbc:mysql://172.17.0.3:3306/sbtest" ], "table": [ "sbtest1" ] } ] } }, "writer": { "name": "oceanbasev10writer", "parameter": { "writeMode": "insert", "username": "root", "password": "lhr", "writerThreadCount": 5, "column": [ "*" ], "connection": [ { "jdbcUrl": "||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table": [ "sbtest1" ] } ] } } } ] } } EOF |
启动同步作业
1 | python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/my2ob.json |
日志：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 | [admin@0209306f10bf ~]$ python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/my2ob.json DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2022-01-10 09:04:59.884 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2022-01-10 09:04:59.898 [main] INFO Engine - the machine info => osInfo: Red Hat, Inc. 1.8 25.312-b07 jvmInfo: Linux amd64 3.10.0-1127.10.1.el7.x86_64 cpu num: 16 totalPhysicalMemory: -0.00G freePhysicalMemory: -0.00G maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size PS Eden Space | 256.00MB | 256.00MB Code Cache | 240.00MB | 2.44MB Compressed Class Space | 1,024.00MB | 0.00MB PS Survivor Space | 42.50MB | 42.50MB PS Old Gen | 683.00MB | 683.00MB Metaspace | -0.00MB | 0.00MB 2022-01-10 09:04:59.935 [main] INFO Engine - { "content":[ { "reader":{ "name":"mysqlreader", "parameter":{ "column":[ "*" ], "connection":[ { "jdbcUrl":[ "jdbc:mysql://172.17.0.3:3306/sbtest" ], "table":[ "sbtest1" ] } ], "password":"***", "splitPk":"id", "username":"root" } }, "writer":{ "name":"oceanbasev10writer", "parameter":{ "column":[ "*" ], "connection":[ { "jdbcUrl":"||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table":[ "sbtest1" ] } ], "password":"***", "username":"root", "writeMode":"insert", "writerThreadCount":5 } } } ], "setting":{ "errorLimit":{ "record":10 }, "speed":{ "channel":2 } } } 2022-01-10 09:04:59.963 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2022-01-10 09:04:59.969 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2022-01-10 09:04:59.994 [main] INFO JobContainer - DataX jobContainer starts job. 2022-01-10 09:04:59.999 [main] INFO JobContainer - Set jobId = 0 2022-01-10 09:05:00.632 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. 2022-01-10 09:05:00.634 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改. 2022-01-10 09:05:00.659 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:00.686 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8 2022-01-10 09:05:01.073 [job-0] INFO DbUtils - value for query [SHOW VARIABLES LIKE 'ob_compatibility_mode'] is [MYSQL] 2022-01-10 09:05:01.086 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.117 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.136 [job-0] INFO OriginalConfPretreatmentUtil - table:[sbtest1] all columns:[ id,k,c,pad ]. 2022-01-10 09:05:01.164 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改. 2022-01-10 09:05:01.167 [job-0] INFO OriginalConfPretreatmentUtil - Write data [ insert INTO %s (id,k,c,pad) VALUES(?,?,?,?) ], which jdbcUrl like:[||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true] 2022-01-10 09:05:01.168 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2022-01-10 09:05:01.169 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work . 2022-01-10 09:05:01.169 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do prepare work . 2022-01-10 09:05:01.171 [job-0] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.171 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.191 [job-0] INFO DbUtils - value for query [show variables like 'version'] is [3.1.1] 2022-01-10 09:05:01.206 [job-0] INFO JobContainer - jobContainer starts to do split ... 2022-01-10 09:05:01.206 [job-0] INFO JobContainer - Job set Channel-Number to 2 channels. 2022-01-10 09:05:01.226 [job-0] INFO SingleTableSplitUtil - split pk [sql=SELECT MIN(id),MAX(id) FROM sbtest1] is running... 2022-01-10 09:05:01.256 [job-0] INFO SingleTableSplitUtil - After split(), allQuerySql=[ select * from sbtest1 where (1 <= id AND id < 10001) select * from sbtest1 where (10001 <= id AND id < 20001) select * from sbtest1 where (20001 <= id AND id < 30001) select * from sbtest1 where (30001 <= id AND id < 40001) select * from sbtest1 where (40001 <= id AND id < 50001) select * from sbtest1 where (50001 <= id AND id < 60001) select * from sbtest1 where (60001 <= id AND id < 70001) select * from sbtest1 where (70001 <= id AND id < 80001) select * from sbtest1 where (80001 <= id AND id < 90001) select * from sbtest1 where (90001 <= id AND id <= 100000) select * from sbtest1 where id IS NULL ]. 2022-01-10 09:05:01.256 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [11] tasks. 2022-01-10 09:05:01.260 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] splits to [11] tasks. 2022-01-10 09:05:01.300 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2022-01-10 09:05:01.315 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2022-01-10 09:05:01.320 [job-0] INFO JobContainer - Running by standalone Mode. 2022-01-10 09:05:01.337 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [2] channels for [11] tasks. 2022-01-10 09:05:01.358 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated. 2022-01-10 09:05:01.358 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2022-01-10 09:05:01.373 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2022-01-10 09:05:01.378 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started 2022-01-10 09:05:01.379 [0-0-0-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:01.379 [0-0-1-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:01.380 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (1 <= id AND id < 10001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:01.380 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (10001 <= id AND id < 20001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:01.382 [0-0-1-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:01.382 [0-0-0-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:01.403 [0-0-1-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:01.431 [0-0-0-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:01.449 [0-0-1-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:01.455 [0-0-1-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:01.456 [0-0-1-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:01.464 [0-0-0-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:01.464 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.464 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.469 [0-0-0-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:01.470 [0-0-0-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:01.482 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.482 [0-0-1-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:01.483 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.483 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:01.500 [0-0-0-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:01.502 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:01.512 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:01.527 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:01.543 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:01.562 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:01.575 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:01.589 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:01.615 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:01.627 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:01.659 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.659 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.673 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.673 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.677 [0-0-1-writer] INFO ColumnMetaCache - fetch columnMeta of table sbtest1 success 2022-01-10 09:05:01.773 [0-0-1-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:01.813 [0-0-0-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:02.043 [0-0-1-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (10001 <= id AND id < 20001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.131 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (1 <= id AND id < 10001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.186 [0-0-0-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:02.260 [0-0-1-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:02.781 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1409]ms 2022-01-10 09:05:02.782 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[1405]ms 2022-01-10 09:05:02.784 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started 2022-01-10 09:05:02.784 [0-0-2-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:02.785 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (20001 <= id AND id < 30001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.785 [0-0-2-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:02.786 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started 2022-01-10 09:05:02.812 [0-0-3-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:02.812 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (30001 <= id AND id < 40001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.813 [0-0-3-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:02.820 [0-0-2-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:02.837 [0-0-3-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:02.845 [0-0-2-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:02.849 [0-0-2-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:02.850 [0-0-2-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:02.854 [0-0-3-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:02.854 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.854 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.857 [0-0-3-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:02.858 [0-0-3-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:02.863 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.863 [0-0-2-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:02.863 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.863 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:02.874 [0-0-3-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:02.874 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:02.884 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:02.893 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:02.917 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:02.925 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:02.933 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:02.941 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:02.949 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:02.957 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:02.972 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.973 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.980 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.980 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.000 [0-0-2-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.011 [0-0-3-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.091 [0-0-2-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (20001 <= id AND id < 30001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.099 [0-0-3-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (30001 <= id AND id < 40001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.117 [0-0-2-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:03.126 [0-0-3-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:03.589 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[805]ms 2022-01-10 09:05:03.592 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started 2022-01-10 09:05:03.592 [0-0-4-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:03.593 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (40001 <= id AND id < 50001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.627 [0-0-4-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:03.639 [0-0-4-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:03.647 [0-0-4-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:03.651 [0-0-4-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:03.652 [0-0-4-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:03.653 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.653 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.661 [0-0-4-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:03.661 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:03.669 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:03.690 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:03.699 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:03.706 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:03.725 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.726 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.726 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[940]ms 2022-01-10 09:05:03.729 [0-0-5-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:03.729 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started 2022-01-10 09:05:03.729 [0-0-5-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (50001 <= id AND id < 60001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.730 [0-0-5-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:03.753 [0-0-5-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:03.764 [0-0-4-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.768 [0-0-5-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:03.772 [0-0-5-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:03.772 [0-0-5-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:03.773 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.774 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.790 [0-0-5-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:03.790 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:03.810 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:03.839 [0-0-4-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (40001 <= id AND id < 50001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.848 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:03.857 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:03.866 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:03.875 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.876 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.906 [0-0-5-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.034 [0-0-5-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (50001 <= id AND id < 60001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.051 [0-0-5-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.061 [0-0-4-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.531 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[939]ms 2022-01-10 09:05:04.531 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[803]ms 2022-01-10 09:05:04.533 [0-0-6-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:04.533 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started 2022-01-10 09:05:04.534 [0-0-6-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (60001 <= id AND id < 70001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.534 [0-0-6-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:04.537 [0-0-7-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:04.537 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started 2022-01-10 09:05:04.537 [0-0-7-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (70001 <= id AND id < 80001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.537 [0-0-7-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:04.545 [0-0-6-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:04.554 [0-0-7-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:04.566 [0-0-6-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:04.571 [0-0-6-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:04.572 [0-0-6-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:04.576 [0-0-7-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:04.576 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.576 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.580 [0-0-7-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:04.580 [0-0-7-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:04.585 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.585 [0-0-6-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:04.585 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:04.585 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.595 [0-0-7-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:04.595 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:04.603 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:04.612 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:04.620 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:04.628 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:04.636 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:04.643 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:04.651 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:04.659 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:04.673 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.673 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.683 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.683 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.711 [0-0-6-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.719 [0-0-7-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.811 [0-0-6-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (60001 <= id AND id < 70001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.822 [0-0-7-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (70001 <= id AND id < 80001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.834 [0-0-6-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.846 [0-0-7-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:05.339 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[803]ms 2022-01-10 09:05:05.343 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started 2022-01-10 09:05:05.343 [0-0-8-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (80001 <= id AND id < 90001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.343 [0-0-8-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:05.344 [0-0-8-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:05.352 [0-0-8-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:05.360 [0-0-8-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:05.364 [0-0-8-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:05.365 [0-0-8-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:05.370 [0-0-8-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.370 [0-0-8-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.381 [0-0-8-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:05.381 [0-0-8-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:05.389 [0-0-8-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:05.398 [0-0-8-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:05.408 [0-0-8-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:05.416 [0-0-8-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:05.423 [0-0-8-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.423 [0-0-8-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.444 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[910]ms 2022-01-10 09:05:05.446 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] attemptCount[1] is started 2022-01-10 09:05:05.446 [0-0-9-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:05.446 [0-0-9-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (90001 <= id AND id <= 100000) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.447 [0-0-9-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:05.452 [0-0-8-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:05.456 [0-0-9-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:05.502 [0-0-9-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:05.506 [0-0-9-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:05.506 [0-0-9-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:05.510 [0-0-9-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.510 [0-0-9-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.524 [0-0-9-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:05.524 [0-0-9-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:05.540 [0-0-9-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:05.552 [0-0-8-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (80001 <= id AND id < 90001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.560 [0-0-9-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:05.568 [0-0-8-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:05.579 [0-0-9-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:05.601 [0-0-9-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:05.608 [0-0-9-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.608 [0-0-9-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.837 [0-0-9-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:06.462 [0-0-9-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (90001 <= id AND id <= 100000) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.483 [0-0-9-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:06.827 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] is successed, used[1485]ms 2022-01-10 09:05:06.831 [0-0-10-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:06.831 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] attemptCount[1] is started 2022-01-10 09:05:06.831 [0-0-10-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where id IS NULL ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.832 [0-0-10-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:06.928 [0-0-10-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where id IS NULL ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.928 [0-0-10-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:07.015 [0-0-10-writer] INFO CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:07.038 [0-0-10-writer] INFO CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:07.038 [0-0-10-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:07.039 [0-0-10-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:07.040 [0-0-10-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:07.139 [0-0-10-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:07.139 [0-0-10-writer] INFO ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:07.216 [0-0-10-writer] INFO ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:07.247 [0-0-10-writer] INFO ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:07.317 [0-0-10-writer] INFO ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:07.356 [0-0-10-writer] INFO ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:07.366 [0-0-10-writer] INFO DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:07.366 [0-0-10-writer] INFO DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:07.375 [0-0-10-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 0, finished = 0 2022-01-10 09:05:07.469 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] is successed, used[639]ms 2022-01-10 09:05:07.573 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] is successed, used[2127]ms 2022-01-10 09:05:07.574 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2022-01-10 09:05:11.356 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 18788895 bytes | Speed 1.79MB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 3.153s | All Task WaitReaderTime 0.719s | Percentage 100.00% 2022-01-10 09:05:11.356 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks. 2022-01-10 09:05:11.356 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do post work. 2022-01-10 09:05:11.356 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work. 2022-01-10 09:05:11.356 [job-0] INFO JobContainer - DataX jobId [0] completed successfully. 2022-01-10 09:05:11.357 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook 2022-01-10 09:05:11.358 [job-0] INFO JobContainer - [total cpu info] => averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s PS Scavenge | 3 | 3 | 3 | 0.237s | 0.237s | 0.237s 2022-01-10 09:05:11.358 [job-0] INFO JobContainer - PerfTrace not enable! 2022-01-10 09:05:11.359 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 18788895 bytes | Speed 1.79MB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 3.153s | All Task WaitReaderTime 0.719s | Percentage 100.00% 2022-01-10 09:05:11.360 [job-0] INFO JobContainer - 任务启动时刻 : 2022-01-10 09:04:59 任务结束时刻 : 2022-01-10 09:05:11 任务总计耗时 : 11s 任务平均流量 : 1.79MB/s 记录写入速度 : 10000rec/s 读出记录总数 : 100000 读写失败总数 : 0 |
datax 实现 OceanBase 以 csv 格式的导入导出
OceanBase 以 csv 格式导出数据
生成 datax 配置 json 模板
1 | python2 /usr/local/datax/bin/datax.py -r oceanbasev10reader -w txtfilewriter > /usr/local/datax/job/oceanbase2csv.json |
修改 json 文件：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | cat > /usr/local/datax/job/oceanbase2csv.json <<"EOF" { "job": { "setting": { "speed": { "channel": 4 }, "errorLimit": { "record": 0, "percentage": 0.1 } }, "content": [ { "reader": { "name": "oceanbasev10reader", "parameter": { "column": ["*"], "connection": [ { "jdbcUrl": ["||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8"], "table": ["sbtest1"] } ], "password": "lhr", "username": "root", "where": "" } }, "writer": { "name": "txtfilewriter", "parameter": { "dateFormat": "yyyy-MM-dd hh:mm:ss", "fieldDelimiter": ",", "fileName": "ob2csv", "path": "/tmp/", "writeMode": "truncate" } } } ] } } EOF |
执行任务：
1 | python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/oceanbase2csv.json |
查看结果：
1 2 3 4 5 6 7 8 9 10 11 | [admin@0209306f10bf tmp]$ ll total 19M drwxr-xr-x 2 admin admin 4.0K Jan 10 18:07 hsperfdata_admin drwxr-xr-x 2 root root 4.0K Jan 10 16:22 hsperfdata_root -rw-rw-r-- 1 admin admin 19M Jan 10 18:07 ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 [admin@0209306f10bf tmp]$ more ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 | wc -l 100000 [admin@0209306f10bf tmp]$ head -n 2 ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 1,50147,68487932199-96439406143-93774651418-41631865787-96406072701-20604855487-25459966574-28203206787-41238978918-19503783441,22195207048-70116052123-74140395089-76317954521-98694025897 2,50248,13241531885-45658403807-79170748828-69419634012-13605813761-77983377181-01582588137-21344716829-87370944992-02457486289,28733802923-10548894641-11867531929-71265603657-36546888392 [admin@0209306f10bf tmp]$ |
OceanBase 以 csv 格式导入数据
生成 datax 配置 json 模板
1 | python2 /usr/local/datax/bin/datax.py -r oceanbasev10reader -w txtfilewriter > /usr/local/datax/job/csv2oceanbase.json |
修改json文件：
本人提供Oracle(OCP、OCM)、MySQL(OCP)、PostgreSQL(PGCA、PGCE、PGCM)等数据库的培训和考证业务，私聊QQ646634621或微信db_bao，谢谢！
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | cat > /usr/local/datax/job/csv2oceanbase.json <<"EOF" { "job": { "setting": { "speed": { "channel": 4 }, "errorLimit": { "record": 0, "percentage": 0.1 } }, "content": [ { "reader": { "name": "txtfilereader", "parameter": { "path": ["/tmp/ob/"], "fileName": "ob2csv", "encoding": "UTF-8", "column": ["*"], "dateFormat": "yyyy-MM-dd hh:mm:ss" , "nullFormat": "\\N" , "fieldDelimiter": "," } }, "writer": { "name": "oceanbasev10writer", "parameter": { "obWriteMode": "insert", "column": ["*"], "preSql": [ "truncate table sbtest11" ], "connection": [ { "jdbcUrl": "||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table": [ "sbtest11" ] } ], "username": "root", "password": "lhr", "writerThreadCount": 10, "batchSize": 100, "memstoreThreshold": "0.9" } } } ] } } EOF |
执行导入操作：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | MySQL [sbtest]> create table sbtest11 like sbtest1; Query OK, 0 rows affected (0.58 sec) MySQL [sbtest]> desc sbtest11; +-------+-----------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------+-----------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | k | int(11) | NO | MUL | 0 | | | c | char(120) | NO | | | | | pad | char(60) | NO | | | | +-------+-----------+------+-----+---------+----------------+ 4 rows in set (0.06 sec) MySQL [sbtest]> select count(*) from sbtest11; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.05 sec) -- 导入 python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/csv2oceanbase.json |
结果：
OB数据库查询：
1 2 3 4 5 6 7 | MySQL [sbtest]> select count(*) from sbtest11; +----------+ | count(*) | +----------+ | 100000 | +----------+ 1 row in set (0.32 sec) |
- 注意：
Path：需要给定文件的路径，该路径下不要有其他文件，执行会报错。
Canal 实现 mysql 与 OceanBase 的增量同步
使用 canal 配置 MySQL 的增量同步到 OceanBase 的 MySQL 租户。
参考：https://www.modb.pro/db/222647
https://open.oceanbase.com/blog/10900160
mysql源端开启二进制日志
1 2 3 4 | [mysqld] log-bin=mysql-bin # 开启 binlog binlog-format=ROW # 选择 ROW 模式 server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复 |
部署 canal admin
参考：https://www.xmmup.com/alishujutongbugongjuotterhecanaljianjie.html
1 2 3 4 5 6 | wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.admin-1.1.5.tar.gz tar -zxvf canal.admin-1.1.5.tar.gz -C /usr/local/canal-admin -- 修改配置文件： cd /usr/local/canal-admin/conf vim application.yml |
修改元数据库信息：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [admin@0209306f10bf conf]$ cat application.yml server: port: 8089 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 spring.datasource: address: 172.17.0.3:3306 database: canal_manager username: root password: lhr driver-class-name: com.mysql.jdbc.Driver url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false hikari: maximum-pool-size: 30 minimum-idle: 1 canal: adminUser: admin adminPasswd: admin |
初始化元数据：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | mysql -uroot -P3306 -plhr -h 172.17.0.3 source conf/canal_manager.sql -- 查看表 MySQL [sbtest]> use canal_manager; Database changed MySQL [canal_manager]> show tables; +-------------------------+ | Tables_in_canal_manager | +-------------------------+ | canal_adapter_config | | canal_cluster | | canal_config | | canal_instance_config | | canal_node_server | | canal_user | +-------------------------+ 6 rows in set (0.05 sec) |
启动canal
1 | cd /usr/local/canal-admin/bin/ && ./startup.sh |
过程：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | [admin@0209306f10bf local]$ cd /usr/local/canal-admin/bin/ && ./startup.sh cd to /usr/local/canal-admin/bin for workaround relative path CLASSPATH :/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: cd to /usr/local/canal-admin/bin for continue [admin@0209306f10bf bin]$ cd /usr/local/canal-admin/bin/ [admin@0209306f10bf bin]$ ll total 20K -rw-rw-r-- 1 admin admin 6 Jan 10 18:46 admin.pid -rwxr-xr-x 1 admin admin 205 Apr 19 2021 restart.sh -rwxr-xr-x 1 admin admin 747 Apr 19 2021 startup.bat -rwxr-xr-x 1 admin admin 2.0K Apr 19 2021 startup.sh -rwxr-xr-x 1 admin admin 1.4K Apr 19 2021 stop.sh [admin@0209306f10bf bin]$ netstat -tulnp | grep 8089 tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 28703/java [admin@0209306f10bf bin]$ ps -ef|grep java admin 28703 0 66 18:46 pts/1 00:01:01 /usr/bin/java -server -Xms1024m -Xmx1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: com.alibaba.otter.canal.admin.CanalAdminApplication admin 29139 23 0 18:47 pts/1 00:00:00 grep --color=auto java |
查看 canal-admin 的 web 服务:
http://172.17.0.2:8089 admin/123456
这里，需要在图形界面修改密码为admin123 ,然后在mysql中查询：
1 2 3 4 5 6 7 | MySQL [canal_manager]> select * from canal_user; +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ | id | username | password | name | roles | introduction | avatar | creation_date | +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ | 1 | admin | 01a6717b58ff5c7eafff6cb7c96f7428ea65fe4c | Canal Manager | admin | NULL | NULL | 2022-01-10 19:25:50 | +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ 1 row in set (0.23 sec) |
部署canal deployer
下载安装canal deployer并解压
1 2 | wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz tar -zxvf canal.deployer-1.1.5.tar.gz -C /usr/local/canal-deployer |
修改配置文件
canal Admin 部署 server 和 instance，配置文件 conf/canal_local.properties 替换 conf/canal.properties。需要修改 conf/canal.properties 里的 manager 地址，其他参数值可以保持默认。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | cd /usr/local/canal-deployer/conf mv canal_local.properties canal.properties && vim canal.properties [admin@0209306f10bf conf]$ cat canal.properties # register ip canal.register.ip = # canal admin config canal.admin.manager = 127.0.0.1:8089 canal.admin.port = 11110 canal.admin.user = admin canal.admin.passwd = 01a6717b58ff5c7eafff6cb7c96f7428ea65fe4c # admin auto register canal.admin.register.auto = true canal.admin.register.cluster = canal.admin.register.name = |
这里是本机部署，需要修改canal.admin.passwd即可。
启动 canal-deployer 服务
1 | cd /usr/local/canal-deployer/bin/ && ./startup.sh |
查看：
1 2 3 4 5 6 7 8 | [admin@0209306f10bf ~]$ ps -ef|grep java admin 3097 0 6 19:27 pts/1 00:00:05 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=otter-canal -Dlogback.configurationFile=/usr/local/canal-deployer/bin/../conf/logback.xml -Dcanal.conf=/usr/local/canal-deployer/bin/../conf/canal.properties -classpath .:/usr/local/canal-deployer/bin/../conf:/usr/local/canal-deployer/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-deployer/bin/../lib/zkclient-0.10.jar:/usr/local/canal-deployer/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_pushgateway-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_httpserver-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_hotspot-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_common-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-deployer/bin/../lib/oro-2.0.8.jar:/usr/local/canal-deployer/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-deployer/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-deployer/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-deployer/bin/../lib/mybatis-spring-2.0.4.jar:/usr/local/canal-deployer/bin/../lib/mybatis-3.5.4.jar:/usr/local/canal-deployer/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-deployer/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-deployer/bin/../lib/jctools-core-2.1.2.jar:/usr/local/canal-deployer/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-deployer/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-deployer/bin/../lib/httpcore-4.4.3.jar:/usr/local/canal-deployer/bin/../lib/httpclient-4.5.1.jar:/usr/local/canal-deployer/bin/../lib/h2-1.4.196.jar:/usr/local/canal-deployer/bin/../lib/guava-22.0.jar:/usr/local/canal-deployer/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-deployer/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-deployer/bin/../lib/druid-1.2.6.jar:/usr/local/canal-deployer/bin/../lib/disruptor-3.4.2.jar:/usr/local/canal-deployer/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-deployer/bin/../lib/commons-lang3-3.7.jar:/usr/local/canal-deployer/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-deployer/bin/../lib/commons-io-2.4.jar:/usr/local/canal-deployer/bin/../lib/commons-compress-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-deployer/bin/../lib/canal.store-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.sink-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.server-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.prometheus-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.driver-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.dbsync-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.meta-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.spring-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.manager-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.filter-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.deployer-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-deployer/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.deployer.CanalLauncher admin 3327 23 0 19:28 pts/1 00:00:00 grep --color=auto java admin 31216 0 4 19:01 pts/1 00:01:11 /usr/bin/java -server -Xms2048m -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: com.alibaba.otter.canal.admin.CanalAdminApplication [admin@0209306f10bf ~]$ netstat -tulnp | grep 11110 tcp 0 0 0.0.0.0:11110 0.0.0.0:* LISTEN 3097/java [admin@0209306f10bf ~]$ netstat -tulnp | grep 8089 tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 31216/java |
通过 canal-admin 的 web 服务配置 canal server 和 canal instance
创建 canal instance 并启动
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ################################################# ## mysql serverId , v1.0.26+ will autoGen canal.instance.mysql.slaveId=57363418 # enable gtid use true/false canal.instance.gtidon=false # position info canal.instance.master.address=172.17.0.3:3306 canal.instance.master.journal.name= canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid= # rds oss binlog canal.instance.rds.accesskey= canal.instance.rds.secretkey= canal.instance.rds.instanceId= # table meta tsdb info canal.instance.tsdb.enable=true #canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb #canal.instance.tsdb.dbUsername=canal #canal.instance.tsdb.dbPassword=canal #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = #canal.instance.standby.gtid= # username/password canal.instance.dbUsername=root canal.instance.dbPassword=lhr canal.instance.connectionCharset = UTF-8 # enable druid Decrypt database password canal.instance.enableDruid=false #canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ== # table regex canal.instance.filter.regex=.*\\..* # table black regex canal.instance.filter.black.regex= # table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2) #canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch # table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2) #canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch # mq config canal.mq.topic=example # dynamic topic route by schema or table regex #canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..* canal.mq.partition=0 # hash partition config #canal.mq.partitionsNum=3 #canal.mq.partitionHash=test.table:id^name,.*\\..* ################################################# |
部署 canal adapter
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.adapter-1.1.5.tar.gz tar -zxvf canal.adapter-1.1.5.tar.gz -C /usr/local/canal-adapter cd /usr/local/canal-adapter/conf/ && vim application.yml cat > /usr/local/canal-adapter/conf/application.yml <<"EOF" server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: consumerProperties: # canal tcp consumer canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: admin canal.tcp.password: 4ACFE3202A5FF5CF467898FC58AAB1D615029441 canalAdapters: - instance: canal1 # canal instance Name or mq topic name groups: - groupId: group1 outerAdapters: - name: logger - name: rdb key: oceanbase1 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://172.17.0.2:2883/sbtest?useUnicode=true jdbc.username: root@sys#obce-single jdbc.password: lhr EOF |
修改RDB 映射文件：conf/rdb/mytest_user.yml
1 2 3 4 5 6 7 8 9 10 11 | cat > /usr/local/canal-adapter/conf/rdb/mytest_user.yml <<"EOF" dataSourceKey: defaultDS destination: canal1 groupId: g1 outerAdapterKey: oceanbase1 concurrent: true dbMapping: mirrorDb: true database: sbtest EOF |
- destination: 指定的是
canal instance名称 - outerAdapterKey: 指定的是application.yml文件中的key
启动 canal adapter 服务
1 | cd /usr/local/canal-adapter/bin/ && ./startup.sh |
进程：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | [admin@0209306f10bf bin]$ ps -ef|grep java admin 1537 0 99 17:32 pts/3 00:00:53 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-adapter -classpath .:/usr/local/canal-adapter/bin/../conf:/usr/local/canal-adapter/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-adapter/bin/../lib/zkclient-0.10.jar:/usr/local/canal-adapter/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-security-crypto-5.0.4.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-cloud-context-2.0.0.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-adapter/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-adapter/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-adapter/bin/../lib/postgresql-42.1.4.jar:/usr/local/canal-adapter/bin/../lib/oro-2.0.8.jar:/usr/local/canal-adapter/bin/../lib/ojdbc6-11.2.0.3.jar:/usr/local/canal-adapter/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-adapter/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-adapter/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-adapter/bin/../lib/mssql-jdbc-7.0.0.jre8.jar:/usr/local/canal-adapter/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-adapter/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-adapter/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/log4j-1.2.17.jar:/usr/local/canal-adapter/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-adapter/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-adapter/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-adapter/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-adapter/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-adapter/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-adapter/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-adapter/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-adapter/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/local/canal-adapter/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-adapter/bin/../lib/hbase-shaded-client-1.1.2.jar:/usr/local/canal-adapter/bin/../lib/guava-22.0.jar:/usr/local/canal-adapter/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/canal-adapter/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-adapter/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-adapter/bin/../lib/druid-1.2.6.jar:/usr/local/canal-adapter/bin/../lib/curator-recipes-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/curator-framework-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/curator-client-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-adapter/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-adapter/bin/../lib/commons-io-2.4.jar:/usr/local/canal-adapter/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-adapter/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-adapter/bin/../lib/client-adapter.launcher-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/client-adapter.common-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-adapter/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-adapter/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication admin 3097 0 0 Jan10 pts/1 00:05:17 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=otter-canal -Dlogback.configurationFile=/usr/local/canal-deployer/bin/../conf/logback.xml -Dcanal.conf=/usr/local/canal-deployer/bin/../conf/canal.properties -classpath .:/usr/local/canal-deployer/bin/../conf:/usr/local/canal-deployer/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-deployer/bin/../lib/zkclient-0.10.jar:/usr/local/canal-deployer/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_pushgateway-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_httpserver-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_hotspot-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_common-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-deployer/bin/../lib/oro-2.0.8.jar:/usr/local/canal-deployer/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-deployer/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-deployer/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-deployer/bin/../lib/mybatis-spring-2.0.4.jar:/usr/local/canal-deployer/bin/../lib/mybatis-3.5.4.jar:/usr/local/canal-deployer/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-deployer/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-deployer/bin/../lib/jctools-core-2.1.2.jar:/usr/local/canal-deployer/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-deployer/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-deployer/bin/../lib/httpcore-4.4.3.jar:/usr/local/canal-deployer/bin/../lib/httpclient-4.5.1.jar:/usr/local/canal-deployer/bin/../lib/h2-1.4.196.jar:/usr/local/canal-deployer/bin/../lib/guava-22.0.jar:/usr/local/canal-deployer/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-deployer/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-deployer/bin/../lib/druid-1.2.6.jar:/usr/local/canal-deployer/bin/../lib/disruptor-3.4.2.jar:/usr/local/canal-deployer/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-deployer/bin/../lib/commons-lang3-3.7.jar:/usr/local/canal-deployer/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-deployer/bin/../lib/commons-io-2.4.jar:/usr/local/canal-deployer/bin/../lib/commons-compress-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-deployer/bin/../lib/canal.store-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.sink-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.server-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.prometheus-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.driver-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.dbsync-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.meta-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.spring-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.manager-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.filter-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.deployer-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-deployer/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.deployer.CanalLauncher admin 31216 0 0 Jan10 pts/1 00:10:19 /usr/bin/java -server -Xms2048m -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-< |