使用JMeter测试TDengine数据库性能

TDengine 是一款专为时序数据打造的国产数据库产品,它可对TB量级的时序数据进行存储、分析和分发,常备应用于设备运行状态实时监测和预警等场景。

在前面的文章中我们曾经介绍过使用sysbench进行过数据库性能测试,但sysbench聚焦于mysql和postgresql这两种关系数据库,对其他数据库的支持几乎为零(曾支持过oracle但后来取消了)。因此这里我们使用更通用的测试框架JMeter进行性能测试。

环境信息

TDengine服务器 TDengine驱动 jmeter
3.0.0 3.0.2 5.5

添加依赖

将连接TDengine所需的jar包复制到jmeter安装目录下的lib目录:

taos-jdbcdriver-3.0.2.jar
fastjson-1.2.29.jar

准备数据库

在TDengine数据库里预先创建用于测试的库和表:

create database if not exists test vgroups 10 buffer 10;

测试发现create table既可以创建stable也可以创建table,为避免歧义我们严格用create stable来创建超级表:

create stable test.stable01 (time timestamp, col001 double, col002 double, col003 double, \
    col004 double, col005 double, col006 double, col007 double, col008 double, col009 double, \
    col010 double) tags (device varchar(20));

创建子表:

create table test.table01 using stable01 tags ('device01');

测试计划

在GUI模式下创建测试计划(tdengine_test.jmx):
file

执行测试

执行测试(-n表示不启动GUI,-t参数指定测试计划文件,-l指定测试csv格式的结果文件,-e表示测试结束后生成html格式报告,-o表示指定报告的路径名):

jmeter -n -t tdengine_test.jmx -l log1.jtl -e -o report1

测试机作为客户端,与TDengine服务不在同一个局域网内的情况下测试结果:
file

测试机作为客户端,与TDengine服务在同一个局域网内的情况下测试结果:
file

参考资料

https://jmeter.apache.org/usermanual/build-db-test-plan.html
https://jmeter.apache.org/usermanual/generating-dashboard.html
https://docs.taosdata.com/connector/java/
https://docs.taosdata.com/taos-sql/table/

TDengine 2.6集群版升级到3.0集群版

本文记录了在由三个物理节点组成的集群上,原地将TDengine 2.6版升级到3.0版的过程,由于3.0的改动比较大,所以此次升级并非完全平滑的升级,只升级了数据库服务,没有将数据库里的存量数据导入到新版环境,如需保留数据请联系taos团队。本文使用的TDengine版本均为社区版,非docker安装。

一、卸载2.6版

备份配置文件

> ansible allnode -m shell -a "mv /etc/taos/ /etc/taos.bak"

按taos文档说明卸载程序后配置文件是会保留的,但3.0版与2.6版的配置文件(配置项)有差异,直接让3.0使用2.6版配置文件会导致服务启动失败。因此决定自行保留并在安装后再3.0版的配置文件:

当缺省配置文件( /etc/taos/taos.cfg )存在时,仍然使用已有的配置文件,安装包中携带的配置文件修改为taos.cfg.org保存在 /usr/local/taos/cfg/ 目录

备份数据文件

数据文件默认位置在/usr/local/taos/data

> ansible allnode -m shell -a "mv /data/tdengine /data/tdengine.bak"

如果升级3.0失败需要恢复为2.6版,则备份的数据文件会起作用,否则可以删除。数据文件的位置以taos.cfg文件内的配置为准。

停止taosd服务并卸载程序

> ansible allnode -m shell -a "systemctl stop taosd"
> ansible allnode -m shell -a "rpm -e tdengine"

192.168.130.153 | CHANGED | rc=0 >>
taosadapter is running, stopping it...
TDengine is removed successfully!
192.168.130.154 | CHANGED | rc=0 >>
taosadapter is running, stopping it...
TDengine is removed successfully!
192.168.130.152 | CHANGED | rc=0 >>
taosadapter is running, stopping it...
TDengine is removed successfully!

其中ansible allnode -m shell -a命令是为了方便在集群所有节点上一起执行,如果没有装ansible工具可在每个节点上手工执行,效果是一样的。

二、安装TDengine3.0

下载单机版安装包

TDengine提供了多种格式的安装包,我们在CentOS上选择使用RPM版的。

> wget https://www.taosdata.com/assets-download/3.0/TDengine-server-3.0.1.5-Linux-x64.rpm

由于安装过程中需要输入一些信息,所以建议在每个节点上手工执行(而非通过ansible):

> rpm -ivh TDengine-server-3.0.1.5-Linux-x64.rpm

安装过程会提示输入两个信息,按taos的安装说明输入即可(我两个都选择了回车跳过):

当安装第一个节点时,出现 Enter FQDN: 提示的时候,不需要输入任何内容。只有当安装第二个或以后更多的节点时,才需要输入已有集群中任何一个可用节点的 FQDN,支持该新节点加入集群。当然也可以不输入,而是在新节点启动前,配置到新节点的配置文件中。

三、修改taos配置文件

根据之前备份的2.6版的taos.cfg文件修改3.0版的配置文件,我这里主要改的是firstEpfqdnlogDirdataDirtimezonecharset等几个:

# first fully qualified domain name (FQDN) for TDengine system
firstEp                   taos-1:6030
# local fully qualified domain name (FQDN)
fqdn                      taos-1
# The directory for writing log files
 logDir                    /data/tdengine/log
# All data files are stored in this directory
 dataDir                  /data/tdengine/data
# system time zone
 timezone              Asia/Shanghai (CST, +0800)
# system locale
 locale                    en_US.UTF-8
# default system charset
 charset                   UTF-8

为节约时间,可以先改好第一个节点的,再复制到其他节点,然后只要去每个节点修改fqdn的值即可。

> scp /etc/taos/taos.cfg root@taos-2:/etc/taos/taos.cfg
> scp /etc/taos/taos.cfg root@taos-3:/etc/taos/taos.cfg

四、启动taosd并查看状态

启动第一个节点

在第一个节点(firstEp)上启动taosd服务,并验证服务启动成功:

> systemctl start taosd
> systemctl status taosd

此处如果遇到启动失败可进一步查看日志文件来找原因。遇到过一个DND ERROR failed to start since read config error的错误比较坑,原因提示不明确。后来发现是hostname与taos.cfg不匹配造成的。

在taos命令行里验证此时有一个dnode:

> taos
taos> show dnodes;
     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |              note              |
=================================================================================================================================================
           1 | taos-1:6030                    |      0 |             16 | ready      | 2022-10-26 14:50:57.131 |                                |
Query OK, 1 rows in database (0.003960s)

添加其他节点

与第一个节点类似,只不过启动后要在taos命令行里执行create dnode命令将当前节点添加到集群。

> systemctl start taosd
> systemctl status taosd
> taos

taos> create dnode "taos-2";
Query OK, 0 of 0 rows affected (0.002127s)

taos> show dnodes;
     id      |            endpoint            | vnodes | support_vnodes |   status   |       create_time       |              note              |
=================================================================================================================================================
           1 | taos-1:6030                    |      0 |             16 | ready      | 2022-10-26 14:50:57.131 |                                |
           2 | taos-2:6030                    |      0 |             16 | ready      | 2022-10-26 14:54:48.635 |                                |
Query OK, 2 rows in database (0.003566s)

五、创建数据库用户(可选)

创建必要的数据库用户:

> taos -s "CREATE USER k2data PASS 'xxxxxxxx';"

创建必要的database:

注意在2.6里的DAYS改为DURATION了,且3.0不支持UPDATE参数了(相当于UPDATE=1不可修改):

taos -s 'CREATE DATABASE repos REPLICA 3 KEEP 3650 DURATION 10;'

至此TDengine版本升级完成。

参考资料

TDengine多种安装包的安装和卸载

使用安装包立即开始

集群部署和管理

Zookeeper启动失败和解决一例

服务器重启后zookeeper启动失败,报错日志如下:

zookeeper    | ZooKeeper JMX enabled by default
zookeeper    | Using config: /opt/zookeeper-3.4.13/bin/../conf/zoo.cfg
zookeeper    | 2022-10-05 17:24:53,976 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: /opt/zookeeper-3.4.13/bin/../conf/zoo.cfg
zookeeper    | 2022-10-05 17:24:53,979 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
zookeeper    | 2022-10-05 17:24:53,979 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
zookeeper    | 2022-10-05 17:24:53,980 [myid:] - WARN  [main:QuorumPeerMain@116] - Either no config or no quorum defined in config, running  in standalone mode
zookeeper    | 2022-10-05 17:24:53,980 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
zookeeper    | 2022-10-05 17:24:53,990 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: /opt/zookeeper-3.4.13/bin/../conf/zoo.cfg
zookeeper    | 2022-10-05 17:24:53,990 [myid:] - INFO  [main:ZooKeeperServerMain@98] - Starting server
zookeeper    | 2022-10-05 17:24:53,995 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=fdf9214b4b7e
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.7.0_65
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/opt/zookeeper-3.4.13/bin/../build/classes:/opt/zookeeper-3.4.13/bin/../build/lib/*.jar:/opt/zookeeper-3.4.13/bin/../lib/slf4j-log4j12-1.7.25.jar:/opt/zookeeper-3.4.13/bin/../lib/slf4j-api-1.7.25.jar:/opt/zookeeper-3.4.13/bin/../lib/netty-3.10.6.Final.jar:/opt/zookeeper-3.4.13/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-3.4.13/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.13/bin/../lib/audience-annotations-0.5.0.jar:/opt/zookeeper-3.4.13/bin/../zookeeper-3.4.13.jar:/opt/zookeeper-3.4.13/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.13/bin/../conf:
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
zookeeper    | 2022-10-05 17:24:53,996 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
zookeeper    | 2022-10-05 17:24:53,998 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
zookeeper    | 2022-10-05 17:24:53,998 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
zookeeper    | 2022-10-05 17:24:53,998 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=4.4.249-1.el7.elrepo.x86_64
zookeeper    | 2022-10-05 17:24:53,998 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=root
zookeeper    | 2022-10-05 17:24:53,999 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/root
zookeeper    | 2022-10-05 17:24:53,999 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.13
zookeeper    | 2022-10-05 17:24:54,000 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
zookeeper    | 2022-10-05 17:24:54,000 [myid:] - INFO  [main:ZooKeeperServer@836] - tickTime set to 2000
zookeeper    | 2022-10-05 17:24:54,000 [myid:] - INFO  [main:ZooKeeperServer@845] - minSessionTimeout set to -1
zookeeper    | 2022-10-05 17:24:54,000 [myid:] - INFO  [main:ZooKeeperServer@854] - maxSessionTimeout set to -1
zookeeper    | 2022-10-05 17:24:54,007 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
zookeeper    | 2022-10-05 17:24:54,010 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
zookeeper    | 2022-10-05 17:24:54,799 [myid:] - ERROR [main:ZooKeeperServerMain@66] - Unexpected exception, exiting abnormally
zookeeper    | java.io.EOFException
zookeeper    |  at java.io.DataInputStream.readInt(DataInputStream.java:392)
zookeeper    |  at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219)
zookeeper    |  at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176)
zookeeper    |  at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217)
zookeeper    |  at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284)
zookeeper    |  at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407)

一开始怀疑是zoo.cfg损坏,但查看后发现文件没有明显异常。将/opt/zookeeper-3.4.13/data/version-2目录改名为version-2.bak后重启zookeeper正常(但client连接报错 zxid 0x02 our last zxid is 0x0 client must try another server,因为前面的操作导致zk这边的事务id被重置了),说明可能是data下有文件损坏。

进一步查看version-2.bak下的文件,发现有一个0字节的log文件:

file

删除此文件并将version-2.bak改回原来的名字version-2,重启zookeeper成功,并且client连接正常。