TrueNAS

【TrueNAS】读写性能测试

Posted by 西维蜀黍 on 2021-05-13, Last Modified on 2022-01-02

可以测试

  • 物理 disk 的读写速度
  • ZFS 的读写速度（内存会作为ARC，因此内存越大，自然速度越快）
    • 在 TrueNAS 主机上进行测试
    • 在 client端 进行测试（会被ARC和网络影响）

Disk

  • 海康威视 C2000Pro 2TB
    • 官方标速：最大顺序读 3400MB/S，最大顺序写 3100MB/S

diskinfo

$ diskinfo -tv nvd0
...
Transfer rates:
	outside:       102400 kbytes in   0.037547 sec =  2727249 kbytes/sec
	middle:        102400 kbytes in   0.036066 sec =  2839239 kbytes/sec
	inside:        102400 kbytes in   0.097002 sec =  1055648 kbytes/sec

$ diskinfo -tv nvd0
...
Transfer rates:
	outside:       102400 kbytes in   0.037414 sec =  2736943 kbytes/sec
	middle:        102400 kbytes in   0.035900 sec =  2852368 kbytes/sec
	inside:        102400 kbytes in   0.212731 sec =   481359 kbytes/sec

SSD

  • 2.73GB/s

dd

SSD

Read

海康威视 SSD

# 1GB, block 1KB
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes transferred in 12.824254 secs (79848702 bytes/sec)

# 1GB, block 1MB
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.705722 secs (1485820772 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.657703 secs (1594300718 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.660621 secs (1587257485 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.662474 secs (1582817905 bytes/sec)

# 10GB, block 1MB
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 26.722108 secs (392400185 bytes/sec)
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 29.718711 secs (352833610 bytes/sec)
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 31.348440 secs (334,490,644 bytes/sec)

  • block 1KB, 写1GB, 79MB/s

  • block 1MB, 写1GB, 1.58GB/s

  • block 1MB, 写10GB, 392MB/s

    • 这里慢猜测是因为内存用完了（16GB），而内存中无法存储一个大小为10GB的文件，因此不断发生swap（虽然swap utilization一直是0，我不确定是监控的问题，还是我理解错了）

三星 970 EVO plus 1T

# 1GB, block 1MB
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.594870 secs (1762697451 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.594439 secs (1763976215 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.604709 secs (1734017247 bytes/sec)

# 10GB, block 1MB
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 5.918966 secs (1771552612 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 5.942668 secs (1764486906 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out

  • block 1MB, 写1GB, 1.77GB/s

  • block 1MB, 写10GB, 1770MB/s

Write

海康威视 SSD

$ dd if=/dev/zero of=/dev/nvd0 bs=1M count=1000
dd: /dev/nvd0: Operation not permitted

ZSF

Read

海康威视 SSD

# block 1MB, 1GB
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.090049 secs (11644463796 bytes/sec)
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.088272 secs (11878941100 bytes/sec)
$  if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.089497 secs (11716342837 bytes/sec)
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.088368 secs (11866037137 bytes/sec)

# block 1MB, 1GB
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.890868 secs (11770267488 bytes/sec)
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.888398 secs (11803004795 bytes/sec)
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.887600 secs (11,813,611,078 bytes/sec)
  • block 1MB, 1GB 11.8GB/s
  • block 1MB, 10GB 11.8GB/s

三星 970 EVO plus 1T

# block 1MB, 1GB
root@sw-truenas[~]# dd if=/mnt/SSDPool2/DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.091078 secs (11512975413 bytes/sec)

# block 1MB, 1GB
root@sw-truenas[~]# dd if=/mnt/SSDPool2/DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.909169 secs (11533351590 bytes/sec)
  • block 1MB, 1GB 11.5GB/s
  • block 1MB, 10GB 11.5GB/s

Write

海康威视 SSD

# 1G 连续文件
$ dd if=/dev/zero of=/mnt/SWPoolSSD/SSD_DS/test bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.208581 secs (5027195733 bytes/sec)

# 10G 连续文件
$ dd if=/dev/zero of=/mnt/SWPoolSSD/SSD_DS/test bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 2.299416 secs (4,560,184,058 bytes/sec)
  • 1G 连续文件，5.0GB/s
  • 10G 连续文件，4.5GB/s

三星 970 EVO plus 1T

# 1G 连续文件
root@sw-truenas[~]# dd if=/dev/zero of=/mnt/SSDPool2/DS/test bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.137217 secs (7641718312 bytes/sec)

# 10G 连续文件
root@sw-truenas[~]# dd if=/dev/zero of=/mnt/SSDPool2/DS/test bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 1.547743 secs (6774871512 bytes/sec)
  • 1G 连续文件，7.6GB/s
  • 10G 连续文件，6.7GB/s

via SMB

Read

海康威视 SSD

# block 1MB, 1GB in total
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1,000,000,000 bytes transferred in 0.117495 secs (8,510,996,124 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test4 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.095148 secs (10509932846 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.096621 secs (10349713022 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.095712 secs (10,448,016,540 bytes/sec)

# block 1MB, 10GB in total
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.856220 secs (11679241222 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.837105 secs (11945932198 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.841456 secs (11884163587 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.839158 secs (11916706159 bytes/sec)
  • block 1MB, 1GB in total: 10.5GB/s
  • block 1MB, 10GB in total: 11.9GB/s

三星 970 EVO plus 1T

Write

海康威视 SSD

# 1G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.162612 secs (860132214 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.136232 secs (880102029 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.157193 secs (864159947 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.191698 secs (839138891 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.182802 secs (845450069 bytes/sec)

# 10G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 12.628445 secs (791863137 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 13.044244 secs (766621672 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 13.161974 secs (759764457 bytes/sec)
  • 1G 连续文件，880MB/s
  • 10G 连续文件，791MB/s

三星 970 EVO plus 1T

# 1G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.362847 secs (733758033 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.328901 secs (752501473 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.347447 secs (742144131 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.253311 secs (797886458 bytes/sec)

# 10G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 14.405663 secs (694171451 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 14.603652 secs (684760223 bytes/sec)
  • 1G 连续文件，797MB/s
  • 10G 连续文件，694MB/s

bonnie++

$ bonnie++ -d [TEST_LOCATION] -s [TEST_SIZE] -n 0 -m [TEST_NAME] -f -b -u [TEST_USER]

# For example:
$ bonnie++ -d /tmp -s 4G -n 0 -m TEST -f -b -u root
  • -d – is used to specify the file system directory to use to benchmark.
  • -u – is used to run a a particular user. This is best used if you run the program as root. This is the UID or the name.
  • -g – is used to run as a particular group. This is the GID or the name.
  • -r – is used to specify the amount of RAM in MB the system has installed. This is total RAM, and not free RAM. Use free -m to find out how much RAM is on your system.
  • -b – removes write buffering and performs a sync at the end of each bonnie++ operation.
  • -s – specifies the dataset size to use for the IO test in MB.
  • -n – is the number of files to use for the create files test.
  • -m – this adds a label to the output so that you can understand what the test was at a later date.
  • -x n – is used to repeat the tests n times. Change n to the number of how many times to run the tests.

直接拷贝文件

$ zpool iostat SWPoolSSD 1
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SWPoolSSD    142G  1.72T      6     13   738K   176K
SWPoolSSD    142G  1.72T      0      0      0      0

从TrueNAS 拷贝文件至 client:

说明10Gb 网络几乎被跑满了。

但很奇怪的是，对于 disk usage 的监测，无论是使用

$ systat -iostat

还是

$ iostat -c 100000 -d -K 1

还是

$ zpool iostat -n 1

都看不出有数据的变化（这点非常奇怪，暂时没找到问题在哪里，猜测有可能是TrueNAS会把数据写到内存中，然后异步的写入到disk中）。

从client 拷贝文件至 TrueNAS：

Reference

FEATURED TAGS
algorithm algorithmproblem architecturalpattern architecture aws c# cachesystem codis compile concurrentcontrol database dataformat datastructure debug design designpattern distributedsystem django docker domain engineering freebsd git golang grafana hackintosh hadoop hardware hexo http hugo ios iot java javaee javascript kafka kubernetes linux linuxcommand linuxio lock macos markdown microservices mysql nas network networkprogramming nginx node.js npm oop openwrt operatingsystem padavan performance programming prometheus protobuf python redis router security shell software testing spring sql systemdesign truenas ubuntu vmware vpn windows wmware wordpress xml zookeeper
FRIENDS

TOC