【TrueNAS】读写性能测试

Posted by 西维蜀黍 on 2021-05-13, Last Modified on 2022-01-02

可以测试

  • 物理 disk 的读写速度
  • ZFS 的读写速度(内存会作为ARC,因此内存越大,自然速度越快)
    • 在 TrueNAS 主机上进行测试
    • 在 client端 进行测试(会被ARC和网络影响)

Disk

  • 海康威视 C2000Pro 2TB
    • 官方标速:最大顺序读 3400MB/S,最大顺序写 3100MB/S

diskinfo

$ diskinfo -tv nvd0
...
Transfer rates:
	outside:       102400 kbytes in   0.037547 sec =  2727249 kbytes/sec
	middle:        102400 kbytes in   0.036066 sec =  2839239 kbytes/sec
	inside:        102400 kbytes in   0.097002 sec =  1055648 kbytes/sec

$ diskinfo -tv nvd0
...
Transfer rates:
	outside:       102400 kbytes in   0.037414 sec =  2736943 kbytes/sec
	middle:        102400 kbytes in   0.035900 sec =  2852368 kbytes/sec
	inside:        102400 kbytes in   0.212731 sec =   481359 kbytes/sec

SSD

  • 2.73GB/s

dd

SSD

Read

海康威视 SSD

# 1GB, block 1KB
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes transferred in 12.824254 secs (79848702 bytes/sec)

# 1GB, block 1MB
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.705722 secs (1485820772 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.657703 secs (1594300718 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.660621 secs (1587257485 bytes/sec)
$ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.662474 secs (1582817905 bytes/sec)

# 10GB, block 1MB
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 26.722108 secs (392400185 bytes/sec)
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 29.718711 secs (352833610 bytes/sec)
➜  ~ dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 31.348440 secs (334,490,644 bytes/sec)
  • block 1KB, 写1GB, 79MB/s

  • block 1MB, 写1GB, 1.58GB/s

  • block 1MB, 写10GB, 392MB/s

    • 这里慢猜测是因为内存用完了(16GB),而内存中无法存储一个大小为10GB的文件,因此不断发生swap(虽然swap utilization一直是0,我不确定是监控的问题,还是我理解错了)

三星 970 EVO plus 1T

# 1GB, block 1MB
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.594870 secs (1762697451 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.594439 secs (1763976215 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.604709 secs (1734017247 bytes/sec)

# 10GB, block 1MB
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 5.918966 secs (1771552612 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 5.942668 secs (1764486906 bytes/sec)
root@sw-truenas[~]# dd if=/dev/nvd0 of=speed_test_temp_file bs=1M count=10000
10000+0 records in
10000+0 records out
  • block 1MB, 写1GB, 1.77GB/s

  • block 1MB, 写10GB, 1770MB/s

Write

海康威视 SSD

$ dd if=/dev/zero of=/dev/nvd0 bs=1M count=1000
dd: /dev/nvd0: Operation not permitted

ZSF

Read

海康威视 SSD

# block 1MB, 1GB
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.090049 secs (11644463796 bytes/sec)
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.088272 secs (11878941100 bytes/sec)
$  if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.089497 secs (11716342837 bytes/sec)
$ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.088368 secs (11866037137 bytes/sec)

# block 1MB, 1GB
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.890868 secs (11770267488 bytes/sec)
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.888398 secs (11803004795 bytes/sec)
➜  ~ dd if=/mnt/SWPoolSSD/SSD_DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.887600 secs (11,813,611,078 bytes/sec)
  • block 1MB, 1GB 11.8GB/s
  • block 1MB, 10GB 11.8GB/s

三星 970 EVO plus 1T

# block 1MB, 1GB
root@sw-truenas[~]# dd if=/mnt/SSDPool2/DS/test of=/dev/zero bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.091078 secs (11512975413 bytes/sec)

# block 1MB, 1GB
root@sw-truenas[~]# dd if=/mnt/SSDPool2/DS/test of=/dev/zero bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 0.909169 secs (11533351590 bytes/sec)
  • block 1MB, 1GB 11.5GB/s
  • block 1MB, 10GB 11.5GB/s

Write

海康威视 SSD

# 1G 连续文件
$ dd if=/dev/zero of=/mnt/SWPoolSSD/SSD_DS/test bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.208581 secs (5027195733 bytes/sec)

# 10G 连续文件
$ dd if=/dev/zero of=/mnt/SWPoolSSD/SSD_DS/test bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 2.299416 secs (4,560,184,058 bytes/sec)
  • 1G 连续文件,5.0GB/s
  • 10G 连续文件,4.5GB/s

三星 970 EVO plus 1T

# 1G 连续文件
root@sw-truenas[~]# dd if=/dev/zero of=/mnt/SSDPool2/DS/test bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.137217 secs (7641718312 bytes/sec)

# 10G 连续文件
root@sw-truenas[~]# dd if=/dev/zero of=/mnt/SSDPool2/DS/test bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 1.547743 secs (6774871512 bytes/sec)
  • 1G 连续文件,7.6GB/s
  • 10G 连续文件,6.7GB/s

via SMB

Read

海康威视 SSD

# block 1MB, 1GB in total
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1,000,000,000 bytes transferred in 0.117495 secs (8,510,996,124 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test4 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.095148 secs (10509932846 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.096621 secs (10349713022 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 0.095712 secs (10,448,016,540 bytes/sec)

# block 1MB, 10GB in total
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.856220 secs (11679241222 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.837105 secs (11945932198 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.841456 secs (11884163587 bytes/sec)
$ dd if=/Volumes/SWPoolSSD/test2 of=/dev/zero bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 0.839158 secs (11916706159 bytes/sec)
  • block 1MB, 1GB in total: 10.5GB/s
  • block 1MB, 10GB in total: 11.9GB/s

三星 970 EVO plus 1T

Write

海康威视 SSD

# 1G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.162612 secs (860132214 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.136232 secs (880102029 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.157193 secs (864159947 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.191698 secs (839138891 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.182802 secs (845450069 bytes/sec)

# 10G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 12.628445 secs (791863137 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 13.044244 secs (766621672 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SWPoolSSD/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 13.161974 secs (759764457 bytes/sec)
  • 1G 连续文件,880MB/s
  • 10G 连续文件,791MB/s

三星 970 EVO plus 1T

# 1G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.362847 secs (733758033 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.328901 secs (752501473 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.347447 secs (742144131 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=1000
1000+0 records in
1000+0 records out
1000000000 bytes transferred in 1.253311 secs (797886458 bytes/sec)

# 10G 连续文件
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 14.405663 secs (694171451 bytes/sec)
➜  ~ dd if=/dev/zero of=/Volumes/SSDPool2/test5 bs=1000000 count=10000
10000+0 records in
10000+0 records out
10000000000 bytes transferred in 14.603652 secs (684760223 bytes/sec)
  • 1G 连续文件,797MB/s
  • 10G 连续文件,694MB/s

bonnie++

$ bonnie++ -d [TEST_LOCATION] -s [TEST_SIZE] -n 0 -m [TEST_NAME] -f -b -u [TEST_USER]

# For example:
$ bonnie++ -d /tmp -s 4G -n 0 -m TEST -f -b -u root
  • -d – is used to specify the file system directory to use to benchmark.
  • -u – is used to run a a particular user. This is best used if you run the program as root. This is the UID or the name.
  • -g – is used to run as a particular group. This is the GID or the name.
  • -r – is used to specify the amount of RAM in MB the system has installed. This is total RAM, and not free RAM. Use free -m to find out how much RAM is on your system.
  • -b – removes write buffering and performs a sync at the end of each bonnie++ operation.
  • -s – specifies the dataset size to use for the IO test in MB.
  • -n – is the number of files to use for the create files test.
  • -m – this adds a label to the output so that you can understand what the test was at a later date.
  • -x n – is used to repeat the tests n times. Change n to the number of how many times to run the tests.

直接拷贝文件

$ zpool iostat SWPoolSSD 1
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SWPoolSSD    142G  1.72T      6     13   738K   176K
SWPoolSSD    142G  1.72T      0      0      0      0

从TrueNAS 拷贝文件至 client:

说明10Gb 网络几乎被跑满了。

但很奇怪的是,对于 disk usage 的监测,无论是使用

$ systat -iostat

还是

$ iostat -c 100000 -d -K 1

还是

$ zpool iostat -n 1

都看不出有数据的变化(这点非常奇怪,暂时没找到问题在哪里,猜测有可能是TrueNAS会把数据写到内存中,然后异步的写入到disk中)。

从client 拷贝文件至 TrueNAS:

Reference