Specification
Storage
Hybrid Storage
In a hybrid configuration, large-capacity spinning disks store the data, while DRAM and flash act as hyper-fast read and write caching.
The technologies work in conjunction with a flash-based separate write log (SLOG), which can be thought of as a write cache that keeps what’s called the ZFS-intent log (ZIL), used to accelerate writes. On the read side, flash can be used as a level two adaptive replacement (read) cache (L2ARC) to keep the hottest data sets on the faster flash media. Workloads with synchronous writes such as NFS and databases consistently benefit from SLOG devices, while workloads with frequently-accessed data may benefit from an L2ARC device. The reason that an L2ARC device is not always the best choice is because the level one ARC in RAM will always provide a faster cache; also, some RAM will be used by the L2ARC table.
Keep in mind that for every data block in the L2ARC, the primary ARC needs an 88 byte entry; this can cause the ARC to fill up unexpectedly and actually reduce performance in a poorly-designed system. For example, a 480GB L2ARC filled with 4KiB blocks will need more than 10GiB of metadata storage in the primary ARC!
About L2CARC and ARC
- a level two ARC, or L2ARC, is typically an SSD or NVMe-based read cache, instead of write and read cache, which means adding L2CARC won’t boost the write speed.
- An SSD or NVMe card is often used for a ZFS separate log device, or SLOG, to boost the performance of synchronized writes, such as over NFS or with a database.
Boot Devices
- All TrueNAS Mini and Certified systems ship with either M.2 drives or SATA DOMs.
- Popular to boot TrueNAS systems from 2.5” SSDs and HDDs (often mirrored for added redundancy). 8 GB is the recommended minimum size for the TrueNAS boot volume, but using 16 or 32 GB (or a 120 GB 2.5” SATA SSD) provides room for more boot environments.
Storage Device Sizing
how many drives should I use to achieve my desired capacity?
Just because one can mirror two 16TB drives to achieve 16TB of available capacity, it doesn’t necessarily mean that one should. Mirroring two large drives offers the advantage of redundancy and balancing reads between the two devices, potentially lowering power draw, but little else. The write performance of two large drives will be at most that of a single drive.
By contrast, an array of eight 4TB drives would offer a wide range of configurations to optimize performance and redundancy at a lower cost. If configured as striped mirrors, eight drives could yield four times greater write performance with similar total capacity. You may also consider adding a “hot spare” drive with any zpool configuration to allow for the zpool to automatically rebuild itself in the event a primary drive fails in the zpool.
Ref
- https://www.truenas.com/docs/core/introduction/corehardwareguide/#storage-considerations
- https://www.ixsystems.com/blog/visualizing-zfs-performance/
Memory, CPU, and Network Considerations
Memory Sizing
TrueNAS has higher memory requirements than many Network Attached Storage solutions for good reason: it shares dynamic random-access memory (DRAM or simply RAM) between sharing services, add-on plugins, jails, and virtual machines, and sophisticated read caching. RAM rarely goes unused on a TrueNAS system and enough RAM is key to maintaining peak performance. You should have at least 8 GB of RAM for basic TrueNAS operations with up to eight drives. Other use cases each have distinct RAM requirements:
- Add 1 GB for each drive added after eight to benefit most use cases.
- Add extra RAM (in general) if more clients will connect to the TrueNAS system. A 20 TB pool backing lots of high-performance VMs over iSCSI might need more RAM than a 200 TB pool storing archival data. If using iSCSI to back VMs, plan to use at least 16 GB of RAM for reasonable performance and 32 GB or more for optimal performance.
- Add 2 GB of RAM for directory services for the winbind internal cache.
- Add more RAM as required for plugins and jails as each has specific application RAM requirements.
- Add more RAM for virtual machines with a guest operating system and application RAM requirements.
- Add the suggested 5 GB per TB of storage for deduplication that depends on an in-RAM deduplication table.
- Add approximately 1 GB of RAM (conservative estimate) for every 50 GB of L2ARC in your pool. Attaching an L2ARC drive to a pool uses some RAM, too. ZFS needs metadata in ARC to know what data is in L2ARC.
Permission Management
Data Backups
Snapshot
Ref https://www.truenas.com/docs/core/storage/snapshots/
Periodic Snapshot Tasks
Ref https://www.truenas.com/docs/core/tasks/periodicsnapshottasks/
Replication Tacks
Ref https://www.truenas.com/docs/core/tasks/replicationtasks/local/
Cloud Sync Tasks
S3
https://swsmile.info/post/truenas-sync-to-s3/
Ref
Data/Disk Health Checks
Scrub Tasks
Initiate a data integrity check of the pool. Any problems detected during the scrub are either automatically corrected or will generate an alert in the web interface. By default, every pool is automatically checked on a reoccurring scrub schedule.
Ref https://www.truenas.com/docs/core/tasks/scrubtasks/
S.M.A.R.T Tests
- Long - runs SMART Extended Self Test. This will scan the entire disk surface and can take many hours on large-volume disks.
- Short - runs SMART Short Self Test (usually under ten minutes). These are basic disk tests that vary by manufacturer.
- Conveyance - runs a SMART Conveyance Self Test. This self-test routine is intended to identify damage incurred during transporting of the device. This self-test routine requires only minutes to complete.
- Offline - runs SMART Immediate Offline Test. The effects of this test are visible only in that it updates the SMART Attribute values, and if the test finds errors, they appear in the SMART error log.
Ref https://www.truenas.com/docs/core/tasks/smarttests/
安全性
使用ACL
- 对Pools 中的 DataSet 使用 ACL
SSH
-
Untick Log in as Root with Password
-
Random ssh Port Number
-
Untick Allow Password Authentication
Performance
MTU 设置
- 以下均使用 iperf3 进行 benchmark
- 10Gb 网络
两边都是1500
主机 -> truenas
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.00 sec 34.2 GBytes 4.89 Gbits/sec receiver
TrueNAS设置为9000
主机 -> truenas
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.00 sec 33.2 GBytes 4.75 Gbits/sec sender
[ 5] 0.00-60.00 sec 33.2 GBytes 4.75 Gbits/sec receiver
两边都设置为9000
主机 -> truenas
[ ID] Interval Transfer Bitrate
[ 5] 0.00-60.00 sec 68.8 GBytes 9.85 Gbits/sec sender
[ 5] 0.00-60.00 sec 68.8 GBytes 9.85 Gbits/sec receiver
truenas-> 主机
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 68.2 GBytes 9.76 Gbits/sec 1222 sender
[ 5] 0.00-60.08 sec 68.2 GBytes 9.75 Gbits/sec receiver
结论
- 10Gb网络下，设置MTU 为 9000能大大提高性能
软件（package）安装
- 安装在 Jail 中
Monitor
使用 Node Exporter + Prometheus + Grafana 进行监控，具体参考 https://swsmile.info/post/truenas-use-node-exporter-to-monitor/。
开机启动
Efficiency
oh-my-zsh
$ sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
$ cd ~/.oh-my-zsh/custom/plugins
# zsh-autosuggestions
$ git clone https://github.com/zsh-users/zsh-autosuggestions
# zsh-syntax-highlighting
$ git clone https://github.com/zsh-users/zsh-syntax-highlighting.git
$ vim ~/.zshrc
source ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh
source ~/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
Setting
Network Interface
设置 Interface 开机后自动为 up
Disk
每天执行 S.M.A.R.T 以检查硬盘状态
