A ‘brief’ primer on NAS
TL;DR move to the next section if you find this boring …
NAS, being the (mostly) cheaper cousin to SANs, is used pervasively everywhere from SOHO environments to large corporates. Netapp (Network Appliance) basically invented the category in the 90’s with their F-series of units combining SMB and NFS access, as well as their proprietary RAID-4 like system with WAFL (write anywhere file layout) which provided high performance and effective zero-latency snapshots (fairly impressive seeing a 1TB volume being reverted to a previous version in a few seconds). Their OS, Data ONTAP, was and still is, fairly revolutionary in the world of NAS systems with advanced file system, RAID and snapshot features. More recent filesystems like ZFS and BTRFS are copying some of these features (like copy-on-write).
Many other vendors have come into the NAS market over the years, the bulk of them in the SMB/SME markets (Qnap, Synology, Intel, WD, Buffalo, etc.), and a few in enterprise environments (EMC, Hitachi, Isilon, Spinnaker).
There has also been a lot of interest in the open source arena with systems being developed/provided by OpenFiler, OMV, Nexenta and FreeNAS. FreeNAS (or TrueNAS as it’s more recently known) in particular has been very successful and provides a rich environment for users with its ZFS-based filesystem and consumer-oriented plugin architecture. TrueNAS is used heavily in SMB/SME and consumer environments and as such, might be considered the benchmark for NAS systems in general.
As an aside, ZFS’ feature set is rich and includes snapshots, pools, checksums, replication, encryption and a variety of RAID-like configurations (RAID-Z) for data/disk integrity. From Wikipedia:
RAID-Z is a data/parity distribution scheme like RAID-5, but uses dynamic stripe width: every block is its own RAID stripe, regardless of blocksize, resulting in every RAID-Z write being a full-stripe write. This, when combined with the copy-on-write transactional semantics of ZFS, eliminates the write hole error. RAID-Z is also faster than traditional RAID 5 because it does not need to perform the usual read-modify-write sequence.
What disk drives are typically used in NAS systems?
There are generally 2 types of drives that are split across consumer/SMB and SME/enterprise solutions.
On the high-end, there are enterprise drives in the form of SAS/FC including both performance (eg. 900GB SAS 15k) and archival (eg. 8TB SAS 7.2k) versions.
In the mid to low-end, we’re looking at entry-level SAS as well as SATA drives. Two of the most popular lines are WD’s Red series (in both standard and Pro versions) and Seagate’s IronWolf drives.
NAS/SAN drives have some unique properties that advise their use in RAID use cases:
- vibration and noise control – critical when you have large groups of disks operating within enclosures
- enhanced error correction
- advanced power management
Consumer versions of NAS drives (eg. WD Red) are priced fairly similarly to their more standard desktop versions and so are often used in consumer-based or open source NAS solutions. You (normally) wouldn’t think twice about putting a 4-16 drive system together for home/photo/video/audio use using these drives.
And in general, these cheaper consumer NAS drives perform very well. Until …
SMR vs CMR
CMR/PMR (conventional magnetic recording) has been around with various minor variations for many years now. Almost every drive (desktop, laptop, enterprise) manufactured in the last 15 years has used this technology – it’s generally reliable and provides for decent performance for spinning rust.
But disk manufacturers are always looking for opportunities to save a buck, and so have looked to newer technologies like SMR (shingled magnetic recording) and HAMR (heat assisted magnetic recording) to boost capacity while saving costs. SMR does this by overlapping tracks allowing for higher data density, and the resulting possibility of using less platters.
But SMR has a big achilles heel in that zones of overlapping tracks need to be written in full (similar to flash disk/SSD) resulting in potentially slow and non-deterministic performance.
However, when the data on an SMR drive is edited or overwritten, the write head will not overwrite the data onto the existing magnetic track. Instead, the new data will be written onto an empty area of the disk, while the original track with the old data will temporarily remain. When the HDD becomes idle, it will enter a reorganization mode where the old bits of data on the original track will be erased and made fully available for future use.
Because NAS drives are used in high performance applications and with a requirement for RAID-like redundancy, one expects a certain deterministic level of reliability and performance from these disks.
An example of this is that when a disk in a RAID group/pool fails, the controller needs to rebuild the failed disk’s data onto a new/replacement disk. Depending on the RAID level/type, and if some form of parity is involved, it can take a lot of computation and drive reads to rebuild this data resulting in additional pressure on the remaining drives and an impact on performance. The longer a RAID group/pool takes to rebuild (or resilver in the case of ZFS), the higher the likelihood of a 2nd drive failing.
Using Memset’s RAID Calculator, we could calculate that rebuilding a failed RAID 5 set with 12 x 3TB disks would take almost 4 days (at 10MB/s rebuild rate) with 1 in 8 data loss odds/year. That’s pretty scary.
Another critical factor is URE (unrecoverable read error) rate which is the rate at which a read error will occur in drives. To put it plainly, the numbers are not good.
The post boy for SMR – Western Digital
So what’s all the hooha?
Well SMR is not good. At least, it’s not good when you need deterministic performance. And NAS systems require deterministic performance.
Some manufacturers have recently been caught (I’ll explain shortly) replacing CMR drive models with SMR versions. This means that someone who has been using CMR drives in a NAS or RAID-like system, could be in for a shock if they need to replace a failed drive and do so (unknowingly) with an SMR version. The non-deterministic performance nature of SMR drives could (and has) caused rebuilds to fail …
The reason I say caught is because they hadn’t indicated the change in drive technology anywhere in their literature at the time this issue surfaced. As well, the initially dismissed this issue.
The first manufacturer to be found using SMR for NAS drives was WD when some folks started having issues with rebuilds in TrueNAS/ZFS while using WD Red drives. These folks realised that the issue was caused by these newer versions of the WD Red drives using SMR technology. The difference on paper was a letter in the drive model no. – WD Red 2TB:
- WD20EFAX (SMR version)
- WD20EFRX (CMR version)
But that letter caused havoc for many – the first apparent issues surfaced in a message popped up on the zfs-discuss mailing list:
WD and Seagate are both submarining Drive-managed SMR (DM-SMR) drives into channels, disguised as “normal” drives.
For WD REDs this shows as EFRX (standard drive) suffix being changed to EFAX suffix (DM-SMR) […] The only clue you’ll get about these drives being SMR is the appalling sequential write speeds (~40MB/s from blank) and the fact that they report a “trim” function.
There has been speculation that the drives got kicked out of the arrays due to long timeouts—SMR disks need to perform garbage-collection routines in the background and store incoming writes in a small CMR-encoded write-cache area of the disk, before moving them to the main SMR encoded storage. It’s possible that long periods of time with no new writes accepted triggered failure-detection routines that marked the disk as bad.
Blocksandfiles covered the issue extensively ….
ixSystems, the guys behind TrueNAS, also comment on the issue at that time.
The intial response from WD indicated that SMR was only used in new 20TB enterprise drives. But WD had 14TB and 20TB drives with SMR written on the labels since 2018.
WD then responded with:
All our WD Red drives are designed to meet or exceed the performance requirements and specifications for common and intended small business/home NAS workloads. WD Red capacities 2TB-6TB currently employ device-managed shingled magnetic recording (DMSMR) to maximize areal density and capacity. WD Red 8-14TB drives use conventional magnetic recording (CMR).
And:
In our testing of WD Red drives, we have not found RAID rebuild issues due to SMR technology.
It was already quite clear that “meet or exceed” was a bit of an overstatement. The rebuild performance differential is huge – ARSTechnica recently did some tests confirming this.
WD is not the only manufacturer using SMR, but they appeared to be the only one using SMR in an area where it’s not typically used – low-end NAS drives.
Seagate clarified at the time that all their IronWolf and IronWolf Pro drives used CMR …
WD have since come clean and updated their documentation to reflect whether a drives uses CMR or SMR. But buyer beware – many buyers will be unaware of this issue and could run into problems when using the SMR drives in RAID setups.
Long story short? Take extra care when purchasing (especially replacement) drives for NAS/RAID systems as an SMR drive could at best reduce performance significantly and at worst, cause a RAID rebuild to fail.