Storage drive durability and speed

The durability of a given drive model is generally not a concern for traditional spinning drives, claims about enterprise versus consumer drives notwithstanding. Flash / SSD drives do, however, vary considerably. In the past, conventional wisdom held that Single-Level Cell (SLC) flash devices would last longer than denser/less-expensive Multi-Level Cell (MLC) models, though more recently this has been questioned. So long as you avoid the lowest-end consumer/desktop SATA drives, this distinction should not be a significant concern.

Modern enterprise-class drives incorporate greater over-provisioning, which means that a fraction of the storage cells are reserved so that the drive's on-board controller can more effectively balance writes evenly across storage cells, and as needed bring spare blocks into service over time to replace aging sectors. For example, you might see an inexpensive consumer drive advertised as 480 GB, but an enterprise model from the same manufacturer rated at only 400 GB.

Even among enterprise SSDs, the durability—the number of data writes of the device's entire capacity is rated for over its lifetime—can vary considerably. This is often measured in Drive Writes Per Day (DWPD)—over the drive's rated lifetime, which is often five-years but sometimes as little as three years. Say a 1TB device has a five year warranty and is rated at 3.65 PB of writes over its lifetime. This works out to ~ 2 DWPD.

Consider this example of an SSD product line with three tiers.

Tier 1:

Tier 2:

Tier 3:

You can see that durability within even a single product line can vary by a factor of more than 30, and that speeds vary considerably as well. Attend carefully to both ratings especially for drives intended for journal use, as they will dictate how many OSD journal partitions each can host before becoming a bottleneck. As a rule of thumb, divide the journal device's write rating by the OSD device's write rating. For example, if your journal device is rated at writing 1GB/s, and your OSDs are rated at 100MB/s, you would not want to pack more than 10 journals onto each journal device. As journals are relatively small, at most a set-and-forget size of 10GB each, it is not uncommon to run out of speed before capacity.

Choosing a durability target for your cluster's journal and OSD devices can be complex and rely on difficult-to-predict future traffic. This author's personal philosophy is that the peace of mind gained from top-tier durability tends to outweigh the incremental cost of upselling from a middle-tier, but your constraints may vary. Many clusters do fine with 3 DWPD-class drives, but this author would not recommend going any lower on the price/durability curve. Note also that SSDs used for Ceph Monitor DBs can easily experience 2 DPWD or more in a cluster with hundreds of OSDs. For this reason, you are strongly encouraged to provision MON SSDs rated at least 3 DWPD, with 10 DWPD better yet. The authors have experienced multiple near-simultaneous MON failures due to lesser drives wearing out; we want you to avoid suffering the same anguish.

Circumstances sometimes force us to deploy pre-loved servers. In the past, used HDDs could be an ongoing headache, but today's drives enjoy substantially improved longevity. Used SSDs, however, can be a (silently) ticking time bomb. At the very least, you would need to examine each drive's remaining lifetime counter with a tool such as smartctl or isdct to avoid a (literal) rude awakening.