IOPS Limit – vSAN & Virtual Hard Disk

Jeff Wong
HCI

Occasionally, I will have a technical conversation with a customer on IOPS limit so rather than repeating myself, I thought I should document it here for everyone’s benefit. Cormac Hogan had written up a good introduction to vSAN IOPS Limit storage policy that was introduced back in vSAN 6.2.

Beyond, the feature, I thought I should illustrate IOPS limit in action and highlight use cases when you would potentially want to use it.

My first test is a single HCIbench VM with an IO profile of 8K IO size, 100% random, 50% read, 50% write against a 50gb virtual drive.

First, I would like to point out that the standard vSphere IOPS Limit within the virtual machine properties do not apply when the virtual disk resides on vSAN. To prove this, I applied a 500 IOPS limit on the 50gb virtual drive after the 5 minutes of artificial IO applied stabilizes at about 5800 IOPS (2900 reads, and 2900 writes) and it shows no effect.

Virtual hard disk IOPS limit property

Virtual Hard disk property IOPS Limit has no effect on vSAN backed disks.

Next, to illustrate the real IOPS limit taking effect when applying with a vSAN storage policy. The illustration below shows a policy named ‘IOPS Limit – 800’ is applied with a setting of IOPS limit for object of 800. The graph below shows a total IOPS limit applied of 800 (400 read + 400 write).

So then the common question is, when would you want to use such policy setting? Don’t admins want the best performance for all workloads when possible? Well, like any storage solution, there is a finite amount of performance you can get out of it and vSAN is no different. Possible use cases are…

  1. For service providers or tiering reason. You may want to limit performance to a tier your customer is willing to pay for to ensure better quality of service for your higher paying customers. ie. a bronze, silver, gold service may offer different IOPS limits.

  2. Avoid noisy neighbors. You may have a VM that has very high uncontrollable IO when a batch job runs in a shared environment. When this happens, other workloads may be impacted and you want a solution to minimize the impact by limiting the number of IOPS it can use.

  3. Minimize cache exhaustion. In certain scenario, a vSAN disk group cache may overfill by certain workloads. When this happens just like any cache backed storage solution, it can have negative effects on other workloads utilizing the same cache. IOPS limits can protect from such situations until a longer term solution is in place such as larger or faster cache device.

A reminder that this storage policy can be applied on the fly live on a running workload and the settings would take affect immediately. This is another example of the power of storage policy based management!