{"id":1175,"date":"2020-08-04T15:40:07","date_gmt":"2020-08-04T22:40:07","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=1175"},"modified":"2020-08-10T11:04:42","modified_gmt":"2020-08-10T18:04:42","slug":"using-nfs-on-azure-blob-and-some-performance-tests","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2020\/08\/04\/using-nfs-on-azure-blob-and-some-performance-tests\/","title":{"rendered":"Using NFS on Azure blob and some performance tests"},"content":{"rendered":"\n

Recently, Microsoft announced<\/a> that they’re enabling the NFS protocol on top of blob storage. This is growing the strategy that started with ADLS gen 2, which put the HDFS protocol on top of blob storage. <\/p>\n\n\n\n

I am working on a number of projects where this support would be helpful for customers. It’s still early days for the NFS support, so it’s not an immediate help, but I wanted to give it a try.<\/p>\n\n\n\n

In this post, I’ll explain how to set it up, and then I’ll take you through some intensive performance testing I did. This blog post might be updated in the future, <\/p>\n\n\n\n

Some technical details<\/h2>\n\n\n\n

NFS on blob uses the NFS v3 protocol. This has one impact: NFS v3 is not encrypted. The Azure storage team knows about this, and will – to protect users – only allow NFS access over either a service endpoint<\/a> or private link<\/a>.<\/p>\n\n\n\n

Right now, the access to NFS is either NFS or blob on a storage account. This means you can’t (like ADLS gen 2) use blob and NFS on the same account. Based on the Azure documentation, it seems like the plan is to support this multi-protocol write. <\/p>\n\n\n\n

\"\"
Some of the features not yet supported.<\/figcaption><\/figure>\n\n\n\n

For now, it’s also important to note that you cannot enable\/disable NFS on an existing storage account. Which means you’ll have to create a new one, dedicated to NFS.<\/p>\n\n\n\n

During the preview, NFS 3.0 is available to BlockBlobStorage accounts with premium performance in the following regions: US East, US Central, and Canada Central. Support for GPV2 accounts with standard tier performance will be announced soon<\/em>. <\/p>\n\n\n\n

Pre-setup<\/h2>\n\n\n\n

Before we setup a new storage account, we need to register the feature. This requires three commands (using Azure PowerShell):<\/p>\n\n\n\n

Register-AzProviderFeature -FeatureName AllowNFSV3 -ProviderNamespace Microsoft.Storage \nRegister-AzProviderFeature -FeatureName PremiumHns -ProviderNamespace Microsoft.Storage  \nRegister-AzResourceProvider -ProviderNamespace Microsoft.Storage<\/code><\/pre>\n\n\n\n
This registration process will take a couple of minutes. I believe that in my case it took about 10 minutes to get the provider registered.<\/p>\n\n\n\n
Since I have no resources in any of the preview regions, I decided to spin up a new VNET in East US with a Linux VM.<\/p>\n\n\n\n
Creating an NFS blob storage account<\/h2>\n\n\n\n
To create a new storage account, I’ll use the Azure portal. Since right now this is only support on BlockBlobStorage accounts with premium performance, we’ll create such a storage account.<\/p>\n\n\n\n
\"\"
Creating a new storage account<\/figcaption><\/figure>\n\n\n\n
In terms of networking – remember, this is required for NFS – I’ll use regular public service endpoints. Private endpoints are also supported, but I decided to go with the public variant for my testing.<\/p>\n\n\n\n
\"\"
Setting up networking<\/figcaption><\/figure>\n\n\n\n
I skipped the data protection tab, and enabled the hierarchical namespace and NFS v3 support in the Advanced tab.<\/p>\n\n\n\n
\"\"
Enabling NFS<\/figcaption><\/figure>\n\n\n\n
And with that, we can create the storage account.<\/p>\n\n\n\n
\"\"
Review and create<\/figcaption><\/figure>\n\n\n\n
Finally, once the account is created, we’ll need to create a container on this account. I created a container called ‘nillstest’ on my account.<\/p>\n\n\n\n
Mounting blob via NFS in a VM<\/h2>\n\n\n\n
Mounting blob via NFS in a VM was very easy. First step was to install nfs-common,<\/code> second step was creating a directory to mount into, final step was mounting. That was it.<\/p>\n\n\n\n
#install nfs-common\nsudo apt-get update\nsudo apt-get install nfs-common -y\n\n#create mount point\nsudo mkdir \/mnt\/blob\n\n#mount\nmount -o sec=sys,vers=3,nolock,proto=tcp nfblobnfs.blob.core.windows.net:\/nfblobnfs\/nillstest  \/mnt\/blob<\/code><\/pre>\n\n\n\n
And that seemed to do the job.<\/p>\n\n\n\n
I wanted to also test the performance of this new feature, so that’s what we’ll cover next.<\/p>\n\n\n\n
Testing performance<\/h2>\n\n\n\n
Using dd<\/h3>\n\n\n\n
Let’s start with a couple of rudimentary performance tests using the dd <\/code>command. For reference, I’m running all of this on a Standard E16as_v4, which has 8Gbps of network throughput. I did a couple of regular dd <\/code>tests first.<\/p>\n\n\n\n
dd if=\/dev\/zero of=\/tmp\/test-1.img bs=1G count=1\ndd if=\/dev\/zero of=\/tmp\/test-4.img bs=8k count=100k<\/code><\/pre>\n\n\n\n
Both commands resulted in performance that was all over the map, anywhere between 15MB\/s and 250MB\/s. Not sure what was causing this, whether it was caching or throttling. Some of the results of these tests:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
As you can see in the screenshots above, the performance was all over the place.<\/p>\n\n\n\n
I’ve seen this before, dd <\/code>isn’t the most reliable way to do disk performance testing. I’ve also used fio <\/code>in the past, so I decided to give that a spin to get an idea of the performance.<\/p>\n\n\n\n
Using fio<\/h3>\n\n\n\n
As I mentioned in the previous section, dd<\/code> can deliver inconsistent test results. fio <\/code>is a more full featured io testing tool. To install fio<\/code>, you can run:<\/p>\n\n\n\n
sudo apt-get install fio -y<\/code><\/pre>\n\n\n\n
And we can then do a couple tests. First test is doing a sequential write test using synchronous IO (i.e. without OS level caching) with a 32k block size:<\/p>\n\n\n\n
fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=32k --numjobs=4 --size=1G --runtime=300 --group_reporting<\/code><\/pre>\n\n\n\n
One thing to notice when running fio is that I’m getting an IO error, which actually isn’t blocking the tests itself:<\/p>\n\n\n\n
\"\"
Error about the native_fallocate call. This does not limit the test from running.<\/figcaption><\/figure>\n\n\n\n
Ignoring that error and looking at the performance, I could see performance at about 70 to 90 IOPS initially, dropping to about 30 IOPS after about a minute. On average the IO performance was 48 write IOPS Certainly not spectacular:<\/p>\n\n\n\n
\"\"
Doing a 32k direct IO write test results in an avg of 48 IOPS.<\/figcaption><\/figure>\n\n\n\n
This performance can be improved by adding the --iodepth<\/code> parameter to fio. This means that fio would use more parallel writers to achieve better performance. Using –iodepth=50, I was able to achieve 403IOPS using this 32k block size sequential write test.<\/p>\n\n\n\n
\"\"
Using a higher iodepth increased write performance to 403IOPS.<\/figcaption><\/figure>\n\n\n\n
To offer some comparison, I ran the same fio tests on the temporary drive of the VM (also called ephemeral drive), and was able to achieve the following results:<\/p>\n\n\n\n