{"id":1540,"date":"2021-01-31T15:34:49","date_gmt":"2021-01-31T23:34:49","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=1540"},"modified":"2021-02-04T11:24:54","modified_gmt":"2021-02-04T19:24:54","slug":"key-vault-csi-driver-integrated-with-aad-managed-pod-identities","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2021\/01\/31\/key-vault-csi-driver-integrated-with-aad-managed-pod-identities\/","title":{"rendered":"Key Vault CSI driver integrated with AAD-managed pod identities"},"content":{"rendered":"\n

Tomorrow I’m joining the 425 show on Twitch<\/a> for a discussion on AAD-managed pod identities and how to integrate them with Key Vault. This blog post is both a sneak preview of what we’ll be discussing tomorrow as well as a review of the episode if you want to study things more in-depth after it aired. Update<\/strong>: the video is now also available on YouTube<\/a>.<\/p>\n\n\n\n

So, in this post, I’ll introduce AAD-managed pod identities, and later on, show you how to integrate them with the Key Vault CSI driver for Kubernetes secrets. All the code for this blog post is hosted on GitHub<\/a>: <\/p>\n\n\n\n

Let’s start by explaining AAD-managed pod identities<\/p>\n\n\n\n

AAD-managed pod identities<\/h2>\n\n\n\n

I discussed AAD-managed pod identities in a previous post already<\/a>. You can check that out as well, but I’ll provide another overview here. <\/p>\n\n\n\n

AAD-managed pod identities allow you to assign Azure managed identities to Pods in Kubernetes. From an Azure PoV, managed identities are assigned to “compute” resources, such VMs, Functions or even to a Data Factory. Azure itself is totally unaware of what runs on those VMs, and Azure doesn’t have the concept of a Pod.<\/p>\n\n\n\n

Managed identity works by allowing a compute resource (let’s say a VM) to get an Azure AD token without authenticating with a password or service principal by calling an internal endpoint. This endpoint will then call out to Azure AD and get the identity token.<\/p>\n\n\n\n

\"\"
Managed Identity on a typical compute resource<\/figcaption><\/figure>\n\n\n\n

What this means in Kubernetes land (running in a VM) is that each Pod running on a VM can call the managed identity endpoint and get the identity token. This is typically not what you’d want, since you can be running multiple applications, managed by different teams on the same cluster.<\/p>\n\n\n\n

This is where AAD-managed pod identity comes in. It provides two features: first, it influences the networking on your AKS cluster, intercepting all requests to the managed identity endpoint and sending those requests to a DaemonSet running on your cluster. This Daemonset (called Node Managed Identity or NMI for short) provides the second piece of functionality of AAD-managed pod identity, namely, it checks whether or not Pods should have access to certain identities. If they have access, the DaemonSet forwards those requests to the managed identity endpoint to get the token.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Finally, let’s do a little history lesson. AAD Pod identity originally was an open-source project, hosted on GitHub. You installed and maintained it on your cluster yourself, and it came with limited best-effort support. Recently, Microsoft made the same functionality available as a managed add-on on AKS, providing more stability and support for it.<\/p>\n\n\n\n

OK. Theory covered, let’s see it in action!<\/p>\n\n\n\n

AAD-managed pod identity in action<\/h3>\n\n\n\n

In this section, we’ll create a new AKS cluster, a new managed identity, a new blob storage account and we’ll configure everything in such a way that the managed identity is allowed to upload and download files from blob. To make things reproducible, I’ll provide you with the individual steps in AZ CLI, so it’s easy to rebuild (except for the preview registration which you see below). <\/p>\n\n\n\n

First, you need to register for the preview. You can do so this way<\/p>\n\n\n\n

az feature register --name EnablePodIdentityPreview  \\\n --namespace Microsoft.ContainerService\naz extension add --name aks-preview<\/code><\/pre>\n\n\n\n
Wait until the following command shows registered. It can take about 15 to 20 minutes. <\/p>\n\n\n\n
az feature show --name EnablePodIdentityPreview \\\n  --namespace Microsoft.ContainerService -o table<\/code><\/pre>\n\n\n\n
Setting up the basics<\/h4>\n\n\n\n
Now you’re ready, let’s create a new cluster. The following AZ CLI can be used to create a new cluster using AAD managed pod identity:<\/p>\n\n\n\n
RGNAME=425-aad-csi\nAKSNAME=425-aad-csi\nLOCATION=westus2\n\naz group create -n $RGNAME -l $LOCATION\naz aks create -g $RGNAME -n $AKSNAME \\\n  --enable-managed-identity --enable-pod-identity \\\n  --network-plugin azure --node-vm-size Standard_DS2_v2 \\\n  --node-count 2 --generate-ssh-keys\naz aks get-credentials -n $AKSNAME -g $RGNAME<\/code><\/pre>\n\n\n\n
Then, we’ll create a new managed identity, and a new blob storage account. In that storage account we’ll create a new container, and give the managed identity permissions to upload\/download files in the storage account:<\/p>\n\n\n\n
az identity create --name 425-blob-access \\\n  --resource-group $RGNAME\n\nSTACC=425csiaadnf #needs to be unique\naz storage account create \\\n  -n $STACC -g $RGNAME -l $LOCATION \\\n  --sku Standard_LRS\naz storage container create --name testing \\\n  --account-name $STACC --resource-group $RGNAME\n\nSCOPE=`az storage account show -n $STACC -g $RGNAME --query id -o tsv`\nASSIGNEE=`az identity show --name 425-blob-access \\\n  --resource-group $RGNAME --query clientId -o tsv`\n\naz role assignment create --role \"Storage Blob Data Contributor\" \\\n--assignee $ASSIGNEE --scope $SCOPE<\/code><\/pre>\n\n\n\n
Now, we can link the managed identity to the AKS cluster as well.<\/p>\n\n\n\n
RESOURCEID=`az identity show --name 425-blob-access \\\n  --resource-group $RGNAME --query id -o tsv`\naz aks pod-identity add --resource-group $RGNAME \\\n  --cluster-name $AKSNAME --namespace default \\\n  --name access-blob \\\n  --identity-resource-id $RESOURCEID<\/code><\/pre>\n\n\n\n
Testing things out<\/h4>\n\n\n\n
To test things out, we’ll create a new Pod with the Azure-CLI installed. We’ll then use this Pod to test out if we can access that blob storage account.<\/p>\n\n\n\n
apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: access-blob\nspec:\n  selector:\n    matchLabels:\n      app: access-blob\n  template:\n    metadata:\n      labels:\n        app: access-blob\n        aadpodidbinding: access-blob\n    spec:\n      containers:\n      - name: azure-cli\n        image: mcr.microsoft.com\/azure-cli\n        command: [ \"\/bin\/bash\", \"-c\", \"sleep inf\" ]<\/code><\/pre>\n\n\n\n
Notice the label here that contains aadpodidbinding: access-blob<\/code> – which is the link between the Pods that get created and the actual identity.<\/p>\n\n\n\n
Let’s create this using kubectl create -f deploy-with-id.yaml<\/code>. Give it a couple seconds, then list the pods, and use the following commands to test things out:<\/p>\n\n\n\n
echo $ASSIGNEE # You'll need this in a sec\nkubectl get pods # Get the full pod name\nkubectl exec -it <pod-name> -- sh\naz login -i -u <client-id> --allow-no-subscription -o table<\/code><\/pre>\n\n\n\n
This will lead to an output like this:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now that you’re logged in, try to create and upload a file:<\/p>\n\n\n\n
echo \"425 show is awesome\" > test.txt\naz storage blob upload -f test.txt -c testing \\\n  -n testfrompod.txt --auth-mode login \\\n  --account-name 425csiaadnf<\/code><\/pre>\n\n\n\n
This will create the blob on your storage account, as you can see below:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Let’s now also try if this would work without the label. Let’s create this as a new file deploy-without-id.yaml<\/code>.<\/p>\n\n\n\n
apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: no-access-blob\nspec:\n  selector:\n    matchLabels:\n      app: no-access-blob\n  template:\n    metadata:\n      labels:\n        app: no-access-blob\n    spec:\n      containers:\n      - name: azure-cli\n        image: mcr.microsoft.com\/azure-cli\n        command: [ \"\/bin\/bash\", \"-c\", \"sleep inf\" ]<\/code><\/pre>\n\n\n\n
Create this using kubectl create -f deploy-without-id.yaml<\/code>.<\/p>\n\n\n\n
You can try the same commands as earlier, which will fail with a message that the CLI didn’t get a token:<\/p>\n\n\n\n
kubectl get pods # Get the full pod name\nkubectl exec -it <pod-name> -- sh\naz login -i --allow-no-subscription -o table<\/code><\/pre>\n\n\n\n
Which – as expected – threw an error:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
OK, that covers how AAD-managed pod identities work. Let’s now see how we can get this to work with Key Vault and the CSI driver for Key Vault.<\/p>\n\n\n\n
CSI driver for Key Vault<\/h2>\n\n\n\n
I also covered the CSI driver for Key Vaul<\/a>t in the past, although without mentioning AAD-pod managed identity. Let’s start by discussing what the CSI driver for Key Vault does:<\/p>\n\n\n\n
The CSI driver for Key Vault (Azure Key Vault Provider for Secrets Store CSI Driver in full) is a way for you to mount Key Vault secrets in Kubernetes Pods. It’s based on a Kubernetes CSI driver for secrets that supports more than Azure alone. <\/p>\n\n\n\n
The way the driver works is by allowing you to mount Key Vault secrets as volumes in Pods. Optionally, you can also sync Key Vault secrets with Kubernetes Secret if you would which to do so. <\/p>\n\n\n\n
Why are we discussing the CSI driver and AAD-managed pod identities in the same blob post? Let me explain: The CSI driver needs an identity to access key vault. It could use any identity, let it be a service principal or a managed identity. In AKS, with the support for pod-managed identities, why not link this to pod-managed identities! <\/p>\n\n\n\n
So, with the final piece of theory out of the way, let’s build us a Key Vault and get rocking with the CSI driver for Key Vault:<\/p>\n\n\n\n
Setting up the basics, a Key Vault, a secret and Key Vault permissions<\/h3>\n\n\n\n
OK, let’s go ahead and create what we need. Let’s start by creating a Key Vault<\/p>\n\n\n\n
KVNAME=425nfcsi\naz keyvault create --location $LOCATION \\\n--name $KVNAME --resource-group $RGNAME<\/code><\/pre>\n\n\n\n
Now, let’s create a secret:<\/p>\n\n\n\n
az keyvault secret set --name secret-425 \\\n  --vault-name $KVNAME --value \"super secret for 425 show\"<\/code><\/pre>\n\n\n\n
And finally, our managed identity needs to get permissions to this secret. Let’s also set this up:<\/p>\n\n\n\n
az keyvault set-policy -n $KVNAME \\\n--secret-permissions get list --spn $ASSIGNEE<\/code><\/pre>\n\n\n\n
That covers the basics, let’s now setup the CSI driver.<\/p>\n\n\n\n
Setting up the CSI driver<\/h3>\n\n\n\n
You can install the CSI driver on your cluster using Helm. The installation is simple and straightforward:<\/p>\n\n\n\n
helm repo add csi-secrets-store-provider-azure https:\/\/raw.githubusercontent.com\/Azure\/secrets-store-csi-driver-provider-azure\/master\/charts\nhelm install csi csi-secrets-store-provider-azure\/csi-secrets-store-provider-azure<\/code><\/pre>\n\n\n\n
It literally is that simple. To confirm the install worked, check for the CRDs on your cluster and you should see the secretproviderclass CRDs using kubectl get crd<\/code>.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
OK, so now you’re ready to access the Secrets from Pods in Kubernetes!<\/p>\n\n\n\n
Accessing Key Vault secrets from a Kubernetes Pod<\/h3>\n\n\n\n
To link a Key Vault secret to a Kubernetes Pod you need to create an object called a SecretProviderClass. Once you have created a SecretProviderClass, you can then mount it in a pod. Let’s start by creating a SecretProviderClass:<\/p>\n\n\n\n
apiVersion: secrets-store.csi.x-k8s.io\/v1alpha1\nkind: SecretProviderClass\nmetadata:\n  name: key-vault-secret\nspec:\n  provider: azure\n  parameters:\n    usePodIdentity: \"true\"\n    keyvaultName: \"nf425csi\"\n    objects:  |\n      array:\n        - |\n          objectName: secret-425\n          objectType: secret\n    tenantId: \"72f988bf-86f1-41af-91ab-2d7cd011db47\"\n  secretObjects:\n  - secretName: key-vault-secret\n    type: Opaque\n    data:\n    - objectName: secret-425\n      key: secret-content<\/code><\/pre>\n\n\n\n
Couple of things to call out here:<\/p>\n\n\n\n
  1. We tell this SecretProviderClass we need it to use PodIdentity.<\/li>
  2. We give it the key vault name<\/li>
  3. We then provide it an array of secrets to grab. A single SecretProviderClass could grab multiple secrets from the same vault.<\/li>
  4. We then link this to a secret in Kubernetes. This is optional, but useful for some. Hence I included it in this demo.<\/li><\/ol>\n\n\n\n
    Create this using kubectl create spc.yaml<\/code>.<\/p>\n\n\n\n
    Now you’re ready to create the pod that will access this secret.<\/p>\n\n\n\n
    apiVersion: v1\nkind: Pod\nmetadata:\n  name: csi-demo\n  labels:\n    aadpodidbinding: access-blob\nspec:\n  containers:\n  - name: nginx\n    image: nginx\n    env:\n      - name: KEYVAULT_SECRET\n        valueFrom:\n          secretKeyRef:\n            name: key-vault-secret\n            key: secret-content\n    volumeMounts:\n      - name: keyvault\n        mountPath: \/mnt\/secrets-store\n        readOnly: true\n  volumes:\n    - name: keyvault\n      csi:\n        driver: secrets-store.csi.k8s.io\n        readOnly: true\n        volumeAttributes:\n          secretProviderClass: key-vault-secret<\/code><\/pre>\n\n\n\n
    Things to note here:<\/p>\n\n\n\n
    1. We’re accessing the synced Secret here as well. In the env: section you’ll see how we’re doing this. This is exactly the same as with a regular kubernetes secret.<\/li>
    2. Second, you’ll see we’re mounting the secrets in \/mnt\/secrets-store<\/li>
    3. Finally, in the volumes section, you’ll see how we’re referring to the secretProviderClass created earlier.<\/li><\/ol>\n\n\n\n
      Create this pod using: kubectl create -f pod-with-secret.yaml<\/code><\/p>\n\n\n\n
      Then, you can exec into this pod and get the secrets. <\/p>\n\n\n\n
      kubectl exec -it csi-demo -- sh\necho $KEYVAULT_SECRET\ncat \/mnt\/secrets-store\/secret-425<\/code><\/pre>\n\n\n\n
      And that concludes the demo.<\/p>\n\n\n\n
      Summary<\/h2>\n\n\n\n
      In this blog post, you learned how you can use AAD-managed pod identities and how you can integrate the CSI driver for Key Vault with those managed identities. This will allow you to securely managed and access secrets in Key Vault from Kubernetes.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Tomorrow I’m joining the 425 show on Twitch for a discussion on AAD-managed pod identities and how to integrate them with Key Vault. This blog post is both a sneak preview of what we’ll be discussing tomorrow as well as a review of the episode if you want to study things more in-depth after it […]<\/p>\n","protected":false},"author":1,"featured_media":1549,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2,3,58,5,85],"tags":[164,29,115,117,18,116,50],"class_list":["post-1540","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-devops","category-kubernetes","category-open-source","category-security","tag-azure-active-directory","tag-azure-ad","tag-azure-kubernetes-service","tag-key-vault","tag-kubernetes","tag-secrets","tag-security"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/01\/2021-01-31-15_33_15-PowerPoint-Slide-Show-Customize-core-dumps-in-Azure-Kubernetes.pptx-PowerP.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1540","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=1540"}],"version-history":[{"count":3,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1540\/revisions"}],"predecessor-version":[{"id":1552,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1540\/revisions\/1552"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/1549"}],"wp:attachment":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=1540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=1540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=1540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}