{"id":394,"date":"2019-10-08T08:04:44","date_gmt":"2019-10-08T15:04:44","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=394"},"modified":"2019-10-08T11:33:31","modified_gmt":"2019-10-08T18:33:31","slug":"using-a-api-management-in-front-of-an-azure-kubernetes-cluster","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2019\/10\/08\/using-a-api-management-in-front-of-an-azure-kubernetes-cluster\/","title":{"rendered":"Using a API Management in front of an Azure Kubernetes cluster"},"content":{"rendered":"\n

I have had the idea about writing about container and kubernetes networking for a while. Every time I dove in the research, I started looking at the networking aspects from a bottom-up lens. This means, starting at the Docker layer, moving into IPtables and into Kubernetes networking.<\/p>\n\n\n\n

What I learned about\ncustomer conversations about Kubernetes networking – and more specifically AKS\nnetworking – is that most people deal with top-down questions (e.g. how to\nintegrate AKS with API management) rather than bottom-up questions (what do all\nthese IP-tables rules mean).<\/p>\n\n\n\n

The impetus for me publishing this post is a customer call that one of my colleagues Moonis Tahir<\/a> pulled me into last week. A customer had some questions about integrating their AKS cluster with API Management and the Application Gateway. This has nothing to do with docker, iptables, and not even with the discussion around basic vs advanced networking in AKS. This has to do with how does a kubernetes cluster and its resources present themselves in a customer scenario, and how can the surrounding infrastructure integrate with that cluster.<\/p>\n\n\n\n

During this post I hope to first introduce you to a couple of concepts around Kubernetes networking and how to present your services (or applications) outside of the AKS cluster. Afterwards we’ll put this into practice by publishing an API through API management. All of this done as much as possible through Terraform. I read that it should be possible to even deploy your Kubernetes deployments and services using Terraform, and I want to give that a spin. <\/p>\n\n\n\n

I would also like to\ntouch on how to integrate the Application Gateway with AKS, and I’ll reserve\nthat right for a follow-up post.<\/p>\n\n\n\n

All code for this blog post can be found here<\/a>.<\/p>\n\n\n\n

With that, let’s get started with a couple of concepts:<\/p>\n\n\n\n

Kubernetes Services<\/h2>\n\n\n\n

In this section of the post, we’ll take at why it’s a bad idea to route traffic directly to pods and why you should use a Kubernetes service. We’ll also look into how we can publish a service privately in AKS. <\/p>\n\n\n\n

A deployment in Kubernetes is a cluster managed resource that will create and manage the containers or pods that your application requires. A deployment will create a ReplicaSet, which will in turn make sure the right amount of pods is always present for your application. If a pod dies, or a host dies and pods running on it dies, the Replicaset will ensure that your deployment will match its desired state and will create new pods on a new host.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This has impact on the networking aspects of your application. While it is feasible to connect directly to your pods, this is typically not something you should be planning on. A pod is an ephemeral resource, meaning it might (and will) not be persistent in your cluster, its artifacts might change. Practically speaking, a pod might be running on Host A today, and be rescheduled to run on Host B tomorrow – meaning it’s IP address will change. <\/p>\n\n\n\n

\"\"
A pod is an ephemeral resource. The picture shows Pod-1 running on Host A. When host A dies – Kubernetes will reschedule Pod-1 to a healthy host, which will change the IP address of that pod. <\/figcaption><\/figure>\n\n\n\n

To solve this challenge, Kubernetes has an object called a service. A service is a network level abstraction on top of the pods in your deployment. Where I mentioned before that you typically do not directly connect to your pods, you do connect to a service in Kubernetes.<\/p>\n\n\n\n

In practice, a service is nothing more than a load balancer and a couple of NAT rules. On a cloud provider’s platform (this is not AKS specific), when you deploy a service, Kubernetes will actually deploy a load balancer from that cloud provider (an Azure Load Balancer in the AKS case). This load balancer will do health checks and will perform the actual balancing of the load between your back-end nodes.<\/p>\n\n\n\n

By default on AKS, this will create a load balancer with an external IP address. This is ideal to publish your service to the outside world. This is however not a desired outcome if you want to publish your API through API management as we are intending to do today. With AKS, you have the option to publish your service to an internal load balancer through the use of a kubernetes annotation. The following code snippet will create a service that is ‘fronted’ by an internal load balancer:<\/p>\n\n\n\n

apiVersion: v1\nkind: Service\nmetadata:\n  name: internal-app\n  annotations:\n    service.beta.kubernetes.io\/azure-load-balancer-internal: \"true\"\nspec:\n  type: LoadBalancer\n  ports:\n  - port: 80\n  selector:\n    app: internal-app<\/code><\/pre>\n\n\n\n
This means that our service will only be available to other resources in the private address space that our Kubernetes cluster is deployed in. In the case of API Management that works out well, as we have the ability to deploy both our Kubernetes cluster as well as our API management instance into a VNET. Looking at this graphically, the following two pictures depict the architecture we’ll build in the next. If you’re wondering why two different visualizations, check out my blog post about PlantUML<\/a>.<\/p>\n\n\n\n
\"\"
The architecture we’ll be building in our next section. We’ll build an API Management instance, in front of a service running on AKS. The service on AKS will create an internal load balancer.<\/figcaption><\/figure>\n\n\n\n
\"The
The high level architecture we’ll build in PlantUML.<\/figcaption><\/figure>\n\n\n\n
Putting API management in front of our Kubernetes cluster<\/h2>\n\n\n\n
API management is a\nmanaged service in Azure. It provides an API gateway as well as a developer\nportal. The service itself can be deployed in a VNET, although only the\ndeveloper and premium SKU support being deployed in a VNET. Because our use\ncase here is just development, and we don’t need an SLA or support, we’ll build\nthis demo using the development SKU. If you’re building this for a production\nscenario, I highly recommend the premium SKU, as this gives you both an SLA as\nwell as support.<\/p>\n\n\n\n
As we’re diving into the deployment, I want to deploy everything with Terraform. In previous posts you might picked up that I’m getting more and more fond of the tool. And I’ve read as well that you can control deployments in your Kubernetes cluster through Terraform, which I have never been able try out, so here goes. <\/p>\n\n\n\n
Let’s do this\ndeployment and development iteratively, meaning we’ll want to break things up:<\/p>\n\n\n\n
  1. Create our support infrastructure: a VNET with 2 subnets.<\/li>
  2. Deploy an AKS cluster<\/li>
  3. Deploy an internal application to that AKS cluster<\/li>
  4. Deploy API management<\/li>
  5. Deploy our API definition to API management. <\/li><\/ol>\n\n\n\n
    Before diving into this however, I did some preparation work, and I discovered that the Terraform Provider for Azure does not yet support VNET deployments for API Management.  This PR on Github<\/a> discusses some of the details. I’m going to be persistent and do as much as possible in Terraform – as I should be able to create the API Management instance and integrate it into a VNET after deployment manually. At least I hope so.<\/em><\/p>\n\n\n\n
    All code for this blog post can be found here<\/a>.<\/p>\n\n\n\n
    Step 1: Create a VNET with 2 subnets<\/h3>\n\n\n\n
    Let’s start our deployment with a VNET and 2 subnets in it. This is relatively straightforward Terraform code:<\/p>\n\n\n\n
    resource \"azurerm_resource_group\" \"apim-aks\" {\n  name     = \"blog-apim-and-aks\"\n  location = \"WestUS2\"\n}\n\nresource \"azurerm_virtual_network\" \"apim-aks\" {\n  name                = \"apim-aks-vnet\"\n  address_space       = [\"10.10.0.0\/16\"]\n  location            = \"${azurerm_resource_group.apim-aks.location}\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n}\n\nresource \"azurerm_subnet\" \"aks\" {\n  name                 = \"aks-subnet\"\n  resource_group_name  = \"${azurerm_resource_group.apim-aks.name}\"\n  virtual_network_name = \"${azurerm_virtual_network.apim-aks.name}\"\n  address_prefix       = \"10.10.1.0\/24\"\n}\n\nresource \"azurerm_subnet\" \"apim\" {\n  name                 = \"apim-subnet\"\n  resource_group_name  = \"${azurerm_resource_group.apim-aks.name}\"\n  virtual_network_name = \"${azurerm_virtual_network.apim-aks.name}\"\n  address_prefix       = \"10.10.2.0\/24\"\n}<\/code><\/pre>\n\n\n\n
    If you are not familiar with Terraform, you’ll need three commands to start your deployment:<\/p>\n\n\n\n
    terraform init #this will download the Azure provider, only needed the first time you run terraform on this directory\nterraform plan \nterraform apply<\/code><\/pre>\n\n\n\n
    Step 2: creating an AKS cluster<\/h3>\n\n\n\n
    When creating an AKS cluster, you need a service principal. This SP is used by AKS to create a new RG and deploy the necessary AKS resources (nodes, disks, load balancers …). I decided to give this a try as well in Terraform, instead of re-using an existing one I had. The following code creates a service principal, with a random password. Note: If you’re doing this in a production scenario, be careful with how you manage secret strings in Terraform. My type random_string is actually readable, and stored in the tfstate file. This is not what you would want for a production scenario. Key vault might be a better solution to store the password. As I’m just developing, I don’t care that much – but you should certainly for a production scenario as you don’t want your keys to leak.<\/em><\/p>\n\n\n\n
    resource \"random_string\" \"sp-password\" {\n  length = 16\n  special = true\n  override_special = \"\/@\u00a3$\"\n}\n\nresource \"azuread_application\" \"aksapim\" {\n  name                       = \"nf-aksapim-sp\"\n  homepage                   = \"https:\/\/homepage\"\n  reply_urls                 = [\"https:\/\/replyurl\"]\n  available_to_other_tenants = false\n  oauth2_allow_implicit_flow = true\n}\n\nresource \"azuread_service_principal\" \"aksapim\" {\n  application_id = \"${azuread_application.aksapim.application_id}\"\n}\n\nresource \"azuread_service_principal_password\" \"aksapim\" {\n  service_principal_id = \"${azuread_service_principal.aksapim.id}\"\n  value                = \"${random_string.sp-password.result}\"\n  end_date             = \"2020-01-01T01:02:03Z\"\n}<\/code><\/pre>\n\n\n\n
    With our SP created (remember, terraform plan, terraform apply<\/code>) – we can go ahead and spin up our kubernetes cluster with the following Terraform template:<\/p>\n\n\n\n
    resource \"azurerm_kubernetes_cluster\" \"test\" {\n  name                = \"aks-for-apim\"\n  location            = \"${azurerm_resource_group.apim-aks.location}\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n  dns_prefix          = \"nfaksapim\"\n\n  agent_pool_profile {\n    name            = \"pool1\"\n    count           = 1\n    vm_size         = \"Standard_D2_v2\"\n    os_type         = \"Linux\"\n    os_disk_size_gb = 30\n    type = \"VirtualMachineScaleSets\"\n    vnet_subnet_id = \"${azurerm_subnet.aks.id}\"\n  }\n\n  service_principal {\n    client_id     = \"${azuread_application.aksapim.application_id}\"\n    client_secret = \"${random_string.sp-password.result}\"\n  }\n\n}\n\noutput \"client_certificate\" {\n  value = \"${azurerm_kubernetes_cluster.test.kube_config.0.client_certificate}\"\n}\n\noutput \"kube_config\" {\n  value = \"${azurerm_kubernetes_cluster.test.kube_config_raw}\"\n}<\/code><\/pre>\n\n\n\n
    Give this about 5 minutes. We can then get our kubeconfig – and store it on our system, and connect to our cluster.<\/p>\n\n\n\n
    echo \"$(terraform output kube_config)\" > .\/azurek8s\nexport KUBECONFIG=.\/azurek8s\nkubectl get nodes<\/code><\/pre>\n\n\n\n
    This should show you our one node in the cluster. And with that we can move on to deploy a simple app on our AKS cluster<\/p>\n\n\n\n
    Step 3: Deploying our app on AKS<\/h2>\n\n\n\n
    Let’s start with testing out the Terraform and Kubernetes integration. We’ll add a new file to our Terraform directory, including a simple pod deployed on Kubernetes.<\/p>\n\n\n\n
    provider \"kubernetes\" {\n    \n}\n\nresource \"kubernetes_pod\" \"nginx\" {\n  metadata {\n    name = \"nginx-example\"\n    labels = {\n      App = \"nginx\"\n    }\n  }\n\n  spec {\n    container {\n      image = \"nginx:1.7.8\"\n      name  = \"example\"\n\n      port {\n        container_port = 80\n      }\n    }\n  }\n}<\/code><\/pre>\n\n\n\n
    We’ll deploy this using:<\/p>\n\n\n\n
    terraform init #needed to download kubernetes provider\nterraform plan -out kubernetes.out\nterraform deploy kubernetes.out<\/code><\/pre>\n\n\n\n
    If we do a kubectl get pods<\/code>, we should see that pod appear. That being said, we won’t be deploying pods, so we’ll remove that part from our Terraform definition. We’ll instead deploy a deployment and a service over an internal load balancer.<\/p>\n\n\n\n
    As I was developing this, I hit an issue with the service, which was unable to create the load balancer in the subnet. I don’t want to bore you with the investigation – let me share the solution: Our service principal had rights on the managed resource group that AKS created, but not on the VNET that it is deploying in. Add the following snippet of Terraform to your AKS definition to assign those permissions.<\/p>\n\n\n\n
    resource \"azurerm_role_assignment\" \"test\" {\n  scope                = \"${azurerm_resource_group.apim-aks.id}\"\n  role_definition_name = \"Contributor\"\n  principal_id         = \"${azuread_application.aksapim.application_id}\"\n}<\/code><\/pre>\n\n\n\n
    This will create a role assignment, to give our service principal access to the resource group that contains our VNET. <\/p>\n\n\n\n
    The following code will then go ahead and create a deployment and a service for you:<\/p>\n\n\n\n
    provider \"kubernetes\" {\n\n}\n\nresource \"kubernetes_deployment\" \"example\" {\n  metadata {\n    name = \"private-api\"\n    labels = {\n      app = \"private-api\"\n    }\n  }\n  spec {\n    replicas = 2\n    selector {\n      match_labels = {\n        app = \"private-api\"\n      }\n    }\n    template {\n      metadata {\n        labels = {\n          app = \"private-api\"\n        }\n      }\n      spec {\n        container {\n          image = \"nginx:1.7.8\"\n          name  = \"private-api\"\n\n          resources {\n            limits {\n              cpu    = \"0.5\"\n              memory = \"512Mi\"\n            }\n            requests {\n              cpu    = \"250m\"\n              memory = \"50Mi\"\n            }\n          }\n        }\n      }\n    }\n  }\n}\n\nresource \"kubernetes_service\" \"example\" {\n  metadata {\n    name = \"private-api\"\n    annotations = {\n        \"service.beta.kubernetes.io\/azure-load-balancer-internal\" = \"true\"\n    }\n  }\n  spec {\n    selector = {\n      app = \"private-api\"\n    }\n\n    port {\n      port        = 80\n      target_port = 80\n    }\n\n    type = \"LoadBalancer\"\n  }\n}<\/code><\/pre>\n\n\n\n
    If you then do a kubectl get svc <\/code>to show your services, you should see the service appear with a private ip as an external IP.<\/p>\n\n\n\n
    AME          TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE\nkubernetes    ClusterIP      10.0.0.1       <none>        443\/TCP        4h9m\nprivate-api   LoadBalancer   10.0.177.167   10.10.1.5     80:30140\/TCP   8m14s<\/code><\/pre>\n\n\n\n
    So, with that out of the way, we can go ahead and add an API management to our VNET.<\/p>\n\n\n\n
    Step 4: Add API Management to our VNET<\/h3>\n\n\n\n
    If you remember from earlier, we’ll deploy our API management instance; but then we’ll need to manually add it to our VNET. So, let’s start off with step 1: adding API management to our deployment. This will take a while, so while this is deploying either grab a cup of coffee, or continue with some other the prep steps for the next step.<\/p>\n\n\n\n
    resource \"azurerm_api_management\" \"apim\" {\n  name                = \"blog-apim\"\n  location            = \"${azurerm_resource_group.apim-aks.location}\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n  publisher_name      = \"Nills\"\n  publisher_email     = \"nilfran@microsoft.com\"\n\n  sku {\n    name     = \"Developer\"\n    capacity = 1\n  }\n}<\/code><\/pre>\n\n\n\n
    Disclaimer: Don’t add API Management to your VNET just yet. Doing it now, will be reversed once we add our API through Terraform. This is a bug<\/a> in the Terraform provider for Azure. <\/em><\/p>\n\n\n\n
    Step 5: Defining our API<\/h2>\n\n\n\n
    The structure within API Management is the following: <\/p>\n\n\n\n
    • From a technical lens, you define your API as an API in APIM. You define each call with its settings. Within this API, you can define multiple operations.<\/li>
    • You need to define a product, so end-users can subscribe to your API.<\/li>
    • To give those end-users access to your API, you need to link an API to a product.<\/li><\/ul>\n\n\n\n
      This looks like this in Terraform:<\/p>\n\n\n\n
      resource \"azurerm_api_management\" \"apim\" {\n  name                = \"blog-apim\"\n  location            = \"${azurerm_resource_group.apim-aks.location}\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n  publisher_name      = \"Nills\"\n  publisher_email     = \"nilfran@microsoft.com\"\n\n  sku {\n    name     = \"Developer\"\n    capacity = 1\n  }\n}\n\nresource \"azurerm_api_management_api\" \"back-end-api\" {\n  name                = \"example-api\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n  api_management_name = \"${azurerm_api_management.apim.name}\"\n  revision            = \"1\"\n  display_name        = \"Example API\"\n  path                = \"nginx\"\n  service_url          = \"http:\/\/${kubernetes_service.example.load_balancer_ingress.0.ip}\"\n  protocols           = [\"http\"]\n}\n\nresource \"azurerm_api_management_api_operation\" \"get\" {\n  operation_id        = \"get\"\n  api_name            = \"${azurerm_api_management_api.back-end-api.name}\"\n  api_management_name = \"${azurerm_api_management.apim.name}\"\n  resource_group_name = \"${azurerm_resource_group.apim-aks.name}\"\n  display_name        = \"get\"\n  method              = \"GET\"\n  url_template        = \"\/\"\n\n  response {\n    status_code = 200\n  }\n}\n\nresource \"azurerm_api_management_product\" \"product\" {\n  product_id            = \"nginx\"\n  api_management_name   = \"${azurerm_api_management.apim.name}\"\n  resource_group_name   = \"${azurerm_resource_group.apim-aks.name}\"\n  display_name          = \"Test Product\"\n  subscription_required = false \n  published             = true\n}\n\nresource \"azurerm_api_management_product_api\" \"example\" {\n  api_name            = \"${azurerm_api_management_api.back-end-api.name}\"\n  product_id          = \"${azurerm_api_management_product.product.product_id}\"\n  api_management_name = \"${azurerm_api_management.apim.name}\"\n  resource_group_name = \"${azurerm_api_management.apim.resource_group_name}\"\n}<\/code><\/pre>\n\n\n\n
       We can apply that using a terraform plan -out apim.out<\/code> and terraform apply apim.out<\/code>. This shouldn’t take long. What will take long, is finally integrating into our VNET, as we’ll do now.<\/p>\n\n\n\n
      We can integrate APIM manually into the VNET. In the APIM blade, search for ‘network’ and open the VNET blade:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      Then go ahead and point the VNET to the VNET we created already:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      And now, we wait again – as this change will take a while as well.<\/p>\n\n\n\n
      Once this change has been performed, we should be able to access our Nginx page through API Management. <\/p>\n\n\n\n
      \"\"
      Our back-end API is now available through API Management.<\/figcaption><\/figure>\n\n\n\n
      Observations and learning in this experience<\/h2>\n\n\n\n
      I actually learned a couple of things when writing this blog.<\/p>\n\n\n\n
      • I don’t like defining Kubernetes resources in Terraform. This is the first time I did this, and I actually prefer using the native YAML definition language in Kubernetes to the HCL (json-like) syntax in Terraform. It might be personal, but I found it less productive.<\/li>
      • Using Terraform to deploy API Management doesn’t work with VNETs. Defining APIs and products worked pretty well, but integrating into a VNET doesn’t work as expected. It even reverts manually adding your APIM to a VNET. But this is a bug that is reported, so it can only get better.<\/li>
      • I also learned – although a details – that different services in Azure use different IDs for either the application ID or service principal ID. The templates in this post include the right IDs.<\/li><\/ul>\n\n\n\n
        In closing<\/h2>\n\n\n\n
        The goal of this post, was to describe how you can integrate a network appliance in front of an AKS cluster by keeping your Kubernetes deployments private. We demonstrated this using API Management in front of our AKS cluster. We deployed everything using Terraform, and only used the portal once.<\/p>\n\n\n\n
        <\/p>\n","protected":false},"excerpt":{"rendered":"
        I have had the idea about writing about container and kubernetes networking for a while. Every time I dove in the research, I started looking at the networking aspects from a bottom-up lens. This means, starting at the Docker layer, moving into IPtables and into Kubernetes networking. What I learned about customer conversations about Kubernetes […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,36],"tags":[37,39,18,38,15],"class_list":["post-394","post","type-post","status-publish","format-standard","hentry","category-azure","category-networking","tag-aks","tag-api-management","tag-kubernetes","tag-networking","tag-terraform"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/394","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=394"}],"version-history":[{"count":8,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/394\/revisions"}],"predecessor-version":[{"id":434,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/394\/revisions\/434"}],"wp:attachment":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=394"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}