Integrating your a helm blue/green deployment into a CI/CD pipeline

nillsf Azure, DevOps, Kubernetes, Open Source

A while ago, I wrote about setting up a Kubernetes blue/green deployment using the service object in Kubernetes. All the work done there, was done editing yaml files and using direct kubectl commands, which you wouldn’t do in production. Later on, we updated those yaml files to Helm charts, so we can update with a helm upgrade.

Today, we’ll take this to the next level and execute updates via a CI/CD pipeline.

Let’s get started.

Environment we’ll use

I posted all the relevant files from my previous demo on Github. We’ll be using this repo to trigger (CI/)CD.

As a CI/CD tool I’m going to be using Azure pipelines. I haven’t tried out YAML pipelines before, so it’s a good time to finally give those a spin.

As a run-time, we’ll use an AKS cluster. If you went along with me and deployed the blue/green app in a cluster and still have them running, that is fine. If you don’t, that’s fine as well. Our pipeline will contain the option to execute the install if you haven’t done that yet.

Let’s get started with building our pipeline.

Building a CI/CD pipeline in Azure Devops

We’ll login to Azure Devops, and create a new pipeline. This asks us where the source code is hosted, which in our case will be Github (if you want to play along, I’d recommend making a fork of my blog repo).

We’ll select Github as our code repo.

We’ll then authorize Azure pipelines in Github.

We need to authorize Azure Devops to use Github.

Next we’ll select a repo, after which Github will again ask us for approval. As I might be doing more pipelines via Github, I’ll allow this in all my repos.

Final approval step.

Next step would be to configure our pipeline. We’ll start of with a starter pipeline, as I don’t have an existing YAML pipeline.

We’ll start of with a starter pipeline.

And this starter pipeline looks like this:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

We won’t make any changes just yet, but we’ll save this pipeline and run it for a first time. At this point, Azure DevOps whether to save the pipeline YAML to master, or to a branch. I was heavily conflicted between saving in a branch or on master (remember this post?). I decided to hold myself accountable, and actually work on this in a branch, which Azure pipelines will create for us.

Saving our default YAML pipeline.

This will trigger out pipeline to run, which looks finishes quickly (as it only does some echos.

Our pipeline finished quickly.

We can drill-down in the job, and actually look at the job output.

We can see detailed logs about each step in the pipeline.

Now, we’ll go ahead and make changes to our pipeline to execute a blue/green deployment.

Making a blue/green deployment pipeline

The first thing we’ll need to do, is add our Kubernetes cluster to the pipeline by creating an environment in Azure Devops.

We’ll go ahead and create a new environment
We’ll name our environment and pick Kubernetes.
There’s a built-in connection to Azure, so we can easily pick our existing AKS cluster.

Next up, we’ll start editing our pipeline itself. We’ll add the ‘Helm tool installer’ step and a ‘Package and deploy Helm charts’ step to our pipeline. We won’t do a Helm upgrade just yet, but we’ll do a helm ls, just to see if Helm is installed correctly. This basic pipeline will look like:

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Update_version
  steps:
  - task: HelmInstaller@1
    inputs:
      helmVersionToInstall: '3.0.0'
  - task: HelmDeploy@0
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'ls'

We can now save our pipeline, and run it.

We can save our pipeline. Notice how I’m playing nicely and commiting to a branch.
Let’s run our pipeline for a first time.

The pipeline should finish quickly, and actually return us the output from our helm ls command:

The output of helm ls, executed in our pipeline.

This worked out pretty well. Let’s update our pipeline to include a couple extra steps:

  • add a variable
variables:
  version: 1
  • install kubectl
  - task: KubectlInstaller@0
    name: Install_kubectl
    inputs:
      kubectlVersion: 'latest'
  • get a kubeconfig via az cli

To achieve this, you’ll need to authorize Azure Devops access to your subscription.

  - task: AzureCLI@2
    name: Get_kubeconfig
    inputs:
      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'
  • get the current production version (and store it as a variable in Azure Devops)
  - task: Bash@3
    name: Get_current_prod
    inputs:
      targetType: 'inline'
      script: |
        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`
        echo $color
        if [ "$color" = "blue" ]; then
          echo "##vso[task.setvariable variable=color]green"
        else
          echo "##vso[task.setvariable variable=color]blue"
        fi
  • update the version (helm upgrade)
  - task: HelmDeploy@0
    name: Update_version
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/Chart.yaml'
      releaseName: 'bluegreen'
      overrideValues: '$(color).version=$(version)'
      arguments: '--reuse-values'
  • wait for the deployment to finish
  - task: Bash@3
    name: Wait_for_deployment
    inputs:
      targetType: 'inline'
      script: 'kubectl rollout status deploy/$(color)'
  • flip the production service
  - task: HelmDeploy@0
    name: Flip_prod
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/Chart.yaml'
      releaseName: 'bluegreen'
      overrideValues: 'production=$(color)'
      arguments: '--reuse-values'

This will make our pipeline look like (spoiler alert, this pipeline isn’t correct and will fail):

# Starter pipeline

# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

variables:
  version: 1

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Update_version
  steps:
  - task: HelmInstaller@1
    name: Install_helm
    inputs:
      helmVersionToInstall: '3.0.0'
  - task: KubectlInstaller@0
    name: Install_kubectl
    inputs:
      kubectlVersion: 'latest'
  - task: AzureCLI@2
    name: Get_kubeconfig
    inputs:
      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'
  - task: Bash@3
    name: Get_current_prod
    inputs:
      targetType: 'inline'
      script: |
        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`
        echo $color
        if [ "$color" = "blue" ]; then
          echo "##vso[task.setvariable variable=color]green"
        else
          echo "##vso[task.setvariable variable=color]blue"
        fi
  - task: HelmDeploy@0
    name: Update_version
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/Chart.yaml'
      releaseName: 'bluegreen'
      overrideValues: '$(color).version=$(version)'
      arguments: '--reuse-values'
  - task: Bash@3
    name: Wait_for_deployment
    inputs:
      targetType: 'inline'
      script: 'kubectl rollout status deploy/$(color)'
  - task: HelmDeploy@0
    name: Flip_prod
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/Chart.yaml'
      releaseName: 'bluegreen'
      overrideValues: 'production=$(color)'
      arguments: '--reuse-values'

To run this release, I will manually set the version variable to 1.2.

Let’s run the pipeline and release a new version.

However, in my case, this deployment failed, since it couldn’t find my Helm chart.

The deployment couldn’t find my helm chart.

I figured I must be accessing the file system the wrong way – and the git pull ended up in a different directory. I decided to add a bash step that shows me a ls , pwd and tree to show me where I am in the file system to the beginning of my pipeline:

 - task: Bash@3
    inputs:
      targetType: 'inline'
      script: |
        ls
        pwd
        tree

This step also fails because the agent doesn’t have the tree command, but at least it showed me my error. I didn’t have the helm-blue-green folder. This makes sense, since I created this pipelines branch before I created the helm-blue-green branch.

I don’t have the helm-blue-green folder.

I decided to solve this by merging the pipelines branch into master, and creating a new branch to continue my work. There might be better ways to rebase, but this worked out easiest for me.

This again showed me another error:

Now we can access the chart, but it seems this doesn’t work just yet.

I have seen this before. I shouldn’t have pointed the update to the Chart.yaml, but to the directory that contains the Chart.yaml.

With that fixed, I hit an issue that every time I updated the version, it went back to version 1, in stead of the version I set. I figured it out after a while, and I was misusing variables in Azure Devops. A variables defined in yaml cannot be overwritten, you have to set it in the editor. So, let’s do that now:

Hit the Variables button to create a variable that can actually be overwritten.
Let’s create our variable.

And with that, our pipeline actually works. It does the update, waits for the deployment to complete and then flips production and non-production. For reference, this is the pipeline YAML that actually works:

# Starter pipeline

# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Update_version
  steps:
  - task: HelmInstaller@1
    name: Install_helm
    inputs:
      helmVersionToInstall: '3.0.0'
  - task: KubectlInstaller@0
    name: Install_kubectl
    inputs:
      kubectlVersion: 'latest'
  - task: AzureCLI@2
    name: Get_kubeconfig
    inputs:
      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'
  - task: Bash@3
    name: Get_current_prod
    inputs:
      targetType: 'inline'
      script: |
        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`
        echo $color
        if [ "$color" = "blue" ]; then
          echo "##vso[task.setvariable variable=color]green"
        else
          echo "##vso[task.setvariable variable=color]blue"
        fi
  - task: HelmDeploy@0
    name: Update_version
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/'
      releaseName: 'bluegreen'
      overrideValues: '$(color).version=$(version)'
      arguments: '--reuse-values'
  - task: Bash@3
    name: Wait_for_deployment
    inputs:
      targetType: 'inline'
      script: 'kubectl rollout status deploy/$(color)'
  - task: HelmDeploy@0
    name: Flip_prod
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'
      command: 'upgrade'
      chartType: 'FilePath'
      chartPath: 'helm-blue-green/blue-green/'
      releaseName: 'bluegreen'
      overrideValues: 'production=$(color)'
      arguments: '--reuse-values'

Conclusion

So, that’s that! We have turned our blue-green deployment into a (CI/) CD pipeline. Every time somebody pushes to master, we’ll trigger a this pipeline. In terms of being fully correct, we are only doing a CD step, not a CI step. We don’t have any software to build, we are just releasing.

So, if you’ve been following along with the different blue/green posts, we have up to this point developed a deployment strategy using the service object in Kubernetes, updated that to allow for Helm upgrades and now we integrated that into a CI/CD pipeline.

Leave a Reply