{"id":543,"date":"2019-12-01T20:00:00","date_gmt":"2019-12-02T04:00:00","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=543"},"modified":"2019-11-24T09:24:14","modified_gmt":"2019-11-24T17:24:14","slug":"integrating-your-a-helm-blue-green-deployment-into-a-ci-cd-pipeline","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2019\/12\/01\/integrating-your-a-helm-blue-green-deployment-into-a-ci-cd-pipeline\/","title":{"rendered":"Integrating your a helm blue\/green deployment into a CI\/CD pipeline"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A while ago, I wrote about setting up a <a href=\"https:\/\/blog.nillsf.com\/index.php\/2019\/11\/10\/simple-kubernetes-blue-green-deployments\/\">Kubernetes blue\/green deployment <\/a>using the service object in Kubernetes. All the work done there, was done editing yaml files and using direct kubectl commands, which you wouldn&#8217;t do in production. Later on, we updated those yaml files to <a href=\"https:\/\/blog.nillsf.com\/index.php\/2019\/11\/23\/writing-a-helm-v3-chart\/\">Helm charts<\/a>, so we can update with a <code>helm upgrade<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today, we&#8217;ll take this to the next level and execute updates via a CI\/CD pipeline.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s get started. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Environment we&#8217;ll use<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I posted all the relevant files from my previous demo on <a href=\"https:\/\/github.com\/NillsF\/blog\/tree\/master\/helm-blue-green\">Github<\/a>. We&#8217;ll be using this repo to trigger (CI\/)CD.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As a CI\/CD tool I&#8217;m going to be using Azure pipelines. I haven&#8217;t tried out YAML pipelines before, so it&#8217;s a good time to finally give those a spin.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As a run-time, we&#8217;ll use an AKS cluster. If you went along with me and deployed the blue\/green app in a cluster and still have them running, that is fine. If you don&#8217;t, that&#8217;s fine as well. Our pipeline will contain the option to execute the install if you haven&#8217;t done that yet.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s get started with building our pipeline.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Building a CI\/CD pipeline in Azure Devops<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;ll login to <a href=\"https:\/\/dev.azure.com\">Azure Devops<\/a>, and create a new pipeline. This asks us where the source code is hosted, which in our case will be Github (if you want to play along, I&#8217;d recommend making a fork of my blog repo).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"737\" height=\"691\" src=\"\/wp-content\/uploads\/2019\/11\/image-40.png\" alt=\"\" class=\"wp-image-544\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-40.png 737w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-40-300x281.png 300w\" sizes=\"auto, (max-width: 737px) 100vw, 737px\" \/><figcaption>We&#8217;ll select Github as our code repo.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;ll then authorize Azure pipelines in Github.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"523\" height=\"739\" src=\"\/wp-content\/uploads\/2019\/11\/image-42.png\" alt=\"\" class=\"wp-image-546\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-42.png 523w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-42-212x300.png 212w\" sizes=\"auto, (max-width: 523px) 100vw, 523px\" \/><figcaption>We need to authorize Azure Devops to use Github.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next we&#8217;ll select a repo, after which Github will again ask us for approval. As I might be doing more pipelines via Github, I&#8217;ll allow this in all my repos.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"542\" height=\"765\" src=\"\/wp-content\/uploads\/2019\/11\/image-44.png\" alt=\"\" class=\"wp-image-548\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-44.png 542w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-44-213x300.png 213w\" sizes=\"auto, (max-width: 542px) 100vw, 542px\" \/><figcaption>Final approval step.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next step would be to configure our pipeline. We&#8217;ll start of with a starter pipeline, as I don&#8217;t have an existing YAML pipeline.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"630\" height=\"256\" src=\"\/wp-content\/uploads\/2019\/11\/image-45.png\" alt=\"\" class=\"wp-image-549\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-45.png 630w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-45-300x122.png 300w\" sizes=\"auto, (max-width: 630px) 100vw, 630px\" \/><figcaption>We&#8217;ll start of with a starter pipeline.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">And this starter pipeline looks like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Starter pipeline\n# Start with a minimal pipeline that you can customize to build and deploy your code.\n# Add steps that build, run tests, deploy, and more:\n# https:\/\/aka.ms\/yaml\n\ntrigger:\n- master\n\npool:\n  vmImage: 'ubuntu-latest'\n\nsteps:\n- script: echo Hello, world!\n  displayName: 'Run a one-line script'\n\n- script: |\n    echo Add other tasks to build, test, and deploy your project.\n    echo See https:\/\/aka.ms\/yaml\n  displayName: 'Run a multi-line script'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We won&#8217;t make any changes just yet, but we&#8217;ll save this pipeline and run it for a first time. At this point, Azure DevOps whether to save the pipeline YAML to master, or to a branch. I was heavily conflicted between saving in a branch or on master (remember <a href=\"https:\/\/blog.nillsf.com\/index.php\/2019\/11\/14\/git-branching-explained\/\">this post<\/a>?). I decided to hold myself accountable, and actually work on this in a branch, which Azure pipelines will create for us.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"464\" height=\"508\" src=\"\/wp-content\/uploads\/2019\/11\/image-46.png\" alt=\"\" class=\"wp-image-550\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-46.png 464w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-46-274x300.png 274w\" sizes=\"auto, (max-width: 464px) 100vw, 464px\" \/><figcaption>Saving our default YAML pipeline.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This will trigger out pipeline to run, which looks finishes quickly (as it only does some echos.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"394\" src=\"\/wp-content\/uploads\/2019\/11\/image-47-1024x394.png\" alt=\"\" class=\"wp-image-551\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-47-1024x394.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-47-300x115.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-47-768x296.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-47.png 1073w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Our pipeline finished quickly.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">We can drill-down in the job, and actually look at the job output.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"415\" src=\"\/wp-content\/uploads\/2019\/11\/image-48-1024x415.png\" alt=\"\" class=\"wp-image-552\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-48-1024x415.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-48-300x121.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-48-768x311.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-48.png 1062w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>We can see detailed logs about each step in the pipeline.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now, we&#8217;ll go ahead and make changes to our pipeline to execute a blue\/green deployment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Making a blue\/green deployment pipeline<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The first thing we&#8217;ll need to do, is add our Kubernetes cluster to the pipeline by creating an environment in Azure Devops. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"784\" height=\"398\" src=\"\/wp-content\/uploads\/2019\/11\/image-49.png\" alt=\"\" class=\"wp-image-553\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-49.png 784w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-49-300x152.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-49-768x390.png 768w\" sizes=\"auto, (max-width: 784px) 100vw, 784px\" \/><figcaption>We&#8217;ll go ahead and create a new environment<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"463\" height=\"479\" src=\"\/wp-content\/uploads\/2019\/11\/image-50.png\" alt=\"\" class=\"wp-image-554\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-50.png 463w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-50-290x300.png 290w\" sizes=\"auto, (max-width: 463px) 100vw, 463px\" \/><figcaption>We&#8217;ll name our environment and pick Kubernetes.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"465\" height=\"482\" src=\"\/wp-content\/uploads\/2019\/11\/image-51.png\" alt=\"\" class=\"wp-image-555\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-51.png 465w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-51-289x300.png 289w\" sizes=\"auto, (max-width: 465px) 100vw, 465px\" \/><figcaption>There&#8217;s a built-in connection to Azure, so we can easily pick our existing AKS cluster.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next up, we&#8217;ll start editing our pipeline itself. We&#8217;ll add the &#8216;Helm tool installer&#8217; step and a &#8216;Package and deploy Helm charts&#8217; step to our pipeline. We won&#8217;t do a Helm upgrade just yet, but we&#8217;ll do a <code>helm ls<\/code>, just to see if Helm is installed correctly. This basic pipeline will look like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Starter pipeline\n# Start with a minimal pipeline that you can customize to build and deploy your code.\n# Add steps that build, run tests, deploy, and more:\n# https:\/\/aka.ms\/yaml\n\ntrigger:\n- master\n\npool:\n  vmImage: 'ubuntu-latest'\n\njobs:\n- job: Update_version\n  steps:\n  - task: HelmInstaller@1\n    inputs:\n      helmVersionToInstall: '3.0.0'\n  - task: HelmDeploy@0\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'ls'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We can now save our pipeline, and run it. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"461\" height=\"391\" src=\"\/wp-content\/uploads\/2019\/11\/image-56.png\" alt=\"\" class=\"wp-image-571\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-56.png 461w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-56-300x254.png 300w\" sizes=\"auto, (max-width: 461px) 100vw, 461px\" \/><figcaption>We can save our pipeline. Notice how I&#8217;m playing nicely and commiting to a branch.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"457\" height=\"349\" src=\"\/wp-content\/uploads\/2019\/11\/image-57.png\" alt=\"\" class=\"wp-image-572\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-57.png 457w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-57-300x229.png 300w\" sizes=\"auto, (max-width: 457px) 100vw, 457px\" \/><figcaption>Let&#8217;s run our pipeline for a first time.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The pipeline should finish quickly, and actually return us the output from our helm ls command:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"752\" height=\"503\" src=\"\/wp-content\/uploads\/2019\/11\/image-58.png\" alt=\"\" class=\"wp-image-573\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-58.png 752w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-58-300x201.png 300w\" sizes=\"auto, (max-width: 752px) 100vw, 752px\" \/><figcaption>The output of helm ls, executed in our pipeline.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This worked out pretty well. Let&#8217;s update our pipeline to include a couple extra steps:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>add a variable<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>variables:\n  version: 1<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>install kubectl<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: KubectlInstaller@0\n    name: Install_kubectl\n    inputs:\n      kubectlVersion: 'latest'<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>get a kubeconfig via az cli<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">To achieve this, you&#8217;ll need to authorize Azure Devops access to your subscription.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: AzureCLI@2\n    name: Get_kubeconfig\n    inputs:\n      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'\n      scriptType: 'bash'\n      scriptLocation: 'inlineScript'\n      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>get the current production version (and store it as a variable in Azure Devops)<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: Bash@3\n    name: Get_current_prod\n    inputs:\n      targetType: 'inline'\n      script: |\n        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`\n        echo $color\n        if [ \"$color\" = \"blue\" ]; then\n          echo \"##vso[task.setvariable variable=color]green\"\n        else\n          echo \"##vso[task.setvariable variable=color]blue\"\n        fi<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>update the version (helm upgrade)<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: HelmDeploy@0\n    name: Update_version\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/Chart.yaml'\n      releaseName: 'bluegreen'\n      overrideValues: '$(color).version=$(version)'\n      arguments: '--reuse-values'<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>wait for the deployment to finish<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: Bash@3\n    name: Wait_for_deployment\n    inputs:\n      targetType: 'inline'\n      script: 'kubectl rollout status deploy\/$(color)'<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>flip the production service<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>  - task: HelmDeploy@0\n    name: Flip_prod\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/Chart.yaml'\n      releaseName: 'bluegreen'\n      overrideValues: 'production=$(color)'\n      arguments: '--reuse-values'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This will make our pipeline look like (<em>spoiler alert, this pipeline isn&#8217;t correct and will fail)<\/em>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Starter pipeline\n\n# Start with a minimal pipeline that you can customize to build and deploy your code.\n# Add steps that build, run tests, deploy, and more:\n# https:\/\/aka.ms\/yaml\n\ntrigger:\n- master\n\nvariables:\n  version: 1\n\npool:\n  vmImage: 'ubuntu-latest'\n\njobs:\n- job: Update_version\n  steps:\n  - task: HelmInstaller@1\n    name: Install_helm\n    inputs:\n      helmVersionToInstall: '3.0.0'\n  - task: KubectlInstaller@0\n    name: Install_kubectl\n    inputs:\n      kubectlVersion: 'latest'\n  - task: AzureCLI@2\n    name: Get_kubeconfig\n    inputs:\n      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'\n      scriptType: 'bash'\n      scriptLocation: 'inlineScript'\n      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'\n  - task: Bash@3\n    name: Get_current_prod\n    inputs:\n      targetType: 'inline'\n      script: |\n        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`\n        echo $color\n        if [ \"$color\" = \"blue\" ]; then\n          echo \"##vso[task.setvariable variable=color]green\"\n        else\n          echo \"##vso[task.setvariable variable=color]blue\"\n        fi\n  - task: HelmDeploy@0\n    name: Update_version\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/Chart.yaml'\n      releaseName: 'bluegreen'\n      overrideValues: '$(color).version=$(version)'\n      arguments: '--reuse-values'\n  - task: Bash@3\n    name: Wait_for_deployment\n    inputs:\n      targetType: 'inline'\n      script: 'kubectl rollout status deploy\/$(color)'\n  - task: HelmDeploy@0\n    name: Flip_prod\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/Chart.yaml'\n      releaseName: 'bluegreen'\n      overrideValues: 'production=$(color)'\n      arguments: '--reuse-values'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">To run this release, I will manually set the version variable to 1.2. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"472\" height=\"401\" src=\"\/wp-content\/uploads\/2019\/11\/image-63.png\" alt=\"\" class=\"wp-image-579\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-63.png 472w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-63-300x255.png 300w\" sizes=\"auto, (max-width: 472px) 100vw, 472px\" \/><figcaption>Let&#8217;s run the pipeline and release a new version.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">However, in my case, this deployment failed, since it couldn&#8217;t find my Helm chart.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"596\" height=\"191\" src=\"\/wp-content\/uploads\/2019\/11\/image-59.png\" alt=\"\" class=\"wp-image-574\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-59.png 596w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-59-300x96.png 300w\" sizes=\"auto, (max-width: 596px) 100vw, 596px\" \/><figcaption>The deployment couldn&#8217;t find my helm chart.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I figured I must be accessing the file system the wrong way &#8211; and the git pull ended up in a different directory. I decided to add a bash step that shows me a <code>ls <\/code>, <code>pwd <\/code>and <code>tree <\/code>to show me where I am in the file system to the beginning of my pipeline:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code> - task: Bash@3\n    inputs:\n      targetType: 'inline'\n      script: |\n        ls\n        pwd\n        tree<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This step also fails because the agent doesn&#8217;t have the <code>tree<\/code> command, but at least it showed me my error. I didn&#8217;t have the helm-blue-green folder. This makes sense, since I created this pipelines branch before I created the helm-blue-green branch.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"513\" src=\"\/wp-content\/uploads\/2019\/11\/image-60-1024x513.png\" alt=\"\" class=\"wp-image-575\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-60-1024x513.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-60-300x150.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-60-768x385.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-60.png 1060w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>I don&#8217;t have the helm-blue-green folder.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I decided to solve this by merging the pipelines branch into master, and creating a new branch to continue my work. There might be better ways to rebase, but this worked out easiest for me.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"> This again showed me another error:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"933\" height=\"31\" src=\"\/wp-content\/uploads\/2019\/11\/image-61.png\" alt=\"\" class=\"wp-image-576\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-61.png 933w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-61-300x10.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-61-768x26.png 768w\" sizes=\"auto, (max-width: 933px) 100vw, 933px\" \/><figcaption>Now we can access the chart, but it seems this doesn&#8217;t work just yet.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I have seen this before. I shouldn&#8217;t have pointed the update to the Chart.yaml, but to the directory that contains the Chart.yaml.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With that fixed, I hit an issue that every time I updated the version, it went back to version 1, in stead of the version I set. I figured it out after a while, and I was misusing variables in Azure Devops. <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/devops\/pipelines\/process\/variables?view=azure-devops&amp;tabs=yaml%2Cbatch#allow-at-queue-time\">A variables defined in yaml cannot be overwritten, you have to set it in the editor<\/a>. So, let&#8217;s do that now:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"231\" height=\"61\" src=\"\/wp-content\/uploads\/2019\/11\/image-64.png\" alt=\"\" class=\"wp-image-580\"\/><figcaption>Hit the Variables button to create a variable that can actually be overwritten.<br><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"463\" height=\"270\" src=\"\/wp-content\/uploads\/2019\/11\/image-65.png\" alt=\"\" class=\"wp-image-581\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-65.png 463w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/11\/image-65-300x175.png 300w\" sizes=\"auto, (max-width: 463px) 100vw, 463px\" \/><figcaption>Let&#8217;s create our variable.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">And with that, our pipeline actually works. It does the update, waits for the deployment to complete and then flips production and non-production. For reference, this is the pipeline YAML that actually works:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Starter pipeline\n\n# Start with a minimal pipeline that you can customize to build and deploy your code.\n# Add steps that build, run tests, deploy, and more:\n# https:\/\/aka.ms\/yaml\n\ntrigger:\n- master\n\npool:\n  vmImage: 'ubuntu-latest'\n\njobs:\n- job: Update_version\n  steps:\n  - task: HelmInstaller@1\n    name: Install_helm\n    inputs:\n      helmVersionToInstall: '3.0.0'\n  - task: KubectlInstaller@0\n    name: Install_kubectl\n    inputs:\n      kubectlVersion: 'latest'\n  - task: AzureCLI@2\n    name: Get_kubeconfig\n    inputs:\n      azureSubscription: 'Nills''s Cloud-scale Datacenter(d19dddf3-9520-4226-a313-ae8ee08675e5)'\n      scriptType: 'bash'\n      scriptLocation: 'inlineScript'\n      inlineScript: 'az aks get-credentials -g KEDA -n nf-keda'\n  - task: Bash@3\n    name: Get_current_prod\n    inputs:\n      targetType: 'inline'\n      script: |\n        color=`kubectl get svc production -o yaml | grep color | awk -F ' ' '{print $2}'`\n        echo $color\n        if [ \"$color\" = \"blue\" ]; then\n          echo \"##vso[task.setvariable variable=color]green\"\n        else\n          echo \"##vso[task.setvariable variable=color]blue\"\n        fi\n  - task: HelmDeploy@0\n    name: Update_version\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/'\n      releaseName: 'bluegreen'\n      overrideValues: '$(color).version=$(version)'\n      arguments: '--reuse-values'\n  - task: Bash@3\n    name: Wait_for_deployment\n    inputs:\n      targetType: 'inline'\n      script: 'kubectl rollout status deploy\/$(color)'\n  - task: HelmDeploy@0\n    name: Flip_prod\n    inputs:\n      connectionType: 'Kubernetes Service Connection'\n      kubernetesServiceConnection: 'k8s-cluster-nf-keda-default-1574378163229'\n      command: 'upgrade'\n      chartType: 'FilePath'\n      chartPath: 'helm-blue-green\/blue-green\/'\n      releaseName: 'bluegreen'\n      overrideValues: 'production=$(color)'\n      arguments: '--reuse-values'<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">So, that&#8217;s that! We have turned our blue-green deployment into a (CI\/) CD pipeline. Every time somebody pushes to master, we&#8217;ll trigger a this pipeline. In terms of being fully correct, we are only doing a CD step, not a CI step. We don&#8217;t have any software to build, we are just releasing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, if you&#8217;ve been following along with the different blue\/green posts, we have up to this point developed a deployment strategy using the service object in Kubernetes, updated that to allow for Helm upgrades and now we integrated that into a CI\/CD pipeline.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A while ago, I wrote about setting up a Kubernetes blue\/green deployment using the service object in Kubernetes. All the work done there, was done editing yaml files and using direct kubectl commands, which you wouldn&#8217;t do in production. Later on, we updated those yaml files to Helm charts, so we can update with a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":585,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2,3,58,5],"tags":[8,60,59,18],"class_list":["post-543","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-devops","category-kubernetes","category-open-source","tag-azure","tag-azure-devops","tag-helm","tag-kubernetes"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/12\/2019-11-24-09_23_39-Window.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/543","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=543"}],"version-history":[{"count":5,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/543\/revisions"}],"predecessor-version":[{"id":583,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/543\/revisions\/583"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/585"}],"wp:attachment":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=543"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=543"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}