Blue-Green Deployments of Java Applications with Cloud VMsManinderjit (Mani) BindraBlockedUnblockFollowFollowingMar 17Azure Virtual Machine Scale Sets (VMSS) are a great way to deploy identical set of VMs (based on same VM image) in true autoscale.
When there are modifications to an application deployed to a VMSS one approach commonly used to reliably push the modified application changes live is using Blue-Green deployments.
Packer is one of the most popular tools to automate creation of VM images with the application code baked in.
In this post Packer has been used to create the VM images which are used by the VMSS.
Azure DevOps pipelines are a great way to build, test and deploy application to any cloud.
In this post we will look at the Automated VMSS based Blue-Green deployment strategy to propagate changes to a sample Java application into production.
Sample Java Application and the code repository structureThe code for the sample application code, the packer template, the other release artifacts (ARM Template and Shell script) can be found at https://github.
com/maniSbindra/vmss-packer-blue-green/ .
The initial version of this repo was forked from Azure-Samples/app-service-maven to reuse the same maven based java application.
The main code is in the simple jsp file https://github.
com/maniSbindra/vmss-packer-blue-green/blob/master/src/main/webapp/index.
jsp.
Once deployed the application (which listens on port 8080) displays a simple message as shown below.
The Application in actionLet us look at the repository structure in more detail :Repository StructureAs illustrated in the Diagram let us look at the files used in the Azure DevOps (ADO) Build phase and those that are used in the Azure DevOps release phaseBuild Files and Folderpom.
xml and src/main/webapp folder : The pom file is used by maven to compile the java application source code in the folder src/main/webapp and package the application as a war filesetup-resgrp-and-service-principal.
sh : Script to create service principal which is used by packer during the generation of the VM imagevm-image-with-webapp-packer : The Packer Template uses the ubuntu 16.
04 base image, installs tomcat and copies the java application war file in the tomcat directory, and then generates the VM image with the application baked inRelease files and foldersrelease-artifacts folder : It has ARM template for Infrastructure creation and shell script used to propagate the Green VMSS into production upon release gate approval.
Blue Green Deployment StrategyThe Blue-Green Deployment can be done in several ways, each variation having its pros and cons.
The strategy used in this post is described in this section.
The Load Balancer configurationLoad Balancer and Backend Pool ConfigurationThe Base infrastructure components needed for this strategy like VNet, Subnet, Load Balancer and Public IP are all created/updated by the ARM template used in the release.
The Load Balancer has 2 backend pools, the blue pool and the green pool.
There are 2 load balancing rules configured.
The first rule sends traffic received on port 80 of the load balancer to port 8080 of the Blue backend pool (Production Traffic).
The second rule sends traffic received on port 8080 of the load balancer to port 8080 of the Green Backend pool (Used to check the Green VMSS before propagating it to production).
The Steady StateSteady State with current version of applicationIn the steady state The port 80 load balancer traffic goes through to the Blue Backend pool where it is received at port 8080 by the VMs of the VMSS (VMSS name: 20190315.
2 in the diagram above) which have the current production version of the application.
The virtual machines are all based on VM images containing the current production version of the application (VM image name simpleapp-20190315.
2 in the diagram above).
In this state traffic on port 8080 of the load balancer will give page not found as there is no VMSS in the Green Backend pool.
Blue-Green side by deployment (Prior to approval for Green VMSS Propagation to Prod)Blue-Green side by deployment (Prior to approval for Green VMSS Propagation to Prod)This state is triggered by modified application code being merged into the git branch of which the ADO build is triggered.
The build then bakes a new VM image with the latest application code baked in (simpleapp-20190315.
3 in the diagram above).
Next the release creates a new VMSS with vnext version of the application (VMSS name: 20190315.
3 in the diagram above), and this VMSS is placed in the Green Backend pool of the Load Balancer.
In this state the production traffic (on port 80 of the load balancer) goes to the blue pool and traffic on port 8080 of the load balancer will go the green pool.
This allows the team to verify and test the vnext version of the application, before giving approval to propagate the green VMSS to production (adding it to the blue pool)Post Approval propagation of the next version of the application to ProdOnce the team decide that the next version of the application is good to move into production they approve the gate and the release progresses to the next stage.
The next stage of the release takes place in 3 steps, all automated using the release shell script.
Post Approval to propagate Green VMSS to Prod — Step 1In this step (shown above) the blue VMSS is removed from the blue backend pool.
Post Approval to propagate Green VMSS to Prod — Step 2In the second step (shown above) the Green VMSS is removed from the green backend poolPost Approval to propagate Green VMSS to Prod — Step 3In the third step (shown above) the Green VMSS is added to the Blue backend pool.
The next version of the application is now in production.
Depending on the need the old VMSS can be deleted after a certain period, or it can be manually deleted.
Steady State with the next version of the applicationThis is similar to the initial steady state other that the fact that the VMSS has the next version of the application.
Steady State with next version of the applicationSome points to ponder about his release strategyWhat I like about this blue-green strategy is that it is easy to design a repeatable release pipeline.
Each time before the release is triggered there is a VMSS only in the Blue backend pool (or in the case of first time execution no VMSS in either backend pool).
After the release is triggered VMSS with latest version of the application is placed in the green backend pool.
After Manual approval the new green VMSS is then moved to the Blue Backend Pool.
A Drawback of this approach is that during the propagation of the green VMSS to the blue backend pool, there is no VMSS in the Blue Backend Pool and the production clients will experience a short downtime.
There are several other strategies to have zero downtime, but the strategy described here does have a short downtime.
So if the short downtime is acceptable you may consider a variation of this strategy.
Ok now that we understand what we are trying to do, let us set up the ADO build and release pipelines to do just that.
Azure DevOps (ADO) Build pipelineFirst create a new Visual Designer based build pipeline.
Start with an empty job.
Now we start configuring the pipeline.
Configure the Github repositorySelect code repository source as github and connect to the repository with your application code and the and files needed for build and release.
In this case it is the github.
com/maniSbindra/vmss-packer-blue-green which has been described in detail earlier in this postConfigure Github RepositoryConfigure the agent poolWe can select most of the Hosted agent options, but for now lets go with the Hosted Ubuntu 1604 agent.
Agent ConfigurationAdd Build TasksAdd the following build tasks “Publish Artifact”, “Maven”, and “Build immutable image (Packer).
Now we begin with the configuration of these tasks.
Add Build TasksConfigure Publish Artifact taskThis task is required to make the ARM template and the shell scripts required during the release process available to the Azure DevOps release.
We select the release-artifacts directory so that is it published as an artifact which will then be available to any release linked to this build.
This task could have been added after the maven task or in any order, but since we are directly publishing these artifacts from the source repository without modifying them during the build, and hence we can have this as the first task.
Configure Publish Artifact taskConfigure Maven TaskWe configure the maven task to use the pom.
xml file.
Configure the Maven TaskThe key section of the maven pom file is show below.
The packaging format is war.
<artifactId>vmss-blue-green</artifactId><packaging>war</packaging><version>0.
0.
1-SNAPSHOT</version>.
Based on this pom file maven will create the application war file at the path target/vmss-blue-green-0.
0.
1-SNAPSHOT.
war relative to the location of the pom fileConfigure the Build Immutable Image (Packer Task)Since we will be using our own packer template file we select “user provided” template, we then select the template file location as “vm-image-with-webapp-packer.
json”.
Configure the Packer TaskNext we look at the template parameters configuration.
Before that let us look at the task’s template parameters, let us look at the packer template file vm-image-with-webapp-packer.
json, in which some key lines have been highlighted.
{ "variables": { "client_id": null, "client_secret": null, "subscription_id": null, "tenant_id": null, "image_name": null, "resource_group": null, "location": null, "tomcat_version": null, "vm_size": "Standard_DS2_v2" },"builders": [ { "type": "azure-arm", "client_id": "{{user `client_id`}}", "client_secret": "{{user `client_secret`}}", "subscription_id": "{{user `subscription_id`}}", "tenant_id": "{{user `tenant_id`}}","managed_image_resource_group_name": "{{user `resource_group`}}", "managed_image_name": "{{user `image_name`}}","os_type": "Linux", "image_publisher": "Canonical", "image_offer": "UbuntuServer", "image_sku": "16.
04-LTS","azure_tags": { "dept": "engineering" },"location": "{{user `location`}}", "vm_size": "{{user `vm_size`}}" } ],"provisioners": [ { "type": "file", "source": "{{ template_dir }}/target/vmss-blue-green-0.
0.
1-SNAPSHOT.
war", "destination": "/tmp/vmss-blue-green-0.
0.
1-SNAPSHOT.
war" }, { "execute_command": "chmod +x {{ .
Path }}; {{ .
Vars }} sudo -E sh '{{ .
Path }}'", "inline": [ "apt-get update", "apt-get dist-upgrade -y", "apt-get install -y tomcat8", "cp /tmp/vmss-blue-green-0.
0.
1-SNAPSHOT.
war /var/lib/tomcat8/webapps/sampleapp.
war", "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync" ], "inline_shebang": "/bin/sh -x", "type": "shell" } ]}Let us first look at the variables section at the top of the above file.
We will see how these variables are set and passed by the build task later in this post.
The way packer works is that it first actually creates a temporary resource group in the Azure Subcription mentioned, creates a Virtual Machine (VM), performs the actions on the VM as indicated in the provisioners section (install tomcat and copy the war) of the template and then finally generalizes the VM and creates a VM Image of which other VMs and VMSS can be based.
The temporary resource group is deleted after that.
The variables client_id, client_secret, subscription_id and tenant_id are needed by packer to get the required access to the Azure Subscription.
The link has details on how to get these 4 values.
The image_name is the name of the VM image which packer will create in the resource_group and location specified.
Next we look at the provisioners section of the above file.
First using the file provisioner we copy the war file create by maven from {{ template_dir }}/target/vmss-blue-green-0.
0.
1-SNAPSHOT.
war (template_dir is the directory where the packer template is created, which in our case is the same directory as the maven pom file) to /tmp/vmss-blue-green-0.
0.
1-SNAPSHOT.
war within the VM image.
Next the shell script provisioner installs tomcat8 (“apt-get install -y tomcat8”) and copies /tmp/vmss-blue-green-0.
0.
1-SNAPSHOT.
war to the tomcat web app directory /var/lib/tomcat8/webapps/sampleapp.
war.
After this the VM is generalized and the VM Image is created.
After this any VM based of this VM image will have java application running at {VM IP or hostname}:8080/sampleapp/index.
jspThe values to be passed to the Packer task are set in the build variables section as shown below.
Setting Build VariablesNow we see how the template parameters section of the packer task is used to pass these variables to packer.
{ "client_id": "$(CLIENTID)", "client_secret": "$(CLIENTKEY)", "subscription_id": "$(SUBSCRIPTIONID)", "tenant_id": "$(TENANTID)", "tomcat_version": "8", "image_name": "$(MANAGED_IMAGE_NAME_PREFIX)-$(Build.
BuildNumber)", "resource_group": "$(RESOURCEGROUP)", "location": "$(LOCATION)"}As we can see the other build variables are passed in as is to packer.
In the case of image_name we concatenate the build variable MANAGED_IMAGE_NAME_PREFIX (value set as simpleapp in our case) and the default build variable Build.
BuildNumber, which by default has the value $(date:yyyyMMdd)$(rev:.
r).
So the second time this build is executed on 3rd-March-2019 would produce a VM image with the name simpleapp-20190315.
2Setting up CI for the BuildIn our case the build will be triggered each time there is a merge in the master branch of our repo.
This can easily be setup using the build trigger section of the BuildSetting UP CIPost Build OutcomeAfter the build executes successfully, a VM image is created in the configured resource group, as shown in the azure portal screenshot belowVM image created by packerNext we look at how the release pipeline uses this VM imageAzure DevOps (ADO) Release pipelineCreate Release PipelineWe will create a new release pipeline.
This release pipeline will be linked with the ADO build which we defined earlier, and it will have 2 stages.
The first stage creates a new green VMSS (based on the latest VM image created by the last build) and pushes it into the the green backend pool of the Load Balancer.
There is a manual approval step post this stage to move into the next stage.
Prior to the approval it is possible to verify that the new Green VMSS with the latest application is functioning as expected.
On approval the Green VMSS is propagated to handle the production traffic by moving it to the blue backend pool of the load balancer.
Create Release PipelineLinking the Release to the Build ArtifactsThe release is linked to Build artifact by adding an artifact as shown below.
Linking the Release to the build artifactConfiguring Stage 1 (Which has only one associated task)The only task in this stage is “Azure Deployment Create or Update Resource Group”.
This task deploys the ARM template https://github.
com/maniSbindra/vmss-packer-blue-green/blob/master/release-artifacts/vmss-blue-green.
json .
The first time a Deployment happens using this template it creates the Vnet, Subnet, Public IP Address, Load Balancer (with blue and green backend pools, and rules), and a VMSS based on the latest VM image created in the linked build, and places this VMSS in the Green Backend Pool of the Load Balancer.
Since the template deployment mode selected in this task is incremental, the subsequent deployments using this template only create a new VMSS based on the latest VM Image available and the VMSS is placed it in the Green Backend Pool.
Configuring the ARM Template Deployment taskThe configuration of this task is shown above.
We select the default parameters file (from build artifact) and override the parameters in the “Override template parameters” section using the release pipeline variables.
The default parameters file is shown below{ "$schema": "https://schema.
management.
azure.
com/schemas/2015-01-01/deploymentParameters.
json#", "contentVersion": "1.
0.
0.
0", "parameters": { "dnsName": { "value": "javabluegreen1" }, "adminUsername": { "value": "azureuser" }, "adminPassword": { "value": "thePassword" }, "greenManagedImageName": { "value": "newImageId" }, "greenVMSSName": { "value": "newimagename" }, "location": { "value": "southeastasia" }, "numberOfVms": { "value": 1 } } }We override these values in the Override template parameters as follows.
The screenshot of the release pipeline variable section is shown later in this section.
-dnsName "$(ARM_DNS_NAME)" -adminUsername "$(VMSS_ADMIN_USER)" -adminPassword "$(VMSS_ADMIN_PASS)" -greenManagedImageName "$(GREEN_VMSS_MANAGED_IMAGE_ID_PREFIX)-$(Build.
BuildNumber)" -greenVMSSName "$(Build.
BuildNumber)"The dnsName of the load balancer is fetched from the ARM_DNS_NAME pipeline variable.
The username and password for the VMs in the VMSS is fetched from VMSS_ADMIN_USER and VMSS_ADMIN_PASS pipeline variables respectively.
The name of the VM image used to create the VMSS is in the same format as the Build, i.
e.
“$(GREEN_VMSS_MANAGED_IMAGE_ID_PREFIX)-$(Build.
BuildNumber)” for build number 20190315.
2 corresponds to VM image name simpleapp-20190315.
2.
Similarly the name of the Green VMSS is same as the build number which would be something like 20190315.
2Setting up post deployment approval after Stage 1Once Stage 1 is complete, and pending approval, the team can test the new version of the application by hitting port 8080 on the loadbalancer (which the load balancer rule sends to the VMSS in the Green backend pool on port 8080).
Once the verification is done the team can propagate the green VMSS to production by manually approving this stage.
This approval is configured in the release gate as shown belowConfiguring Stage 2 (Propagate Green VMSS to handle production traffic)This stage is used to propagate the green VMSS into production.
During the first run of this release there is no VMSS in the Blue backend pool of the load balancer, and the Green VMSS can be added to the empty blue backend pool.
From the second execution onwards we first need to remove the VMSS with the old version of the app out of the blue backend pool and then move the green VMSS with the new version of the app in the blue backend pool.
For this reason we need to maintain the name of the VMSS in the blue backend pool in the release pipeline itself.
We store the blue VMSS name in the release pipeline variable “BLUE_VMSS_NAME”.
Let us first look at the configuration of the first task.
Configure the AgentAs earlier we select the Hosted Ubuntu 1604 agent pool.
In this stage since we need to write the name of the VMSS propagated to the blue backend pool to the release variable BLUE_VMSS_NAME we need to give the agent access to the OAuth token (as shown in the screenshot below), to enable it to edit the release definition and update the release variable.
We also need to give the user “Project Collection Build Service” user access to edit the release pipeline.
The answer in the stack overflow post gives more details on this configuration.
Configure AgentConfigure the Azure CLI task (Which executes the VMSS propagation shell script)We first select the script to be executed by the Azure CLI task as release-artifacts/swap-commands.
sh, and the arguments to pass to this script.
Configure the Azure CLI taskThe script does 3 things.
If the BLUE_VMSS_NAME exists then it is removed from the blue backend pool.
Next the GREEN_VMSS_NAME is removed from Green Backend Pool and finally the GREEN_VMSS_NAME is added to the blue backend pool.
The script accepts 4 parameters, the BLUE_VMSS_NAME, the GREEN_VMSS_NAME, the RESOURCE_GROUP where the VMSS’s exist, and the LOAD_BALANCER_BLUE_BACKEND_POOL_ID, so that the GREEN_VMSS can be added to the blue backend pool.
#!/bin/bashecho "************* Setting Variables from opts"while getopts :r:g:p:b:h opt; do case "$opt" in r) RESOURCE_GROUP="$OPTARG" ;; g) GREEN_VMSS_NAME="$OPTARG" ;; p) LOAD_BALANCER_BLUE_BACKEND_POOL_ID="$OPTARG" ;; b) BLUE_VMSS_NAME="$OPTARG" ;; h) echo "todo add help" ;; esacdoneset -xecho "RESOURCE_GROUP: $RESOURCE_GROUP"echo "BLUE_VMSS_NAME: $BLUE_VMSS_NAME"echo "GREEN_VMSS_NAME: $GREEN_VMSS_NAME"echo "LOAD_BALANCER_BLUE_BACKEND_POOL_ID: $LOAD_BALANCER_BLUE_BACKEND_POOL_ID"## IF BLUE VMSS EXISTS REMOVE IT FROM BLUE BACKEND POOL# ——————————————————if [!.-z "$BLUE_VMSS_NAME" ]thenecho "Removing Blue VMSS from blue backend pool .
"# Remove active blue vmss from blue backend poolaz vmss update -g $RESOURCE_GROUP -n $BLUE_VMSS_NAME –remove virtualMachineProfile.
networkProfile.
networkInterfaceConfigurations[0].
ipConfigurations[0].
loadBalancerBackendAddressPools 0echo "Blue VMSS removed from blue backend pool"fi# ——————————————————## REMOVE GREEN VMSS FROM GREEN BACKEND POOL# ——————————————————echo "Removing Green VMSS from Green backend pool .
"# Remove new green vmss from green backend poolaz vmss update -g $RESOURCE_GROUP -n $GREEN_VMSS_NAME –remove virtualMachineProfile.
networkProfile.
networkInterfaceConfigurations[0].
ipConfigurations[0].
loadBalancerBackendAddressPools 0# Remove new green vmss from green nat poolaz vmss update -g $RESOURCE_GROUP -n $GREEN_VMSS_NAME –remove virtualMachineProfile.
networkProfile.
networkInterfaceConfigurations[0].
ipConfigurations[0].
loadBalancerInboundNatPools 0echo "Green VMSS removed from Green backend pool"# ——————————————————# ADD GREEN VMSS TO BLUE BACKEND POOL# ——————————————————echo "Adding Green VMSS to backend pool .
"# Add new green vmss to blue backend poolaz vmss update -g "$RESOURCE_GROUP" -n "$GREEN_VMSS_NAME" –add virtualMachineProfile.
networkProfile.
networkInterfaceConfigurations[0].
ipConfigurations[0].
loadBalancerBackendAddressPools "{"id": "$LOAD_BALANCER_BLUE_BACKEND_POOL_ID"}"# ——————————————————The arguments are passed to the scripts using the release pipeline variables as follows:-r $(RESOURCE_GROUP) -g "$(Build.
BuildNumber)" -p "$(LOAD_BALANCER_BLUE_BACKEND_POOL_ID)" -b "$(BLUE_VMSS_NAME)"Let us look at the pipeline variable configurationThe LOAD_BALANCER_BLUE_BACKEND_POOL_ID is set in the format /subscriptions/{your-subscription-id}/resourceGroups/vmss-blue-green-rg/providers/Microsoft.
Network/loadBalancers/lb-vmss-ado-blue-green/backendAddressPools/lbBEBlueRelease pipeline Variable configurationConfiguration of the powershell script to add name of new VMSS (which has now been moved to the blue backend pool) to the variable BLUE_VMSS_NAMEThe powershell script is similar to the one shown in the stackoverflow link above.
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/Release/definitions/$($env:RELEASE_DEFINITIONID)?api-version=5.
0-preview.
3"Write-Host "URL: $url"$pipeline = Invoke-RestMethod -Uri $url -Headers @{ Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)" # Update an existing variable named v1030 to its new value 1035# $pipeline.
variables.
v1030.
value = "1035"$pipeline.
variables.
BLUE_VMSS_NAME.
value = "$env:BUILD_BUILDNUMBER"####****************** update the modified object **************************$json = @($pipeline) | ConvertTo-Json -Depth 99$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"} write-host "==========================================================" # Write-host "The value of Varialbe 'v1030' is updated to" $updatedef.
variables.
v1030.
valueWrite-host "The value of Varialbe 'BLUE_VMSS_NAME' is updated to" $updatedef.
variables.
BLUE_VMSS_NAME.
valuewrite-host "=========================================================="Description of the powershell scriptRelease in actionAfter Stage1 Completion pre approvalAfter the release is triggered the first stage completes without any approval.
After this stage is complete an approval notification is triggered.
At this point the new VMSS is in the Green Backend Pool and the earlier version of the app is in the Old VMSS in the blue pool.
The diagram below shows the Release waiting for approval after stage 1 completionStage one Pending approvalThe Azure portal view below shows Both VMSS 20190315.
2 and VMSS 20190315.
3The portal view of resources after stage 1The screenshot below (pre approval) shows the old VMSS in the blue backend pool and the new VMSS in the green backend pool.
Blue and Green backend pool with blue and green VMSSThe screenshot below (pre approval) shows the old version of the application accessible over port 80.
Blue VMSS accessible over port 80The screenshot below (pre approval) shows the new version of the application accessible over port 8080.
Green VMSS accessible over port 8080After release completion post approvalThe screenshot below shows the approval after stage 1Stage 1 approvalThe screenshot below (post approval) shows the VMSS with the new application version in the blue backend poolThe screenshot below (post approval) show the new version of the application accessible over port 80New Application version accessible over port 80.