Release process for distributed applications with data Geo-Replication

One of the most critical parts in software architecture is designing a release process for upgrading the software in your applications. The release plan is crucial and affects several factors such as:

The most common scenarios for traditional web applications are the following:

Entirely shutdown the website, display a friendly message to users that the site is under maintenance and finally upgrade code and SQL database. It’s easy, it works but it also has a huge disadvantage: During the release you prevent users from accessing your application which is unacceptable.

A widely used release plan with zero downtime requires to deploy your web app at two servers at least and put a load balancer on the front. When you decide to release new software, you disable one server at a time using the load balancer and you upgrade the code. SQL updates mustn’t have any breaking changes since both of the servers accessing the same database. While one of the servers are down the load balancer directs the requests to the server that is up and users don’t experience any downtime.

Things get really complicated when your application is deployed in more than one locations and also uses SQL Geo-Replication meaning that there is one primary READ-WRITE SQL database that continuously sends committed transactions asynchronously to its secondary databases so that they are up to date. Secondary databases are READ-ONLY and available for querying and failover as well. The following picture describes this type of architecture.

What makes this scenario difficult to release new software is the SQL Geo-Replication feature. In case you wish to release breaking SQL schema changes you have to apply them in the primary READ-WRITE database. When you do that, changes will be transferred to secondary databases as well which means that all web apps should also be already updated otherwise will crash (due to the breaking changes). The solution to the problem is to provision new web apps and databases and apply the changes on them instead to those that are currently in production. When you finish with the upgrades you disable the previous endpoints from your load balancer, add and enable new endpoints that point to your new web apps which run the latest version of your software. Still, it’s not that simple as it sounds and several steps needs to be taken to accomplish such type of a release so let’s break it down.

Step 1 – Provision new Web Apps and SQL Databases

In this step you provision new web apps and SQL server/databases. You provision the same number of web apps and databases that are currently in production. Let’s assume that you have deployed your application in West Europe, West Central US and South East Asia and that the primary READ-WRITE database is in West Europe.

Step 2 – Setup chaining SQL Geo-Replication

At this step you start replicating your SQL data at your new databases by setting them as secondaries of your primary READ-WRITE database. In Microsoft Azure you can set up to 4 secondary databases using the SQL Active Geo-Replication feature. If you wish to add more secondaries you use a process known as chaining where you create secondary of a secondary.

Step 3 – Switch primary READ-WRITE database

At this step, you stop replicating data and also change the primary READ-WRITE database to one of the newly created. The purpose of this step is to apply the SQL schema changes to the new databases while they contain the latest snapshot of your data. Following are the sub-tasks needed in order to complete this step:

Step 4 – Deploy code and SQL shcema changes

You can safely deploy your code and SQL schema changes to the new web apps and databases. SQL schema changes are applied only in the new READ-WRITE database and replicated to the secondaries.

Notice that the web app is still served by the old version of your software and you haven’t removed SQL Geo-Replication from the old databases even they are all in READ-ONLY mode.

Step 5 – Switch active web apps

It is high time you serve the users the new version of your software. At this step, you add and enable endpoints to your load balancer for your new web apps and disable the old ones.

In case you decide to switch back to the old version, revert again the endpoints in your load balancer. Mind though that before doing so, you have to switch again the primary READ-WRITE database. After a successful release cycle you can safely delete the resources you don’t actually use (web apps and sql server/databases).

Microsoft Azure App Services

In case you use App Services for your web apps you have the option to create staging slots instead of provision new web apps, something that will certainly reduce the costs for your infrastructure. At this scenario you don’t add new endpoints to your Azure Traffic Manager but instead you swap the production and staging slots and the switch is done internally for you by the traffic manager.

About this post

This post is part of the new e-book Globally-Distributed Applications with Microsoft Azure that describes in detail how to build and deploy modern, highly available and planet-scale web applications with Microsoft Azure. The source code associated with the book contains a cross-platform Web App built with .NET Core and Angular and uses the following Azure Services:

• Azure CDN
• Azure Traffic Manager
• Azure Storage
• Azure Cosmos DB
• Azure Search
• Azure Redis Cache
• Service Bus
• App Services & WebJobs
• Azure SQL Databases
• SQL Active Geo-Replication
• Azure Active Directory B2C

Users authenticate using either ASP.NET Core Identity or Azure Active Directory B2C

The first 4 parts of the book describe the features for each Azure Service used and their role in the design of the web application. It provides step by step instructions to configure application settings and secret keys so that you can deploy it all over the globe. You will also find plenty of code for accessing Azure Services programmatically. The final chapter explains the PowerShell scripts that you can use to automate processes in a globally distributed application (resource provisioning, releases or rolling back updates). This means that all of the steps described in this post will be automated by scripts such as the following:

.\init-geo-replication.ps1 `
    -Database "<primary name>" `
    -PrimaryResourceGroupName "<primary-resource-group>" `
    -PrimaryServerName "<primary-server>" `
    -SecondaryResourceGroupName "<secondary-resource-group>" `
    -SecondaryServerName "<sercondary-server>"

.\deploy-webjob.ps1 `
    -PrimaryDatabaseServer "<primary-database-server>" `
    -Database "<database>" `
    -SqlServerLogin "<login>" `
    -SqlServerPassword "<password>" `
    -WebappParentResourceGroup "<parent-resource-group>"`
    -WebappResourceGroup "<webapp-resource-group>" `
    -WebjobAppLocation "<path-to-webjob>" `
    -DeploymentDestinationFolder "<destination-folder>" `
    -slot "upgrade"

.\start-deployment.ps1 `
   -token "<token>" -accountName "<account-name>" `
   -projectSlug "<project-slug>" `
   -webappName "<webapp-name>" `
   -resourceGroupName "<resource-group>" `
   -deploymentEnvironment "<environment>" `
   -slot "upgrade"

In case you find my blog’s content interesting, register your email to receive notifications of new posts and follow chsakell’s Blog on its Facebook or Twitter accounts.

Facebook	Twitter
.NET Web Application Development by Chris S.