Version française disponible

This article is the second one of the series dedicated to Flagger. In a nutshell, Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.

Make sure you have a local Kubernetes cluster running with the service mesh Istio. If you don’t, read the first article: Flagger – Get Started with Istio and Kubernetes.

In this second hands-on guide, we will focus on the installation of Flagger and run multiple canary deployments of the application Mirror HTTP Server (MHS). Remember that this dummy application can simulate valid and invalid responses based on the request. This is exactly what we need to test the capabilities of Flagger. We will cover both happy (rollout) and unhappy (rollback) scenarios.

Installing Flagger

Let’s install Flagger by running these 3 commands. We install Flagger in its own namespace flagger-system.

Reference: Flagger Install on Kubernetes
Flagger depends on Istio telemetry and Prometheus (in that case, we assume Istio is installed in the istio-system namespace).
All parameters are available on the Flagger readme file on GitHub.
We don’t specify a version for Flagger, which means it will use the latest available in the repo (1.0.0-rc5 at the time of writing).

After a few seconds, you should get a message confirming that Flagger has been installed. From the Kube dashboard, verify that a new namespace has been created flagger-system and the Flagger pod is running.

Flagger is deployed in your cluster

Canary deployment – Experiment 0 – Initialization with MHS version 1.1.1

Mirror HTTP Server has multiple versions available. To play with Flagger canary deployment feature, we will switch between version 1.1.1 and 1.1.2 of MHS (the latest version at the time of writing).

Before deploying MHS, let’s create a new namespace application, we don’t want to use the default one at the root of the cluster (this is good practice). The name is too generic, but sufficient for this tutorial, in general you will use the name of the team or the name of a group of features.

Do not forget to activate Istio on this new namespace:

To deploy MHS via Flagger, I created a Helm chart.

This chart was created based on the previous chart without Flagger which itself was created with the helm create mhs-chart command, then adapted. In this “canary flavored” chart, I did some extra adaptation to use 2 replicas instead of 1 to make it more realistic and use a fixed version to 1.1.1, I also added the canary resource where the magic happens.

Clone the chart repo:

And install MHS:

After a few moments, if you look at the dashboard, you should see 2 replicas of MHS in the namespace application.

MHS 1.1.1 is deployed in your cluster

It is important to note that no canary analysis has been performed and the version has been automatically promoted. It was not a “real” canary release.
Why? Because Flagger needs to initialize itself the first time we do a canary deployment of the application. So make sure the version you are deploying with Flagger the first time is fully tested and works well!
You could also guess this auto-promotion happened because there was no initial version of the application in the cluster. Although this is obviously a good reason, it’s important to note that, even if we had a previous version before (e.g. 1.1.0), the canary version 1.1.1 would have still been automatically promoted without analysis.

You can still check the canary events with:

You should have a similar output without a canary analysis:

Or you can also directly check the log from Flagger:

If you take a closer look at the Kube dashboard, you should see some mhs and mhs-primary resources:

• mhs-primary are the primary instances (= the non-canary ones). Flagger automatically add the -primary suffix to differentiate them from the canary instances.
• mhs are the canary instances. They exist only during the canary deployment and will disappear once the canary deployment ends. That’s why, in the screenshot above, you don’t see any mhs canary pods (i.e. 0 / 0 pod).

Why this naming convention? I asked Flagger team directly and there is a technical constraint.

Flagger is now initialized properly and MHS is deployed to your cluster. You can use the terminal to confirm MHS is accessible (thanks to the Istio Gateway):

You should receive an HTTP 200 OK response:

And:

should return an HTTP 500 response:

Canary deployment – Experiment 1 – Healthy MHS version 1.1.2

We are going to install a newer version 1.1.2. You need to manually edit the file mhs-canary-chart/mhs/values.yaml and replace tag: 1.1.1 with tag: 1.1.2 (this line).

Then:

While the canary deployment is in progress, it’s very important to generate some traffic to MHS. Without traffic, Flagger will consider that something went wrong with the new version and will rollback automatically to the previous one. Obviously, you don’t need this extra step in a production environment that continuously receives real traffic.

Run this loop command in another terminal to generate artificial traffic:

Check the Kube dashboard, you should see the canary pod with the new version 1.1.2 at some point:

Canary deployment of MHS 1.1.2 in progress in your cluster

Check the canary events with the same command as before:

After a while (about 6 minutes) you should have a similar event output:

The canary release performed successfully. Now you have version 1.1.2 installed on all the primary pods and the canary pod has been removed.

MHS 1.1.2 is deployed in your cluster

Why did this deployment take about 6 minutes? Because it includes a 5 minutes canary analysis. During this analysis, the traffic was routed progressively to the canary pod. The canary traffic increased by steps of 10% every 1 minute until it reached 50% of the global traffic. The analysis is configurable and defined in the canary.yaml file that was added to the chart.

Below is the analysis configuration:

  analysis:
    # stepper schedule interval
    interval: 1m
    # max traffic percentage routed to canary - percentage (0-100)
    maxWeight: 50
    # canary increment step - percentage (0-100)
    stepWeight: 10
    # max number of failed metric checks before rollback (global to all metrics)
    threshold: 5
    metrics:
      - name: request-success-rate
        # percentage before the request success rate metric is considered as failed (0-100)
        thresholdRange:
          min: 99
        # interval for the request success rate metric check
        interval: 30s
      - name: request-duration
        # maximum req duration P99 in milliseconds before the request duration metric is considered as failed
        thresholdRange:
          max: 500
        # interval for the request duration metric check
        interval: 30s

The canary analysis has been covered with the 2 basic metrics that are provided out of the box by Istio / Prometheus (request success rate + duration). It is possible to define your own custom metrics. In that case, they will need to be provided by your application. Your application will need to expose a Prometheus endpoint that includes your custom metrics. And you will be able to update the Flagger analysis configuration to use them with your own PromQL query. Note this goes beyond the scope of this hands-on guide that uses only the built-in metrics.

Canary deployment – Experiment 2 – Faulty MHS 1.1.1

Again, you need to manually edit the file mhs-canary-chart/mhs/values.yaml and replace tag: 1.1.2 with tag: 1.1.1 like it was originally.

Then:

We generate some artificial traffic:

This time, we also generate invalid traffic to make sure the request success rate is going down!

Check the canary events with the same command as before:

After a while (about 6 minutes) you should have a similar event output:

And you are still on version 1.1.2.

Flagger decided not to go ahead and propagate version 1.1.1 as it could not perform a successful analysis and the error threshold was reached, i.e. 5 times (indeed, each time, about 50% of the requests were ending up in an HTTP 500 response). Flagger has simply redirected all traffic back to the primary instances and removed the canary pod.

Congratulations, you’ve come to the end of this second tutorial!

Observations

Before doing the optional cleaning, let’s wrap up with a list of observations:

• Deleting a deployment will delete all pods (canary / primary). And we don’t end up with orphan resources.
• Prometheus is required. Without it, the canary analysis won’t work.
• It is not possible to re-trigger a canary deployment of the same version if it has just failed. It forces you to bump up the version (even if it was a configugation and not a code issue).
• Flagger off-boarding process is not as simple as removing the canary resource from the chart and deploy a new version. If you delete the canary resource then Flagger won’t trigger the canary process, it will change the version in mhs and remove mhs-primary but mhs has 0 pods so it will make your service unavailable! You need to be careful and adopt a proper manual off-boarding process.
• After multiple deployments, it seems that some events can be missing, the Kubernetes describe command is accumulating them (x<int> over <int>m) sometimes the order is not preserved and/or some events are not showing up. You can look at the phase status (terminal status are Initialized, Succeeded and Failed). The best is to look directly at the logs on the Flagger pod as this is always accurate and complete.
• The canary analysis should be configured to run for a short period of time (i.e. no more than 30 minutes) to leverage continuous deployment and avoid releasing a new version while a canary deployment for the previous one is still in progress. In case you want to do a canary release over multiple hours or days, then maybe you should rethink about your usage and Flagger might not be the best fit.
• We already mentioned it but it’s important: the first time you deploy with Flagger (like with experiment 0), the tool needs to initialize itself (Initialized status) and will not perform any analysis.

Optional Cleaning

You can delete the MHS application and its namespace. We don’t remove Istio and Flagger because we will need them in the next article.

The next article will focus on the Grafana dashboard provided out of the box with Flagger which is a nice addition, so you don’t need to manually run any kuberctl commands to check the result of your canary deployments. Stay tuned! In the meantime, you can stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop. Your computer deserves another break.

 Version française disponible

This series of articles is dedicated to Flagger, a tool that integrates with Kubernetes, the popular container orchestration platform. Flagger enables automated deployments and will be one step closer to a continuous deployment process.

This article is the first of the series and also the only one where we won’t use Flagger yet… It will allow you to run a Kubernetes cluster on your local environment and deploy an application which will be accessible via an Istio gateway.

Docker

Install Docker by installing the Docker Desktop for Mac application, you can refer to the official installation guide. For Windows users, the equivalent application “Docker for Windows” exists.

In the next part, we will also use Docker for Mac to set up our local Kubernetes cluster. Note that this tutorial has been tested with Docker for Mac 2.2.0.5 that includes a Kubernetes Cluster in version 1.15.5.

Mirror HTTP Server

First a few words about the application Mirror HTTP Server we will use in this series of articles.

MHS is a very simple JavaScript application based on Node.js using the framework Express which allows you to customize the HTTP response received by setting specific HTTP headers in the request. The Docker image is publicly available on the Docker Hub. You can consult the Github repo of the project to find out more, please note that I am not the author.

This little app is exactly what we need to test the capabilities of Flagger to simulate 200 OK responses and 500 Internal Server Error responses.

Let’s pull the Docker image:

And run a new container that uses it:

Then let’s make sure it is functioning properly:

You should receive an HTTP 200 OK response:

While:

will return an HTTP 500 response:

For simplicity, we use the curl command, but you can use your favourite tool, e.g. Postman.

Kubernetes

Now that you’ve installed Docker for Mac, having a Kubernetes cluster running locally will be a simple formality. You just need to check a box!

Enable Kubernetes with Docker for Mac

If the light is green, then your Kubernetes cluster has successfully started. Please note, this takes a lot of resources, so don’t panic if the fan is running at full speed…

Kube dashboard

We will install our first application in our Kubernetes cluster.

Kubernetes via Docker does not come with the dashboard by default, you have to install it yourself. This dashboard is very practical and provides a graphical interface of what is going on in your cluster and will save you from having to enter kubectl commands.

The dashboard is protected, you also need to create a user to access it. Create a file dashboard-adminuser.yaml with the content:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system

Add the user:

Then generate some tokens for this user:

You can copy the last token, many are generated.

When the token expires, you will have to generate new ones with the same command! So keep this command in one of your terminal tabs, tokens are expiring quickly when you don’t interact with the dashboard…

Finally, create a proxy to access the dashboard from the browser (this command will need to run indefinitely):

If you access http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login and use the token that you copied to authenticate, you should see this screen.

Kube Dashboard

Helm

We use Homebrew for the installation of Helm. Homebrew is a handy package manager available for Mac.

We will use Helm to install Istio and the MHS application in our cluster. Helm is a bit like Homebrew, but for Kubernetes. We are using version 2.16.1, but using Helm 3 is also possible. Helm will save you from having to enter many kubectl apply commands.

Let’s install the Helm 2.16.1 formula:

To verify that Helm has been installed:

You should have a similar output:

The server version may not be initialized, in this case run:

Istio

Now, we are going to install the Istio Service Mesh. For full explanations and the benefits of using a Service Mesh, I invite you to read the official documentation.

First of all, you must increase the memory limits of your Kubernetes via Docker, otherwise you will run into deployment issues. Your laptop’s fans will recover, don’t worry…

Here is my configuration:

Kubernetes Configuration in Docker for Mac for Istio

I followed the Docker Desktop recommendations for Istio.

Let’s go and install Istio:

After a few minutes, you should get a message confirming that Istio has been installed. And voilà!

To install the latest version of Istio, you can simply replace the first line with curl -L https://istio.io/downloadIstio | sh -.

From the Kube dashboard, verify that a new namespace has been created istio-system and that it contains the Istio tools including Prometheus.

Istio is deployed in your cluster

Why is Prometheus important? Because it is an essential component for Flagger which will provide the metrics to know if the new version of your application is healthy or not, thus it will know when to promote or rollback a version. I will come back to this in detail in the next article.

Deploying Mirror HTTP Server

Before deploying MHS, let’s create a new namespace application, we don’t want to use the default one at the root of the cluster (this is good practice). The name is too generic, but sufficient for this tutorial, in general you will use the name of the team or the name of a group of features.

Do not forget to activate Istio on this new namespace:

To deploy MHS, I created a Helm chart.

This chart was created with the helm create mhs-chart command, then I updated to use the latest image of MHS. I also added a gateway.yaml file to configure the Istio gateway so it can be accessible outside of the cluster.

Clone the chart repo:

And install MHS:

After a few moments, if you look at the dashboard, you should see 1 replica of MHS in the namespace application.

MHS is deployed in your cluster

You now have 1 MHS pod running in your Kubernetes cluster. The pod is exposed to the outside world via an Istio gateway.

To test, use the similar commands that we used against the docker container earlier:

You should receive an HTTP 200 OK response that was handled by Envoy, the proxy used by Istio:

And:

should return an HTTP 500 response:

Congratulations, you’ve come to the end of the tutorial!

For information, you can also access MHS with your favourite browser if you run a proxy command first to expose the pod:

export POD_NAME=$(kubectl get pods --namespace application -l "app.kubernetes.io/name=mhs,app.kubernetes.io/instance=mhs" -o jsonpath="{.items[0].metadata.name}")

kubectl port-forward --namespace application $POD_NAME 8080:80

Then, navigate to http://localhost:8080/.

You should see a… blank page. This is normal, MHS does not return a body in the response and there is no HTML output!

Optional Cleaning

You can delete the MHS application and its namespace. We don’t remove Istio because we will need it in the next article.

The next article will focus on the installation of Flagger and use MHS to try canary deployments. Stay tuned! In the meantime, you can stop the Kubernetes cluster by unchecking the box and restarting Docker Desktop. Your computer deserves a break.