How to upgrade Istio without downtime?

14/10/2022 /

before the start

As of this writing, the highest version of Istio is 1.15.2, and official support for version 1.13 has ended. Please refer to the release status description in the Istio documentation
Determine if you need to upgrade Istio.

Istio official website gives several ways to upgrade Istio
:

  • Canary upgrade
  • In-place upgrade
  • Upgrading with Helm

But in fact, in order to reduce the impact on the business in the grid during the upgrade, it is recommended to use canary upgrade when upgrading Istio
, which is faster than in-place upgrade
More secure, and supports rollback. Using canary upgrade supports spanning two minor versions, while in-place upgrade must upgrade one minor version by one. No matter which method is used, the Ingress Gateway is in-place upgrade.

Istio official documentation
The description of the upgrade steps is not very detailed. This article is a supplement to the official documentation. There are two precautions after the upgrade is completed:

  • Mark the corresponding label for the namespace that needs automatic sidecar injection;
  • Delete the original validatingwebhookconfiguration and add a new one;

Below are the detailed upgrade steps.

Upgrade steps

The canary version was installed using the following command:

 # 将新版本的revision 命名为canary istioctl install --set revision = canary # 取消原先自动注入sidecar 的namespace 中的label 并设置新的label，这样该namespace 就可以注入canary 版本对应的sidecar kubectl label namespace test-ns istio-injection- istio.io/rev = canary # 重启数据平面中的工作负载，将完成新版本的sidecar 自动注入kubectl rollout restart deployment -n test-ns

Note that after the upgrade is complete, when enabling automatic sidecar injection for the new namespace, you need to label the namespace with the label set when installing canary Istio, and execute the following command:

 kubectl label namespace new-ns istio-injection- istio.io/rev = canary

Considerations after Istio upgrade is complete

After the upgrade is complete, there are a few more caveats. For example, if you have labeled other namespaces with a sidecar auto-injected label, be sure to delete it and set the label to istio.io/rev=canary , because it is guaranteed to inject a new version of the sidecar into the pod and connect to the New version of Istiod.

In addition, you need to delete the ValidatingWebhookConfiguration set when you first installed Istio, and execute the following command:

 kubectl delete validatingwebhookconfiguration istiod-default-validator
About ValidatingWebhookConfiguration
When you install a new version of Istio, a ValidatingWebhookConfiguration named istio-validator-canary-istio-system is automatically created. The purpose of this configuration is to check whether all connected Istiods are valid when creating and updating Istio CRs. . See the Kubernetes documentation for a detailed description of dynamic admission control
.

Because when the new version of Istio is installed, the new istio-validator-canary-istio-system is installed. If you don’t delete the old one, you will see the following error when creating an Istio CR.

 Error from server (InternalError): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": service "istiod" not found

The above content is not described in Istio’s official documentation, but in Istio Issue-36526
mentioned.

refer to

This article is reprinted from https://jimmysong.io/blog/istio-canary-upgrade/
This site is for inclusion only, and the copyright belongs to the original author.