Evaluation of the use of Istio compatibility versions and feature flags

[triffer]

Description
With release 1.21.0 Istio introduced compatibility versions. This feature give users an additional option, allowing release versions to be decoupled from behavioral changes.

There are additional information in the Istio documentation about compatibility versions:

Compatibility versions are recommended to be used only when an incompatibility is found, rather than as the default. Each
compatibility version will only persist for a few releases, so eventually you will need to migrate to the new behavior. Currently,
each compatibility version is intended to remain for at least two versions, though this is subject to change.

Additionally, Istio also allows to control the feature flags as environmental variables of the istiod deployment.

We should evaluate how we can use these options to avoid breaking changes during rollout of Istio.

ACs:

• compatibilityMode introduced in Istio CR
• compatibilityMode default setting is false
  - [x] setting compatibilityMode to true sets compatibilityVersion to previous minor version
• setting compatibilityMode to true sets list of specific env variables for previous minor version
• Istio module root documentation version is extended to describe compatibilityMode details
• Istio CR UI is extended to support compatibilityMode flag
• Istio update GH Issue template is extended to handle management of compatibilityMode
• Create ADR

Reasons
During rollout of Istio module version 1.5.1 a breaking change was rolled out that caused issues. We should look into options to avoid this in the future.

DoD:

• Provide unit and integration tests.
• Provide documentation.
• Verify if the solution works for both open-source Kyma and SAP BTP, Kyma runtime.
• If you changed the resource limits, explain why it was needed.
• If the default configuration of Istio Operator has been changed, you performed a manual upgrade test to verify that the change can be rolled out correctly.
• Verify that your contributions don't decrease code coverage. If they do, explain why this is the case.
• Add release notes.

Attachments

• Istio 1.21 breaking ExternalName #787

#849
#881

[strekm]

Idea is to introduce compatibilityMode true / false flag that under the hood will manage compatibilityVersion set to previous minor version and will manage list of env variable that are mentioned in Istio upgrade notes. List will be updated with every Istio minor bump and will only manage changes mentioned in latest Istio minor release.

Proposed solution is compromise giving user more time to adjust to breaking changes introduced by Istio. List of compatibility settings needs to be documented and maintained with every minor Istio bump.

It is assumed that breaking changes are identified as soon as new version is promoted to fast channel for managed kyma users. Work on adoption can be started asap also if needed with use of compatibilityMode in regular channel.

[kolodziejczak]

"The current compatibilityVersion will not override flags defined in the provided IstioOperator.yaml. Unfortunately, the compatibilityVersion handling implemented in Istio will add new flags that do not exist in the IstioOperator. With the compatibilityVersion 1.21, the zipkin address is set. This was removed from the IstioOperator in our module and enabling the compatibilityVersion will bring it back. Due to this, it seems like we cannot use the compatibilityVersion, but need to control this programatically when creating the IstioOperator in the module."
ref

Taking that into account, current implementation will not use compatibilityVersion provided by Istio repository in favour of custom implementation of the compatibility environment variables overwrite.