v1.28.0

Cloud Provider Azure v1.28.0

Full Changelog: v1.27.0..v1.28.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • VmType’s default value is changed from ‘standard’ to ‘vmss’ since v1.28.

    if your cluster is not using any VMSS instances and ‘vmType’ is not configured in the cloud provider config file, please add vmType=standard in the config file before upgrading to v1.28. (#4214, @feiskyer)

Changes by Kind

Testing

Feature

  • Apply the custom probe configs when externalTrafficPolicy is local and PLSProxyProtocol is enabled. (#3931, @MartinForReal)
  • Chore: upgrade to sdk api version 2022-08-01 feat: add PerformancePlus option in disk creation (#3855, @andyzhangx)
  • Feat: Support migrate from NIC-based to IP-based backend pool by migration API by setting "enableMigrateToIpBasedBackendPoolAPI": true. By using this API, there will be no downtime during the migration. (#3972, @nilo19)
  • Feat: add GetLatestAccountKey in account key fetch (#4067, @andyzhangx)
  • Feat: add load balancer backend pool batch updater (#4391, @nilo19)
  • Feat: support load balancer choosing logic for multi-slb (#4075, @nilo19)
  • Feat: support local service in multiple standard load balancer mode (#4450, @nilo19)
  • Feat: support node selection for multiple standard load balancers (#4201, @nilo19)
  • Feat: support sharing IP address across services by public IP name (#4257, @nilo19)
  • Feat: support workload identity (#3378, @cvvz)
  • Health probe port can be any port assigned by customer. (#4380, @MartinForReal)
  • Skip unmanaged Nodes for instancesV2 (#4294, @lzhecheng)
  • Support NSG and clean LBs for dualstack
  • [ARG] Add an option to disable API call cache New option: disableAPICallCache When ARG is enabled, this option should be true. (#4135, @lzhecheng)
  • [DualStack] Support FrontendIPConfig and reconcileLB() * DualStack feature code * Refactor related functions and methods * Refactor and add new UTs (#3819, @lzhecheng)
  • add PickRandomMatchingAccount in account search

Failing Test

  • Chore: skip exclude node label e2e test on aks as it is only supported in self-managed clusters (#4076, @nilo19)

Bug or Regression

  • Fix IPv6/dual-stack EnsureBackendPoolDeleted() failure. IP config of IPv6 is not primary, it should not be skipped in EnsureBackendPoolDeleted(). Updated e2e code. (#4272, @lzhecheng)
  • Fix vmssflex ensureBackendPoolDeletedFromNode
    • Fix loop pointer issue
    • Use lock on nicUpdated var
    • Fix log format (#4074, @lzhecheng)
  • Fix: PerformancePlus setting issue (#4193, @andyzhangx)
  • Fix: add StorageAccountCache to avoid querying storage account frequently (#4422, @andyzhangx)
  • Fix: make sure the pip dns tag will not be removed when systemTags is set (#3956, @nilo19)
  • Fix: remove deleted node IP address from IP-based LB backend pools (#4136, @nilo19)
  • Fix: replace deprecated labels with new labels in nodeCache update (#4047, @andyzhangx)
  • Fix: storage account search default values (#4203, @andyzhangx)
  • Fix: the pip without tags should be user-assigned fix: refresh the pip cache when necessary fix: do not tag user-assigned pip with kubernetes-dns-label-service: <svcName> (#3877, @nilo19)
  • Fix: update the lb list after changing lb to prevent etag mismatches fix: return the existing lb it if the lb exists without creating a new lb when the service was moved to the lb fix: should skip non-existent lb when arranging nodes (#4289, @nilo19)
  • Fixed: Remove shared nsg rule immediately when no destinations left (#3787, @MartinForReal)
  • Fixes issue 4230 and removes the additional filtering on NotReady nodes by the azure cloud provider code (#4234, @alexanderConstantinescu)
  • Increase limit for TCP Idle Timeout to 100 minutes (#4361, @JoelSpeed)
  • Support customization of numOfProbe and probeInterval when externaltrafficpolicy is local (#4207, @MartinForReal)
  • The deprecated beta topology labels are no longer applied by default, to maintain the legacy behaviour use --deprecated-apply-beta-topology-labels (#3685, @JoelSpeed)
  • Virtual node will always exists (#4393, @MartinForReal)
  • [IPv6] Fix reconcileFrontendIPConfigs(). Current logic handles lb.FrontendIPConfigurations according to Service’s IP family, which is incorrect. For an IPv6 cluster, there’re still some IPv4 FIPs due to Azure limitation, which will be removed. For an IPv4 cluster, all resources are of IPv4, which is not affected. (#3914, @lzhecheng)
  • [IPv6] backend pool name should be case-insensitive (#3932, @lzhecheng)

Other (Cleanup or Flake)

  • Build images from debian bullseye (#4066, @jackfrancis)
  • Chore: cleanup unused multi-slb code of the previous design (#3997, @nilo19)
  • Chore: set default loadBalancerSKU to Standard (#3768, @nilo19)
  • Helm: add logVerbosity to cloud-node-manager (#4111, @lzhecheng)
  • Release helm v1.27.7+20230815 (#4446, @nilo19)
  • [DualStack] IPv6 PIP uses suffix only when DualStack. For CCM v1.27.1, the IPv6 PIP created has suffix. After CCM is upgraded, such PIP will be recreated. (#3823, @lzhecheng)
  • [Log] Print Service name and resource basename in ReconcileService(). Adding such log shows relation between Service name and its related resource name (Frontend IP config, etc.). It helps debugging. (#3957, @lzhecheng)
  • 
#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
--> ([#4304](https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4304), [@nilo19](https://github.com/nilo19))

Dependencies

Added

  • cloud.google.com/go/accessapproval: v1.6.0
  • cloud.google.com/go/accesscontextmanager: v1.7.0
  • cloud.google.com/go/aiplatform: v1.37.0
  • cloud.google.com/go/analytics: v0.19.0
  • cloud.google.com/go/apigateway: v1.5.0
  • cloud.google.com/go/apigeeconnect: v1.5.0
  • cloud.google.com/go/apigeeregistry: v0.6.0
  • cloud.google.com/go/appengine: v1.7.1
  • cloud.google.com/go/area120: v0.7.1
  • cloud.google.com/go/artifactregistry: v1.13.0
  • cloud.google.com/go/asset: v1.13.0
  • cloud.google.com/go/assuredworkloads: v1.10.0
  • cloud.google.com/go/automl: v1.12.0
  • cloud.google.com/go/baremetalsolution: v0.5.0
  • cloud.google.com/go/batch: v0.7.0
  • cloud.google.com/go/beyondcorp: v0.5.0
  • cloud.google.com/go/billing: v1.13.0
  • cloud.google.com/go/binaryauthorization: v1.5.0
  • cloud.google.com/go/certificatemanager: v1.6.0
  • cloud.google.com/go/channel: v1.12.0
  • cloud.google.com/go/cloudbuild: v1.9.0
  • cloud.google.com/go/clouddms: v1.5.0
  • cloud.google.com/go/cloudtasks: v1.10.0
  • cloud.google.com/go/compute/metadata: v0.2.3
  • cloud.google.com/go/compute: v1.19.0
  • cloud.google.com/go/contactcenterinsights: v1.6.0
  • cloud.google.com/go/container: v1.15.0
  • cloud.google.com/go/containeranalysis: v0.9.0
  • cloud.google.com/go/datacatalog: v1.13.0
  • cloud.google.com/go/dataflow: v0.8.0
  • cloud.google.com/go/dataform: v0.7.0
  • cloud.google.com/go/datafusion: v1.6.0
  • cloud.google.com/go/datalabeling: v0.7.0
  • cloud.google.com/go/dataplex: v1.6.0
  • cloud.google.com/go/dataproc: v1.12.0
  • cloud.google.com/go/dataqna: v0.7.0
  • cloud.google.com/go/datastream: v1.7.0
  • cloud.google.com/go/deploy: v1.8.0
  • cloud.google.com/go/dialogflow: v1.32.0
  • cloud.google.com/go/dlp: v1.9.0
  • cloud.google.com/go/documentai: v1.18.0
  • cloud.google.com/go/domains: v0.8.0
  • cloud.google.com/go/edgecontainer: v1.0.0
  • cloud.google.com/go/errorreporting: v0.3.0
  • cloud.google.com/go/essentialcontacts: v1.5.0
  • cloud.google.com/go/eventarc: v1.11.0
  • cloud.google.com/go/filestore: v1.6.0
  • cloud.google.com/go/firestore: v1.9.0
  • cloud.google.com/go/functions: v1.13.0
  • cloud.google.com/go/gaming: v1.9.0
  • cloud.google.com/go/gkebackup: v0.4.0
  • cloud.google.com/go/gkeconnect: v0.7.0
  • cloud.google.com/go/gkehub: v0.12.0
  • cloud.google.com/go/gkemulticloud: v0.5.0
  • cloud.google.com/go/gsuiteaddons: v1.5.0
  • cloud.google.com/go/iam: v0.13.0
  • cloud.google.com/go/iap: v1.7.1
  • cloud.google.com/go/ids: v1.3.0
  • cloud.google.com/go/iot: v1.6.0
  • cloud.google.com/go/kms: v1.10.1
  • cloud.google.com/go/language: v1.9.0
  • cloud.google.com/go/lifesciences: v0.8.0
  • cloud.google.com/go/logging: v1.7.0
  • cloud.google.com/go/longrunning: v0.4.1
  • cloud.google.com/go/managedidentities: v1.5.0
  • cloud.google.com/go/maps: v0.7.0
  • cloud.google.com/go/mediatranslation: v0.7.0
  • cloud.google.com/go/memcache: v1.9.0
  • cloud.google.com/go/metastore: v1.10.0
  • cloud.google.com/go/monitoring: v1.13.0
  • cloud.google.com/go/networkconnectivity: v1.11.0
  • cloud.google.com/go/networkmanagement: v1.6.0
  • cloud.google.com/go/networksecurity: v0.8.0
  • cloud.google.com/go/notebooks: v1.8.0
  • cloud.google.com/go/optimization: v1.3.1
  • cloud.google.com/go/orchestration: v1.6.0
  • cloud.google.com/go/orgpolicy: v1.10.0
  • cloud.google.com/go/osconfig: v1.11.0
  • cloud.google.com/go/oslogin: v1.9.0
  • cloud.google.com/go/phishingprotection: v0.7.0
  • cloud.google.com/go/policytroubleshooter: v1.6.0
  • cloud.google.com/go/privatecatalog: v0.8.0
  • cloud.google.com/go/pubsublite: v1.7.0
  • cloud.google.com/go/recaptchaenterprise/v2: v2.7.0
  • cloud.google.com/go/recommendationengine: v0.7.0
  • cloud.google.com/go/recommender: v1.9.0
  • cloud.google.com/go/redis: v1.11.0
  • cloud.google.com/go/resourcemanager: v1.7.0
  • cloud.google.com/go/resourcesettings: v1.5.0
  • cloud.google.com/go/retail: v1.12.0
  • cloud.google.com/go/run: v0.9.0
  • cloud.google.com/go/scheduler: v1.9.0
  • cloud.google.com/go/secretmanager: v1.10.0
  • cloud.google.com/go/security: v1.13.0
  • cloud.google.com/go/securitycenter: v1.19.0
  • cloud.google.com/go/servicedirectory: v1.9.0
  • cloud.google.com/go/shell: v1.6.0
  • cloud.google.com/go/spanner: v1.45.0
  • cloud.google.com/go/speech: v1.15.0
  • cloud.google.com/go/storagetransfer: v1.8.0
  • cloud.google.com/go/talent: v1.5.0
  • cloud.google.com/go/texttospeech: v1.6.0
  • cloud.google.com/go/tpu: v1.5.0
  • cloud.google.com/go/trace: v1.9.0
  • cloud.google.com/go/translate: v1.7.0
  • cloud.google.com/go/video: v1.15.0
  • cloud.google.com/go/videointelligence: v1.10.0
  • cloud.google.com/go/vision/v2: v2.7.0
  • cloud.google.com/go/vmmigration: v1.6.0
  • cloud.google.com/go/vmwareengine: v0.3.0
  • cloud.google.com/go/vpcaccess: v1.6.0
  • cloud.google.com/go/webrisk: v1.8.0
  • cloud.google.com/go/websecurityscanner: v1.5.0
  • cloud.google.com/go/workflows: v1.10.0
  • github.com/alecthomas/kingpin/v2: v2.3.2
  • github.com/antlr/antlr4/runtime/Go/antlr/v4: 8188dc5
  • github.com/google/gnostic-models: v0.6.8
  • github.com/samber/lo: v1.37.0
  • github.com/shopspring/decimal: v1.3.1
  • github.com/xhit/go-str2duration/v2: v2.1.0
  • google.golang.org/genproto/googleapis/api: dd9d682
  • google.golang.org/genproto/googleapis/rpc: 28d5490
  • k8s.io/cri-api: v0.28.0

Changed

  • cloud.google.com/go/bigquery: v1.8.0 → v1.50.0
  • cloud.google.com/go/datastore: v1.1.0 → v1.11.0
  • cloud.google.com/go/pubsub: v1.3.1 → v1.30.0
  • cloud.google.com/go: v0.97.0 → v0.110.0
  • github.com/Azure/azure-kusto-go: v0.10.2 → v0.14.0
  • github.com/Azure/azure-sdk-for-go/sdk/azcore: v1.2.0 → v1.6.1
  • github.com/Azure/azure-sdk-for-go/sdk/azidentity: v1.2.0 → v1.3.0
  • github.com/Azure/azure-sdk-for-go/sdk/internal: v1.1.1 → v1.3.0
  • github.com/Azure/azure-sdk-for-go/sdk/storage/azblob: v0.6.1 → v1.0.0
  • github.com/Azure/go-autorest/autorest: v0.11.28 → v0.11.29
  • github.com/AzureAD/microsoft-authentication-library-for-go: v0.7.0 → v1.0.0
  • github.com/alecthomas/units: f65c72e → b94a6e3
  • github.com/cenkalti/backoff/v4: v4.2.0 → v4.2.1
  • github.com/census-instrumentation/opencensus-proto: v0.2.1 → v0.4.1
  • github.com/cespare/xxhash/v2: v2.1.2 → v2.2.0
  • github.com/cncf/udpa/go: 04548b0 → c52dc94
  • github.com/cncf/xds/go: cb28da3 → 06c439d
  • github.com/coreos/go-oidc: v2.1.0+incompatible → v2.2.1+incompatible
  • github.com/coreos/go-semver: v0.3.0 → v0.3.1
  • github.com/coreos/go-systemd/v22: v22.3.2 → v22.5.0
  • github.com/creack/pty: v1.1.11 → v1.1.18
  • github.com/dnaeon/go-vcr: v1.1.0 → v1.2.0
  • github.com/dustin/go-humanize: v1.0.0 → v1.0.1
  • github.com/envoyproxy/go-control-plane: 49ff273 → v0.10.3
  • github.com/envoyproxy/protoc-gen-validate: v0.1.0 → v0.9.1
  • github.com/go-kit/log: v0.2.0 → v0.2.1
  • github.com/go-openapi/jsonpointer: v0.19.5 → v0.19.6
  • github.com/go-openapi/jsonreference: v0.20.0 → v0.20.2
  • github.com/go-openapi/swag: v0.19.14 → v0.22.3
  • github.com/go-task/slim-sprig: 348f09d → 52ccab3
  • github.com/golang/protobuf: v1.5.2 → v1.5.3
  • github.com/google/cel-go: v0.12.6 → v0.16.0
  • github.com/google/gofuzz: v1.1.0 → v1.2.0
  • github.com/google/pprof: 94a9f03 → 4bb14d4
  • github.com/ianlancetaylor/demangle: 5e5cf60 → 28f6c0f
  • github.com/kr/pretty: v0.2.0 → v0.3.1
  • github.com/mailru/easyjson: v0.7.6 → v0.7.7
  • github.com/matttproud/golang_protobuf_extensions: v1.0.2 → v1.0.4
  • github.com/moby/term: 39b0c02 → 1aeaba8
  • github.com/montanaflynn/stats: v0.6.6 → v0.7.0
  • github.com/onsi/ginkgo/v2: v2.8.1 → v2.11.0
  • github.com/onsi/gomega: v1.27.1 → v1.27.10
  • github.com/prometheus/client_golang: v1.14.0 → v1.16.0
  • github.com/prometheus/client_model: v0.3.0 → v0.4.0
  • github.com/prometheus/common: v0.37.0 → v0.44.0
  • github.com/prometheus/procfs: v0.8.0 → v0.10.1
  • github.com/rogpeppe/go-internal: v1.3.0 → v1.10.0
  • github.com/sirupsen/logrus: v1.8.1 → v1.9.0
  • github.com/stretchr/testify: v1.8.2 → v1.8.4
  • github.com/tmc/grpc-websocket-proxy: e5319fd → 673ab2c
  • go.etcd.io/bbolt: v1.3.6 → v1.3.7
  • go.etcd.io/etcd/api/v3: v3.5.5 → v3.5.9
  • go.etcd.io/etcd/client/pkg/v3: v3.5.5 → v3.5.9
  • go.etcd.io/etcd/client/v2: v2.305.5 → v2.305.9
  • go.etcd.io/etcd/client/v3: v3.5.5 → v3.5.9
  • go.etcd.io/etcd/pkg/v3: v3.5.5 → v3.5.9
  • go.etcd.io/etcd/raft/v3: v3.5.5 → v3.5.9
  • go.etcd.io/etcd/server/v3: v3.5.5 → v3.5.9
  • go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.0 → v0.35.1
  • go.uber.org/atomic: v1.7.0 → v1.10.0
  • go.uber.org/goleak: v1.2.0 → v1.2.1
  • go.uber.org/multierr: v1.6.0 → v1.11.0
  • golang.org/x/crypto: v0.8.0 → v0.12.0
  • golang.org/x/exp: 6cc2880 → a9213ee
  • golang.org/x/lint: 6edffad → 738671d
  • golang.org/x/mod: v0.8.0 → v0.10.0
  • golang.org/x/net: v0.9.0 → v0.13.0
  • golang.org/x/oauth2: ee48083 → v0.8.0
  • golang.org/x/sync: v0.1.0 → v0.3.0
  • golang.org/x/sys: v0.7.0 → v0.11.0
  • golang.org/x/term: v0.7.0 → v0.11.0
  • golang.org/x/text: v0.9.0 → v0.12.0
  • golang.org/x/time: 90d013b → v0.3.0
  • golang.org/x/tools: v0.6.0 → v0.9.3
  • golang.org/x/xerrors: 5ec99f8 → 04be3eb
  • google.golang.org/genproto: c8bf987 → 0005af6
  • google.golang.org/grpc: v1.49.0 → v1.54.0
  • google.golang.org/protobuf: v1.28.1 → v1.30.0
  • gopkg.in/check.v1: 8fa4692 → 10cb982
  • gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1
  • gopkg.in/square/go-jose.v2: v2.2.2 → v2.6.0
  • k8s.io/api: v0.26.3 → v0.28.0
  • k8s.io/apimachinery: v0.26.3 → v0.28.0
  • k8s.io/apiserver: v0.26.3 → v0.28.0
  • k8s.io/client-go: v0.26.3 → v0.28.0
  • k8s.io/cloud-provider: v0.26.3 → v0.28.0
  • k8s.io/component-base: v0.26.3 → v0.28.0
  • k8s.io/component-helpers: v0.26.3 → v0.28.0
  • k8s.io/controller-manager: v0.26.3 → v0.28.0
  • k8s.io/klog/v2: v2.90.1 → v2.100.1
  • k8s.io/kms: v0.26.3 → v0.28.0
  • k8s.io/kube-openapi: 172d655 → 2695361
  • k8s.io/kubelet: v0.26.3 → v0.28.0
  • k8s.io/utils: 99ec85e → d93618c
  • sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.36 → v0.1.2
  • sigs.k8s.io/json: f223a00 → bc3834c

Removed

  • github.com/PuerkitoBio/purell: v1.1.1
  • github.com/PuerkitoBio/urlesc: de5bf2a
  • github.com/alecthomas/template: fb15b89
  • github.com/antlr/antlr4/runtime/Go/antlr: v1.4.10
  • github.com/docopt/docopt-go: ee0de3b
  • github.com/elazarl/goproxy: 947c36d
  • github.com/form3tech-oss/jwt-go: v3.2.3+incompatible
  • github.com/go-kit/kit: v0.9.0
  • github.com/go-stack/stack: v1.8.0
  • github.com/google/gnostic: v0.5.7-v3refs
  • github.com/konsorten/go-windows-terminal-sequences: v1.0.3
  • github.com/kr/logfmt: b84e30a
  • github.com/mitchellh/mapstructure: v1.1.2
  • github.com/niemeyer/pretty: a10e7ca
  • gopkg.in/alecthomas/kingpin.v2: v2.2.6
  • gotest.tools/v3: v3.0.3
Last modified September 11, 2023: [Doc] Replace availabilitySetNodesCacheTTLInSeconds with availabilitySetsCacheTTLInSeconds (79513b138)