E2E tests

• 1: Azure E2E tests
• 2: Kubernetes E2E tests

1 - Azure E2E tests

Overview

Here provides some E2E tests only specific to Azure provider.

Prerequisite

Deploy a Kubernetes cluster with Azure CCM

Refer step 1-3 in e2e-tests for deploying the Kubernetes cluster.

Setup Azure credentials

export AZURE_TENANT_ID=<tenant-id>                    # the tenant ID
export AZURE_SUBSCRIPTION_ID=<subscription-id>        # the subscription ID
export AZURE_CLIENT_ID=<service-principal-id>         # the service principal ID
export AZURE_CLIENT_SECRET=<service-principal-secret> # the service principal secret
export AZURE_ENVIRONMENT=<AzurePublicCloud>           # the cloud environment (optional, default is AzurePublicCloud)
export AZURE_LOCATION=<location>                      # the location
export AZURE_LOADBALANCER_SKU=<loadbalancer-sku>      # the sku of load balancer (optional, default is basic)

Setup KUBECONFIG

• Locate your kubeconfig and set it as env variable export KUBECONFIG=<kubeconfig> or cp <kubeconfig> ~/.kube/config

• Test it via kubectl version

Run tests

• Run default tests

  The following command ensures gingko v2 is installed and then runs default tests.

  make test-ccm-e2e

• Run specific tests

  go test -v ./tests/e2e/ -timeout 0 -ginkgo.focus <focus-keyword> --ginkgo.skip <skip-keyword>

After a long time test, a JUnit report will be generated in a directory named by the cluster name

2 - Kubernetes E2E tests

Prerequisite

• An azure service principal

  Please follow this guide for creating an azure service principal The service principal should either have:

  • Contributor permission of a subscription
  • Contributor permission of a resource group. In this case, please create the resource group first

• Docker daemon enabled

How to run Kubernetes e2e tests locally

1. Prepare dependency project

• kubectl

  Kubectl allows you to run command against Kubernetes cluster, which is also used for deploying CSI plugins. You can follow here to install kubectl. e.g. on Linux

  curl -LO https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
  chmod +x kubectl
  sudo mv kubectl /usr/local/bin/
  

2. Build docker images azure-cloud-controller-manager, azure-cloud-node-manager and push them to your image repository.

   git clone https://github.com/kubernetes-sigs/cloud-provider-azure $GOPATH/src/sigs.k8s.io/cloud-provider-azure
   cd $GOPATH/src/sigs.k8s.io/cloud-provider-azure
   export IMAGE_REGISTRY=<your-registry>
   export IMAGE_TAG=<tag>
   make image # build all images of different ARCHs and OSes
   make push # push all images of different ARCHs and OSes to your registry. Or manually `docker push`
   

3. Deploy a Kubernetes cluster with the above azure-cloud-controller-manager and azure-cloud-node-manager images.

   To deploy a cluster, export all the required environmental variables first and then invoke make deploy-cluster. Please notice that cluster-api-provider-azure is used to provision the management and workload clusters. To learn more about this provisioner, you can refer to its quick-start doc.

   export AZURE_SUBSCRIPTION_ID=<subscription-id>
   export AZURE_TENANT_ID=<tenant-id>
   export AZURE_CLIENT_ID=<client-id>
   export AZURE_CLIENT_SECRET=<client-secret>
   export CLUSTER_NAME=<cluster-name>
   export AZURE_RESOURCE_GROUP=<resource-group>
   export AZURE_CLOUD_CONTROLLER_MANAGER_IMG=<cloud-controller-manager-image>
   export AZURE_CLOUD_NODE_MANAGER_IMG=<cloud-node-manager-image>
   
   make deploy-cluster
   

   To connect the cluster:

   export KUBECONFIG=$GOPATH/src/sigs.k8s.io/cloud-provider-azure/$CLUSTER_NAME-kubeconfig
   kubectl cluster-info
   

   To check out more of the deployed cluster , replace kubectl cluster-info with other kubectl commands. To further debug and diagnose cluster problems, use kubectl cluster-info dump

4. Run Kubernetes E2E tests

   make test-e2e-capz