AzureDisk CSI Driver Known Issues

• azure disk plugin known issues
  • Recommended stable version for azure disk
  • 1. disk attach error
  • 2. disk unavailable after attach/detach a data disk on a node
  • 3. Azure disk support on Sovereign Cloud
  • 4. Time cost for Azure Disk PVC mount
  • 5. Azure disk PVC Multi-Attach error, makes disk mount very slow or mount failure forever
  • 6. WaitForAttach failed for azure disk: parsing “/dev/disk/azure/scsi1/lun1”: invalid syntax
  • 7. uid and gid setting in azure disk
  • 8. Addition of a blob based disk to VM with managed disks is not supported
  • 9. dynamic azure disk PVC try to access wrong storage account (of other resource group)
  • 10. data loss if using existing azure disk with partitions in disk mount
  • 11. Delete azure disk PVC which is already in use by a pod
  • 12. create azure disk PVC failed due to account creation failure
  • 13. cannot find Lun for disk
  • 14. azure disk attach/detach failure, mount issue, i/o error
  • 15. azure disk could be not detached forever
  • 16. potential race condition issue due to detach disk failure retry
  • 17. very slow disk attach/detach issue when disk num is large
  • 18. detach azure disk make VM run into a limbo state
  • 19. disk attach/detach self-healing on VMAS
  • 20. azure disk detach failure if node not exists
  • 21. invalid disk URI error
  • 22. vmss dirty cache issue
  • 23. race condition when delete disk right after attach disk
  • 24. attach disk costs 10min
  • 25. Multi-Attach error
  • 26. attached non-existing disk volume on agent node
  • 27. failed to get azure instance id for node (not a vmss instance)

Recommended stable version for azure disk

1. disk attach error

Issue details:

In some corner case(detaching multiple disks on a node simultaneously), when scheduling a pod with azure disk mount from one node to another, there could be lots of disk attach error(no recovery) due to the disk not being released in time from the previous node. This issue is due to lack of lock before DetachDisk operation, actually there should be a central lock for both AttachDisk and DetachDisk operations, only one AttachDisk or DetachDisk operation is allowed at one time.

The disk attach error could be like following:

Cannot attach data disk 'cdb-dynamic-pvc-92972088-11b9-11e8-888f-000d3a018174' to VM 'kn-edge-0' because the disk is currently being detached or the last detach operation failed. Please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.

Related issues

• Azure Disk Detach are not working with multiple disk detach on the same Node
• Azure disk fails to attach and mount, causing rescheduled pod to stall following node disruption
• Since Intel CPU Azure update, new Azure Disks are not mounting, very critical…
• Busy azure-disk regularly fail to mount causing K8S Pod deployments to halt

Mitigation:

• option#1: Update every agent node that has attached or detached the disk in problem

In Azure cloud shell, run

$vm = Get-AzureRMVM -ResourceGroupName $rg -Name $vmname
Update-AzureRmVM -ResourceGroupName $rg -VM $vm -verbose -debug

In Azure cli, run

az vm update -g <group> -n <name>

• option#2:

1. kubectl cordon node #make sure no scheduling on this node
2. kubectl drain node #schedule pod in current node to other node
3. restart the Azure VM for node via the API or portal, wait until VM is “Running”
4. kubectl uncordon node

Fix

• PR fix race condition issue when detaching azure disk has fixed this issue by add a lock before DetachDisk

2. disk unavailable after attach/detach a data disk on a node

💡 NOTE: Azure platform has fixed the host cache issue, the suggested host cache setting of data disk is ReadOnly now, more details about azure disk cache setting Issue details:

From k8s v1.7, default host cache setting changed from None to ReadWrite, this change would lead to device name change after attach multiple disks on a node, finally lead to disk unavailable from pod. When access data disk inside a pod, will get following error:

[root@admin-0 /]# ls /datadisk
ls: reading directory .: Input/output error

In my testing on Ubuntu 16.04 D2_V2 VM, when attaching the 6th data disk will cause device name change on agent node, e.g. following lun0 disk should be sdc other than sdk.

azureuser@k8s-agentpool2-40588258-0:~$ tree /dev/disk/azure
...
â""â"€â"€ scsi1
    â"œâ"€â"€ lun0 -> ../../../sdk
    â"œâ"€â"€ lun1 -> ../../../sdj
    â"œâ"€â"€ lun2 -> ../../../sde
    â"œâ"€â"€ lun3 -> ../../../sdf
    â"œâ"€â"€ lun4 -> ../../../sdg
    â"œâ"€â"€ lun5 -> ../../../sdh
    â""â"€â"€ lun6 -> ../../../sdi

Related issues

• device name change due to azure disk host cache setting
• unable to use azure disk in StatefulSet since /dev/sd* changed after detach/attach disk
• Disk error when pods are mounting a certain amount of volumes on a node
• unable to use azure disk in StatefulSet since /dev/sd* changed after detach/attach disk
• Input/output error when accessing PV
• PersistentVolumeClaims changing to Read-only file system suddenly

Workaround:

• add cachingmode: None in azure disk storage class(default is ReadWrite), e.g.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: hdd
provisioner: kubernetes.io/azure-disk
parameters:
  skuname: Standard_LRS
  kind: Managed
  cachingmode: None

Fix

• PR fix device name change issue for azure disk could fix this issue too, it will change default cachingmode value from ReadWrite to None.

3. Azure disk support on Sovereign Cloud

Fix

• PR Azure disk on Sovereign Cloud fixed this issue

4. Time cost for Azure Disk PVC mount

Original time cost for Azure Disk PVC mount on a standard node size(e.g. Standard_D2_V2) is around 1 minute, podAttachAndMountTimeout is 2 minutes, total waitForAttachTimeout is 10 minutes, so a disk remount(detach and attach in sequential) would possibly cost more than 2min, thus may fail.

Note: for some smaller VM size which has only 1 CPU core, time cost would be much bigger(e.g. > 10min) since container is hard to get CPU slot.

Related issues

• ’timeout expired waiting for volumes to attach/mount for pod when cluster’ when node-vm-size is Standard_B1s

Fix

• PR using cache fix fixed this issue, which could reduce the mount time cost to around 30s.

5. Azure disk PVC Multi-Attach error, makes disk mount very slow or mount failure forever

💡 NOTE: AKS and current aks-engine won’t have this issue since it’s not using containerized kubelet

Issue details:

When schedule a pod with azure disk volume from one node to another, total time cost of detach & attach is around 1 min from v1.9.2, while in v1.9.x, there is an UnmountDevice failure issue in containerized kubelet which makes disk mount very slow or mount failure forever, this issue only exists in v1.9.x due to PR Refactor nsenter, v1.10.0 won’t have this issue since devicePath is updated in v1.10 code

error logs:

• kubectl describe po POD-NAME

Events:
  Type     Reason                 Age   From                               Message
  ----     ------                 ----  ----                               -------
  Normal   Scheduled              3m    default-scheduler                  Successfully assigned deployment-azuredisk1-6cd8bc7945-kbkvz to k8s-agentpool-88970029-0
  Warning  FailedAttachVolume     3m    attachdetach-controller            Multi-Attach error for volume "pvc-6f2d0788-3b0b-11e8-a378-000d3afe2762" Volume is already exclusively attached to one node and can't be attached to another
  Normal   SuccessfulMountVolume  3m    kubelet, k8s-agentpool-88970029-0  MountVolume.SetUp succeeded for volume "default-token-qt7h6"
  Warning  FailedMount            1m    kubelet, k8s-agentpool-88970029-0  Unable to mount volumes for pod "deployment-azuredisk1-6cd8bc7945-kbkvz_default(5346c040-3e4c-11e8-a378-000d3afe2762)": timeout expired waiting for volumes to attach/mount for pod "default"/"deployment-azuredisk1-6cd8bc7945-kbkvz". list of unattached/unmounted volumes=[azuredisk]

• kubelet logs from the new node

E0412 20:08:10.920284    7602 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/azure-disk//subscriptions/xxx/resourceGroups/MC_xxx_eastus/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-11035a31-3e8d-11e8-82ec-0a58ac1f04cf\"" failed. No retries permitted until 2018-04-12 20:08:12.920234762 +0000 UTC m=+1467.278612421 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-11035a31-3e8d-11e8-82ec-0a58ac1f04cf\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/xxx/resourceGroups/MC_xxx_eastus/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-11035a31-3e8d-11e8-82ec-0a58ac1f04cf\") pod \"symbiont-node-consul-0\" (UID: \"11043b12-3e8d-11e8-82ec-0a58ac1f04cf\") "

Related issues

• UnmountDevice would fail in containerized kubelet
• upgrade k8s process is broke

Mitigation:

If azure disk PVC mount successfully in the end, there is no action, while if it could not be mounted for more than 20min, following actions could be taken:

• check whether volumesInUse list has unmounted azure disks, run:

kubectl get no NODE-NAME -o yaml > node.log

all volumes in volumesInUse should be also in volumesAttached, otherwise there would be issue

• restart kubelet on the original node would solve this issue: sudo kubectl kubelet restart

Fix

• PR fix nsenter GetFileType issue in containerized kubelet fixed this issue

After fix in v1.9.7, it took about 1 minute for scheduling one azure disk mount from one node to another, you could find details here.

Since azure disk attach/detach operation on a VM cannot be parallel, scheduling 3 azure disk mounts from one node to another would cost about 3 minutes.

6. WaitForAttach failed for azure disk: parsing “/dev/disk/azure/scsi1/lun1”: invalid syntax

Issue details: MountVolume.WaitForAttach may fail in the azure disk remount

error logs:

in v1.10.0 & v1.10.1, MountVolume.WaitForAttach will fail in the azure disk remount, error logs would be like following:

• incorrect DevicePath format on Linux

MountVolume.WaitForAttach failed for volume "pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967" : azureDisk - Wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun1 (strconv.Atoi: parsing "/dev/disk/azure/scsi1/lun1": invalid syntax)
  Warning  FailedMount             1m (x10 over 21m)   kubelet, k8s-agentpool-66825246-0  Unable to mount volumes for pod

• wrong DevicePath(LUN) number on Windows

  Warning  FailedMount             1m    kubelet, 15282k8s9010    MountVolume.WaitForAttach failed for volume "disk01" : azureDisk - WaitForAttach failed within timeout node (15282k8s9010) diskId:(andy-mghyb
1102-dynamic-pvc-6c526c51-4a18-11e8-ab5c-000d3af7b38e) lun:(4)

Related issues

• WaitForAttach failed for azure disk: parsing “/dev/disk/azure/scsi1/lun1”: invalid syntax
• Pod unable to attach PV after being deleted (Wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun0 (strconv.Atoi: parsing “/dev/disk/azure/scsi1/lun0”: invalid syntax)

Fix

• PR fix WaitForAttach failure issue for azure disk fixed this issue

7. uid and gid setting in azure disk

Issue details: Unlike azure file mountOptions, you will get following failure if set mountOptions like uid=999,gid=999 in azure disk mount:

Warning  FailedMount             63s                  kubelet, aks-nodepool1-29460110-0  MountVolume.MountDevice failed for volume "pvc-d783d0e4-85a1-11e9-8a90-369885447933" : azureDisk - mountDevice:FormatAndMount failed with mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m436970985 --scope -- mount -t xfs -o dir_mode=0777,file_mode=0777,uid=1000,gid=1000,defaults /dev/disk/azure/scsi1/lun2 /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m436970985
Output: Running scope as unit run-rb21966413ab449b3a242ae9b0fbc9398.scope.
mount: wrong fs type, bad option, bad superblock on /dev/sde,
       missing codepage or helper program, or other error

That’s because azureDisk use ext4,xfs file system by default, mountOptions like [uid=x,gid=x] could not be set in mount time.

Related issues

• Timeout expired waiting for volumes to attach
• Pod failed mounting xfs format volume with mountOptions
• Allow volume ownership to be only set after fs formatting

Solution:

• option#1: Set uid in runAsUser and gid in fsGroup for pod: security context for a Pod

e.g. Following setting will set pod run as root, make it accessible to any file:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 0
    fsGroup: 0

Note: Since gid & uid is mounted as 0(root) by default, if set as non-root(e.g. 1000), k8s will use chown to change all dir/files under that disk, this is a time consuming job, which would make mount device very slow, in this issue: Timeout expired waiting for volumes to attach, it costs about 10 min for chown operation complete.

• option#2: use chown in initContainers

initContainers:
- name: volume-mount
  image: busybox
  command: ["sh", "-c", "chown -R 100:100 /data"]
  volumeMounts:
  - name: <your data volume>
    mountPath: /data

• new upstream feature to address this: Allow volume ownership to be only set after fs formatting

8. Addition of a blob based disk to VM with managed disks is not supported

Issue details:

Following error may occur if attach a blob based(unmanaged) disk to VM with managed disks:

  Warning  FailedMount            42s (x2 over 1m)  attachdetach                    AttachVolume.Attach failed for volume "pvc-f17e5e77-474e-11e8-a2ea-000d3a10df6d" : Attach volume "holo-k8s-dev-dynamic-pvc-f17e5e77-474e-11e8-a2ea-000d3a10df6d" to instance "k8s-master-92699158-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Addition of a blob based disk to VM with managed disks is not supported."

This issue is by design as in Azure, there are two kinds of disks, blob based(unmanaged) disk and managed disk, an Azure VM could not attach both of these two kinds of disks.

Solution:

Use default azure disk storage class in acs-engine, as default will always be identical to the agent pool, that is, if VM is managed, it will be managed azure disk class, if unmanaged, then it’s unmanaged disk class.

9. dynamic azure disk PVC try to access wrong storage account (of other resource group)

Issue details:

In a k8s cluster with blob based VMs(won’t happen in AKS since AKS only use managed disk), create dynamic azure disk PVC may fail, error logs is like following:

Failed to provision volume with StorageClass "default": azureDisk - account ds6c822a4d484211eXXXXXX does not exist while trying to create/ensure default container

Related issues

• Multiple clusters - dynamic PVCs try to access wrong storage account (of other resource group)

Fix

• PR fix storage account not found issue: use ListByResourceGroup instead of List() fixed this issue

Work around:

this bug only exists in blob based VM in v1.8.x, v1.9.x, so if specify ManagedDisks when creating k8s cluster in acs-engine(AKS is using managed disk by default), it won’t have this issue:

    "agentPoolProfiles": [
      {
        ...
        "storageProfile" : "ManagedDisks",
        ...
      }

10. data loss if using existing azure disk with partitions in disk mount

Issue details:

When use an existing azure disk(also called static provisioning) in pod, if that disk has partitions, the disk will be formatted in the pod mounting process, actually k8s volume don’t support mount disk with partitions, disk mount would fail finally. While for mounting existing azure disk that has partitions, data will be lost since it will format that disk first. This issue happens only on Linux.

Related issues

• data loss if using existing azure disk with partitions in disk mount

Fix

• PR fix data loss issue if using existing azure disk with partitions in disk mount will let azure provider return error when mounting existing azure disk that has partitions

Work around:

Don’t use existing azure disk that has partitions, e.g. following disk in LUN 0 that has one partition:

azureuser@aks-nodepool1-28371372-0:/$ ls -l /dev/disk/azure/scsi1/
total 0
lrwxrwxrwx 1 root root 12 Apr 27 08:04 lun0 -> ../../../sdc
lrwxrwxrwx 1 root root 13 Apr 27 08:04 lun0-part1 -> ../../../sdc1

11. Delete azure disk PVC which is already in use by a pod

Issue details:

Following error may occur if delete azure disk PVC which is already in use by a pod:

kubectl describe pv pvc-d8eebc1d-74d3-11e8-902b-e22b71bb1c06
...
Message:         disk.DisksClient#Delete: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="OperationNotAllowed" Message="Disk kubernetes-dynamic-pvc-d8eebc1d-74d3-11e8-902b-e22b71bb1c06 is attached to VM /subscriptions/{subs-id}/resourceGroups/MC_markito-aks-pvc_markito-aks-pvc_westus/providers/Microsoft.Compute/virtualMachines/aks-agentpool-25259074-0."

Fix:

This is a common k8s issue, other cloud provider would also has this issue. There is a PVC protection feature to prevent this, it’s alpha in v1.9, and beta(enabled by default) in v1.10

Work around: delete pod first and then delete azure disk pvc after a few minutes

12. create azure disk PVC failed due to account creation failure

please note this issue only happens on unmanaged k8s cluster

Issue details: User may get Account property kind is invalid for the request error when trying to create a new unmanaged azure disk PVC, error would be like following:

azureuser@k8s-master-17140924-0:/tmp$ kubectl describe pvc
Name:          pvc-azuredisk
Namespace:     default
StorageClass:  hdd
Status:        Bound
...
Events:
  Type     Reason                 Age                From                         Message
  ----     ------                 ----               ----                         -------
  Warning  ProvisioningFailed     31m                persistentvolume-controller  Failed to provision volume with StorageClass "hdd": Create Storage Account: ds10e15ed89c5811e8a0a70, error: storage.AccountsClient#Create: Failure sending request: StatusCode=400 -- Original Error: Code="AccountPropertyIsInvalid" Message="Account property kind is invalid for the request."

Fix

• PR fix azure disk create failure due to sdk upgrade fixed this issue

Work around:

• create a storage account and specify that account in azure disk storage class, e.g.

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: ssd
provisioner: kubernetes.io/azure-disk
parameters:
  skuname: Premium_LRS
  storageAccount: customerstorageaccount
  kind: Dedicated

13. cannot find Lun for disk

Issue details:

Following error may occur if attach a disk to a node:

MountVolume.WaitForAttach failed for volume "pvc-12b458f4-c23f-11e8-8d27-46799c22b7c6" : Cannot find Lun for disk kubernetes-dynamic-pvc-12b458f4-c23f-11e8-8d27-46799c22b7c6

Related issues

• GetAzureDiskLun sometimes costs 1 min which is too long time

Fix

• PR fix azure disk attachment error on Linux will extract the LUN num from device path only on Linux

Work around:

wait for a few more minutes should work

14. azure disk attach/detach failure, mount issue, i/o error

Issue details:

We found a disk attach/detach issue due to dirty vm cache PR introduced from v1.9.2, it would lead to following disk issues:

• disk attach/detach failure for a long time
• disk I/O error
• unexpected disk detachment from VM
• VM running into failed state due to attaching non-existing disk

Note: above error may only happen when there are multiple disk attach/detach operations in parallel and it’s not easy to repro since it happens on a little possibility.

Related issues

• Azure Disks volume attach still times out on Kubernetes 1.10
• Azure Disks occasionally mounted in a way leading to I/O errors

Fix

We changed the azure disk attach/detach retry logic in k8s v1.13, switch to use k8s attach-detach controller to do attach/detach disk retry and clean vm cache after every disk operation, this issue is proved to be fixed in our disk attach/detach stress test and also verified in customer env:

• PR remove retry operation on attach/detach azure disk in azure cloud provider
• PR fix azure disk attach/detach failed forever issue
• PR fix detach azure disk issue due to dirty cache

Work around:

• if there is attach disk failure for long time, restart controller manager may work
• if there is disk not detached for long time, detach that disk manually

Related issues

• Multi Attach Error

15. azure disk could be not detached forever

Issue details:

In some condition when first detach azure disk operation failed, it won’t retry and the azure disk would be still attached to the original VM node.

Following error may occur when move one disk from one node to another(keyword: ConflictingUserInput):

[Warning] AttachVolume.Attach failed for volume "pvc-7b7976d7-3a46-11e9-93d5-dee1946e6ce9" : Attach volume "kubernetes-dynamic-pvc-7b7976d7-3a46-11e9-93d5-dee1946e6ce9" to instance "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Compute/virtualMachines/aks-agentpool-57634498-0" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status= Code="ConflictingUserInput" Message="Disk '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-7b7976d7-3a46-11e9-93d5-dee1946e6ce9' cannot be attached as the disk is already owned by VM '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Compute/virtualMachines/aks-agentpool-57634498-1'."

Fix

We added retry logic for detach azure disk:

• PR add retry for detach azure disk

Work around:

• if there is disk not detached for long time, detach that disk manually

16. potential race condition issue due to detach disk failure retry

Issue details:

In some error condition when detach azure disk failed, azure cloud provider will retry 6 times at most with exponential backoff, it will hold the data disk list for about 3 minutes with a node level lock, and in that time period, if customer update data disk list manually (e.g. need manual operationto attach/detach another disk since there is attach/detach error, ) , the data disk list will be obsolete(dirty data), then weird VM status happens, e.g. attach a non-existing disk, we should split those retry operations, every retry should get a fresh data disk list in the beginning.

Fix

Following PR refined detach azure disk retry operation, make every detach azure disk operation in a standalone function

• PR fix detach azure disk back off issue which has too big lock in failure retry condition
• PR fix azure disk list corruption issue

Work around:

Detach all the non-existing disks from VM (could do that in azure portal by bulk update)

Detaching disk one by one using cli may fail since they are already non-existing disks.

17. very slow disk attach/detach issue when disk num is large

Issue details:

We hit very slow disk attach/detach issue when disk num is large(> 10 disks on one VM)

Fix

Azure disk team are fixing this issue.

Work around:

No workaround.

18. detach azure disk make VM run into a limbo state

Issue details:

In some corner condition, detach azure disk would sometimes make VM run into a limbo state

Fix

Following two PRs would fix this issue by retry update VM if detach disk partially fail:

• fix azure retry issue when return 2XX with error
• fix: retry detach azure disk issue

Work around:

Update VM status manually would solve the problem:

• Update Availability Set VM

az vm update -n <VM_NAME> -g <RESOURCE_GROUP_NAME>

• Update Scale Set VM

az vmss update-instances -g <RESOURCE_GROUP_NAME> --name <VMSS_NAME> --instance-id <ID(number)>

19. disk attach/detach self-healing on VMAS

Issue details: There could be disk detach failure due to many reasons(e.g. disk RP busy, controller manager crash, etc.), and it would fail when attach one disk to other node if that disk is still attached to the old node, user needs to manually detach disk in problem in the before, with this fix, azure cloud provider would check and detach this disk if it’s already attached to the other node, that’s like self-healing. This PR could fix lots of such disk attachment issue.

Fix

Following PR would first check whether current disk is already attached to other node, if so, it would trigger a dangling error and k8s controller would detach disk first, and then do the attach volume operation.

This PR would also fix a “disk not found” issue when detach azure disk due to disk URI case sensitive case, error logs are like following(without this PR):

azure_controller_standard.go:134] detach azure disk: disk  not found, diskURI: /subscriptions/xxx/resourceGroups/andy-mg1160alpha3/providers/Microsoft.Compute/disks/xxx-dynamic-pvc-41a31580-f5b9-4f08-b0ea-0adcba15b6db

Fix

• Fix on VMAS
  • fix: detach azure disk issue using dangling error
  • fix: azure disk name matching issue

• Fix on VMSS
  • fix: azure disk dangling attach issue on VMSS which would cause API throttling

Work around:

manually detach disk in problem and wait for disk attachment happen automatically

20. azure disk detach failure if node not exists

Issue details: If a node with a Azure Disk attached is deleted (before the volume is detached), subsequent attempts by the attach/detach controller to detach it continuously fail, and prevent the controller from attaching the volume to another node.

Fix

• fix: azure disk detach failure if node not exists

Work around:

Restart kube-controller-manager on master node.

21. invalid disk URI error

Issue details:

When user use an existing disk in static provisioning, may hit following error:

AttachVolume.Attach failed for volume "azure" : invalid disk URI: /subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.Compute/disks/Test_Resize_1/"

Fix

• fix: make azure disk URI as case insensitive

Work around:

Use resourceGroups instead of resourcegroups in disk PV configuration

22. vmss dirty cache issue

Issue details:

clean vmss cache should happen after disk attach/detach operation, now it’s before those operations, which would lead to dirty cache. since update operation may cost 30s or more, and at that time period, if there is another get vmss operation, it would get the old data disk list

• VMSS disk attach/detach issues w/ v1.13.12, v1.14.8, v1.15.5, v1.16.2
• Disk attachment/mounting problems, all pods with PVCs stuck in ContainerCreating

Fix

• fix vmss dirty cache issue

Work around:

Detach disk in problem manually

23. race condition when delete disk right after attach disk

Issue details:

There is condition that attach and delete disk happens in same time, azure CRP don’t check such race condition

• should not delete an azure disk when that disk is being attached

Fix

• fix race condition when delete azure disk right after attach azure disk

Work around:

Detach disk in problem manually

24. attach disk costs 10min

Issue details:

PR Fix aggressive VM calls for Azure VMSS change getVMSS cache TTL from 1min to 10min, getVMAS cache TTL from 5min to 10min, that will cause error WaitForAttach ... Cannot find Lun for disk, and it would make attach disk operation costs 10min on VMSS and 15min on VMAS, detailed error would be like following:

Events:
  Type     Reason                  Age                 From                                        Message
  ----     ------                  ----                ----                                        -------
  Normal   Scheduled               29m                 default-scheduler                           Successfully assigned authentication/authentication-mssql-statefulset-0 to aks-nodepool1-29122124-vmss000004
  Normal   SuccessfulAttachVolume  28m                 attachdetach-controller                     AttachVolume.Attach succeeded for volume "pvc-8d9f0ade-1825-11ea-83a0-22ced17d4a3d"
  Warning  FailedMount             23m (x10 over 27m)  kubelet, aks-nodepool1-29122124-vmss000004  MountVolume.WaitForAttach failed for volume "pvc-8d9f0ade-1825-11ea-83a0-22ced17d4a3d" : Cannot find Lun for disk kubernetes-dynamic-pvc-8d9f0ade-1825-11ea-83a0-22ced17d4a3d
  Warning  FailedMount             23m (x3 over 27m)   kubelet, aks-nodepool1-29122124-vmss000004  Unable to mount volumes for pod "authentication-mssql-statefulset-0_authentication(8df467e7-1825-11ea-83a0-22ced17d4a3d)": timeout expired waiting for volumes to attach or mount for pod "authentication"/"authentication-mssql-statefulset-0". list of unmounted volumes=[authentication-mssql-persistent-data-storage]. list of unattached volumes=[authentication-mssql-persistent-data-storage default-token-b7spv]
  Normal   Pulled                  21m                 kubelet, aks-nodepool1-29122124-vmss000004  Container image "mcr.microsoft.com/mssql/server:2019-CTP3.2-ubuntu" already present on machine
  Normal   Created                 21m                 kubelet, aks-nodepool1-29122124-vmss000004  Created container authentication-mssql
  Normal   Started                 21m                 kubelet, aks-nodepool1-29122124-vmss000004  Started container authentication-mssql

This slow disk attachment issue only exists on 1.13.12+, 1.14.8+, fortunately, from k8s 1.15.0, this issue won’t happen, since getDiskLUN logic has already been refactored (already has PR:fix azure disk lun error, won’t depend on getVMSS operation to get disk LUN.

Relate issues:

• GetAzureDiskLun sometimes costs 10min which is too long time

Fix

• fix azure disk lun error

Work around:

Wait for about 10min or 15min, MountVolume.WaitForAttach operation would retry and would finally succeed

25. Multi-Attach error

Issue details:

If two pods on different nodes are using same disk PVC(this issue may also happen when doing rollingUpdate in Deployment using one replica), would probably hit following error:

Events:
Warning  FailedAttachVolume  9m                attachdetach-controller                     Multi-Attach error for volume "pvc-fc0bed38-48bf-43f1-a7e4-255eef48ffb9" Volume is already used by pod(s) sqlserver3-5b8449449-5chzx
Warning  FailedMount         42s (x4 over 7m)  kubelet, aks-nodepool1-15915763-vmss000001  Unable to mount volumes for pod "sqlserver3-55754785bb-jjr6d_default(55381f38-9640-43a9-888d-096387cbb780)": timeout expired waiting for volumes to attach or mount for pod "default"/"sqlserver3-55754785bb-jjr6d". list of unmounted volumes=[mssqldb]. list of unattached volumes=[mssqldb default-token-q7cw9]

The above issue is upstream issue(detailed error code), it could be due to following reasons:

• two pods are using same disk PVC, this issue could happen even using Deployment with one replica(see below workaround)
• one node is in Shutdown(deallocated) state, this is by design now and there is on-going upstream work to fix this issue
  • Propose to taint node “shutdown” condition
  • add node shutdown KEP

• workaround: user could use set terminationGracePeriodSeconds: 0 in deployment or kubectl delete pod PODNAME --grace-period=0 --force to delete pod on the deallocated node
• Azure cloud provider solution: delete shutdown node(in InstanceExistsByProviderID) like what other cloud provider does today, while it may lead to other problem(e.g. node label loss), see details: Common handling of stopped instances across cloud providers.

since azure disk PVC could not be attached to one node.

Relate issues:

• Trouble attaching volume

Work around:

When using disk PVC config in deployment, maxSurge: 0 could make sure there would not be no more than two pods in Running/ContainerCreating state when doing rollingUpdate:

template:
...
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

Refer to Rolling Updates with Kubernetes Deployments for more detailed rollingUpdate config, and you could find maxSurge: 0 setting example here

Note

• error messages:
  • Multi-Attach error for volume "pvc-e9b72e86-129a-11ea-9a02-9abdbf393c78" Volume is already used by pod(s)

two pods are using same disk PVC, this issue could happen even using Deployment with one replica, check detailed explanation and workaround here with above explanation

• Multi-Attach error for volume "pvc-0d7740b9-3a43-11e9-93d5-dee1946e6ce9" Volume is already exclusively attached to one node and can't be attached to another

This could be a transient error when move volume from one node to another, use following command to find attached node:

kubectl get no -o yaml | grep volumesAttached -A 15 | grep pvc-0d7740b9-3a43-11e9-93d5-dee1946e6ce9 -B 10 -A 15

related code: reportMultiAttachError

26. attached non-existing disk volume on agent node

Issue details:

There is little possibility that attach/detach disk and disk deletion happened in same time, that would cause race condition. This PR add remediation when attach/detach disk, if returned 404 error, it will filter out all non-existing disks and try attach/detach operation again.

Fix

• fix: add remediation in azure disk attach/detach
• fix: azure disk remediation issue

Work around:

Detach disk in problem manually

27. failed to get azure instance id for node (not a vmss instance)

Issue details:

PR#81266 does not convert the VMSS node name which causes error like this:

failed to get azure instance id for node \"k8s-agentpool1-32474172-vmss_1216\" (not a vmss instance)

That will make dangling attach return error, and k8s volume attach/detach controller will getVmssInstance, and since the nodeName is in an incorrect format, it will always clean vmss cache if node not found, thus incur a get vmss API call storm.

Fix

• fix: azure disk dangling attach issue on VMSS which would cause API throttling

Work around:

1. Stop kube-controller-manager
2. detach disk in problem from that vmss node manually

az vmss disk detach -g <RESOURCE_GROUP_NAME> --name <VMSS_NAME> --instance-id <ID(number)> --lun number

e.g. per below logs,

E0501 11:15:40.981758       1 attacher.go:277] failed to detach azure disk "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Compute/disks/rg-dynamic-pvc-dc282131-b669-47db-8d57-cb3b9789ac3e", err failed to get azure instance id for node "k8s-agentpool1-32474172-vmss_1216" (not a vmss instance)

• find lun number of disk rg-dynamic-pvc-dc282131-b669-47db-8d57-cb3b9789ac3e:

az vmss show -g rg --name k8s-agentpool1-32474172-vmss --instance-id 1216

• detach vmss disk manually:

az vmss disk detach -g rg --name k8s-agentpool1-32474172-vmss --instance-id 1216 --lun number

3. Start kube-controller-manager