Disk issues on CML cluster - performance and space

[cmm-cisco]

• CML Controller should use an SSD/balanced persistent disk instead of Standard persistent disk.

Performance has proven poor when doing any kind of storage operations, including adding new images to cloud storage. The system sometimes even freezes for everyone when doing disk intensive tasks.

https://cloud.google.com/compute/docs/disks/persistent-disks

[cmm-cisco]

Computes also run out of disk rather quickly. The compute machines have three locally attached 375G SSDs and seem to work well.

Machine type: n2-highmem-32 (32 vCPUs, 256 GB Memory)
CPU platform: Intel Cascade Lake

• Mount Logical Volume on /var/local/virl2/images consisting of locally attached SSDs as physical volumes. Use ext4 filesystem.

root@cml-compute-dt86:/var/local/virl2# pvs
  PV           VG                                      Fmt  Attr PSize    PFree
  /dev/nvme0n2 cml-compute-dt86-var-local-virl2-images lvm2 a--  <375.00g       0
  /dev/nvme0n3 cml-compute-dt86-var-local-virl2-images lvm2 a--  <375.00g       0
  /dev/nvme0n4 cml-compute-dt86-var-local-virl2-images lvm2 a--  <375.00g <100.99g

root@cml-compute-dt86:/var/local/virl2# vgs
  VG                                      #PV #LV #SN Attr   VSize  VFree
  cml-compute-dt86-var-local-virl2-images   3   1   0 wz--n- <1.10t <100.99g

root@cml-compute-dt86:/var/local/virl2# lvs
  LV   VG                                      Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv0  cml-compute-dt86-var-local-virl2-images -wi-ao---- 1.00t

/dev/mapper/cml--compute--dt86--var--local--virl2--images-lv0 1007G  105G  851G  11% /var/local/virl2/images

[cmm-cisco]

Computes also run out of disk rather quickly. The compute machines have three locally attached 375G SSDs and seem to work well.

Machine type: n2-highmem-32 (32 vCPUs, 256 GB Memory) CPU platform: Intel Cascade Lake

• Mount Logical Volume on /var/local/virl2/images consisting of locally attached SSDs as physical volumes. Use ext4 filesystem.

This is working really successfully on IDR. Just understand local disk will be lost when the VM is shutdown to STOP and not rebooted. It needs the same behavior as #6 .