Fix CentOS VM not booting after recent patch

I am seeing CloudLinux7, and CentOS7 Virtual Machines patched to the latest kernel versions not booting on Xen based OnApp platforms.

Looking at the console they are getting the penguins per core, a few lines, then garbage, and hanging.

While this is being investigated, it’s probably wise to get the VM back on its feet again to return a resemblance of service again.

To get these back to life you will need access to the  OnApp Control Panel* and select for that VM “Reboot in Recovery” (ie restart the machine with essentially a recovery CD). Once up – log in with the same details as you usually would over SSH.

  fdisk -l

…. to find your largest drive there (or the drive most likely to contain your /boot partition)…

  mount /dev/chosendrive /mnt

…then edit the grub file…

  vim /boot/grub/grub.conf

…and have a look at the following:

  #
  ## Note that you do not have to rerun grub after making changes to this file
  ## NOTICE: You do not have a /boot partition. This means that
  ## all kernel and initrd paths are relative to /, eg.
  ## root (hd0,0)
  ## kernel /boot/vmlinuz-version ro root=/dev/sda1
  ## initrd /boot/initrd-[generic-]version.img
  ##boot=/dev/sda
  default=0
  timeout=10
  #splashimage=(hd0,0)/boot/grub/splash.xpm.gz
  #
  title CloudLinux (3.10.0-714.10.2.lve1.4.79.el7.x86_64) 7.4 (Georgy Grechko)
  root (hd0,0)
  kernel /boot/vmlinuz-3.10.0-714.10.2.lve1.4.79.el7.x86_64 root=/dev/xvda99 console=tty0 ro vconsole.keymap=us vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8
  initrd /boot/initramfs-3.10.0-714.10.2.lve1.4.79.el7.x86_64.img

  title CloudLinux (3.10.0-714.10.2.lve1.4.77.el7.x86_64) 7.4 (Georgy Grechko)
  root (hd0,0)
  kernel /boot/vmlinuz-3.10.0-714.10.2.lve1.4.77.el7.x86_64 root=/dev/xvda99 console=tty0 ro vconsole.keymap=us vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8
  initrd /boot/initramfs-3.10.0-714.10.2.lve1.4.77.el7.x86_64.img
 […]

The troublesome kernel you can see there is the 3.10.0-714.10.2.lve1.4.79.el7.x86_64 one – and we are good (with booting at least!) with the 3.10.0-714.10.2.lve1.4.77.el7.x86_64. The line you need to me looking at there is the DEFAULT= one – which will always launch the one at the top of the list unless you tell it otherwise. We, are about to do that. Change that from DEFAULT=0 to DEFAULT=1 so it looks a little more like this:

  #
  ## Note that you do not have to rerun grub after making changes to this file
  ## NOTICE: You do not have a /boot partition. This means that
  ## all kernel and initrd paths are relative to /, eg.
  ## root (hd0,0)
  ## kernel /boot/vmlinuz-version ro root=/dev/sda1
  ## initrd /boot/initrd-[generic-]version.img
  ##boot=/dev/sda
  default=1
  timeout=10
  #splashimage=(hd0,0)/boot/grub/splash.xpm.gz
  #
  title CloudLinux (3.10.0-714.10.2.lve1.4.79.el7.x86_64) 7.4 (Georgy Grechko)
  root (hd0,0)
  kernel /boot/vmlinuz-3.10.0-714.10.2.lve1.4.79.el7.x86_64 root=/dev/xvda99 console=tty0 ro vconsole.keymap=us vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8
  initrd /boot/initramfs-3.10.0-714.10.2.lve1.4.79.el7.x86_64.img

  title CloudLinux (3.10.0-714.10.2.lve1.4.77.el7.x86_64) 7.4 (Georgy Grechko)
  root (hd0,0) 
  kernel /boot/vmlinuz-3.10.0-714.10.2.lve1.4.77.el7.x86_64 root=/dev/xvda99 console=tty0 ro vconsole.keymap=us vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8
  initrd /boot/initramfs-3.10.0-714.10.2.lve1.4.77.el7.x86_64.img
  […]

Now ESC then :wq to write the file to disk and quit. Then.

  # sync
  # cd / 
  # umount /mnt
  # reboot

The VM should now restart with the older kernel, the one that was working before the upgrade.

I have raised this with the vendor – however, I am continuing to see a fair amount of this.

*If you have console access, you can also do this by selecting what kernel to boot from the menu if it is honouring that for you – if so – happy days, select a different kernel, boot that : )