{"id":882,"date":"2020-03-24T16:35:54","date_gmt":"2020-03-24T23:35:54","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=882"},"modified":"2020-03-24T16:36:02","modified_gmt":"2020-03-24T23:36:02","slug":"creating-nested-vm-using-kvm-on-azure","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2020\/03\/24\/creating-nested-vm-using-kvm-on-azure\/","title":{"rendered":"Creating nested VM using KVM on Azure"},"content":{"rendered":"\n

I am working with a customer that has the use case of running nested virtual machines on Azure. They’re using KVM and QEMU today, and I wanted to prove out that it is possible to run VMs using KVM and QEMU on Azure.<\/p>\n\n\n\n

KVM <\/a>is a technology that allows you to run full virtual machines on top of Linux. KVM can access hardware virtualization technologies, meaning Intel VT or AMD-V. Without going in to much depth, Intel VT and AMD-V make it a lot more performant to run virtual machines, by offloading some of the virtualization translation steps to actual hardware.<\/p>\n\n\n\n

KVM is typically used in combination with QEMU<\/a>. QEMU is an open-source emulator that enables virtualization. QEMU makes use of KVM to access the Intel VT or AMD-V technologies.<\/p>\n\n\n\n

To make working with KVM and QEMU easier, you can use libvirt<\/a>. Libvirt is a toolkit to manage virtual machines. It interfaces with KVM and QEMU (or other platforms) to create and manage virtual machines. Rather than having to issue commands against the KVM or QEMU API, you typically use libvirt to issue the commands. <\/p>\n\n\n\n

To make all of this work in Azure, you’ll need to use a machine type that supports nested virtualization. The Dv3 and Ev3 support nested virtualization.<\/p>\n\n\n\n

In this post I’ll describe how to create a VM using nested virtualization, and how to get access to that VM. Let’s get started.<\/p>\n\n\n\n

Setting up the VM and installing all the required tools.<\/h2>\n\n\n\n

To start, we’ll need to create a new VM. I decided to go for a D4sv3, and I attached an additional 128GB premium disk to this new VM. I’ll use this additional disk to store the virtual disk for the target VM.<\/p>\n\n\n\n

\"\"
Details of the VM that will turn into a hypervisor<\/figcaption><\/figure>\n\n\n\n

Once the VM is up and running, SSH into it and run the following commands to install all the required tools:<\/p>\n\n\n\n

sudo apt-get update\nsudo apt-get install -y qemu qemu-kvm libvirt-bin  bridge-utils  virt-manager\nsudo service libvirtd start\nsudo update-rc.d libvirtd enable<\/code><\/pre>\n\n\n\n
To verify that everything is running as expected, you can run the following command:<\/p>\n\n\n\n
service libvirtd status<\/code><\/pre>\n\n\n\n
Which should show you something similar to:<\/p>\n\n\n\n
\"\"
Libvirt is up and running<\/figcaption><\/figure>\n\n\n\n
Setting up storage<\/h2>\n\n\n\n
To set up storage, we’ll have to do two tasks:<\/p>\n\n\n\n
  1. Mount the data disk<\/li>
  2. Create a pool in KVM<\/li><\/ol>\n\n\n\n
    To mount the disk, we’ll have to follow a couple of steps. First let’s create a partition table on the disk by using fdisk<\/code>. <\/p>\n\n\n\n
    sudo fdisk -l #note device name, \/dev\/sdc in my case\nsudo fdisk \/dev\/sdc\n# n to create new partition\n# p to make primary\n# 1 to create first partition\n# default\n# default\n# w to write and quit<\/code><\/pre>\n\n\n\n
    \"\"
    Partitioning the hard drive.<\/figcaption><\/figure>\n\n\n\n
    Next we’ll setup a file system using the following command:<\/p>\n\n\n\n
    sudo mkfs.ext4 \/dev\/sdc1<\/code><\/pre>\n\n\n\n
    And finally mounting the drive:<\/p>\n\n\n\n
    sudo mkdir \/vms\nsudo mount \/dev\/sdc1 \/vms<\/code><\/pre>\n\n\n\n
    To make our mount persist across reboots, we’ll also add an entry in the file system table (\/etc\/fstab<\/code>)<\/p>\n\n\n\n
    #first make a backup of \/etc\/fstab\nsudo cp \/etc\/fstab \/etc\/fstab.backup\nsudo bash -c 'echo \"\/dev\/sdc1 \/vms ext4\" >> \/etc\/fstab'<\/code><\/pre>\n\n\n\n
    Next up, we have to create the storage pool that KVM\/Qemu will use:<\/p>\n\n\n\n
    sudo virsh pool-define-as vms-store --type dir --target \/vms\nsudo virsh pool-start vms-store\nsudo virsh pool-autostart vms-store<\/code><\/pre>\n\n\n\n
    Networking<\/h2>\n\n\n\n
    The default installation created both a DHCP server and a virtual network bridge. To confirm the network bridge was setup, run an ifconfig<\/code>.<\/p>\n\n\n\n
    \"\"
    The installation caused a virtual bridge to be created.<\/figcaption><\/figure>\n\n\n\n
    KVM\/QEMU will use that network interface. To verify that configuration, run the following command:<\/p>\n\n\n\n
    sudo virsh net-dumpxml default<\/code><\/pre>\n\n\n\n
    Which will show you something like:<\/p>\n\n\n\n
    \"\"
    The network configuration of our virtual network.<\/figcaption><\/figure>\n\n\n\n
    We’ll revisit networking later on in this post to enable an external IP for the hosted VM. But for now, we can go ahead and create a first VM.<\/p>\n\n\n\n
    Creating a nested VM from an ISO<\/h2>\n\n\n\n
    With storage and networking, setup, we can go ahead and create a first virtual machine. To start, let’s download an ISO to install. In my case, I’ll run Ubuntu 18.04, and I’ll download the ISO on the temp drive of the VM.<\/p>\n\n\n\n
    sudo wget http:\/\/releases.ubuntu.com\/18.04.4\/ubuntu-18.04.4-live-server-am\nd64.iso?_ga=2.17338928.809071564.1585082124-624042363.1585082124 -O \/mnt\/ubuntu.iso<\/code><\/pre>\n\n\n\n
    With the ISO present, we can go ahead and create a VM. <\/p>\n\n\n\n
    sudo virt-install --virt-type=kvm --name=ubuntu-iso \\\n  --ram 1024 --vcpus=1 --virt-type=kvm --hvm \\\n  --cdrom \/mnt\/ubuntu.iso --network network=default \\\n  --disk pool=vms-store,size=20,bus=virtio,format=qcow2 \\\n  --graphics vnc<\/code><\/pre>\n\n\n\n
    This will start the VM creation process. We will need to finish the installation process by connecting to the console. To connect to the console, we’ll need to get the VNC port that our VM is using, tunnel VNC traffic over SSH, and then run a VNC viewer<\/a>.<\/p>\n\n\n\n
    To start, get the VNC port. <\/p>\n\n\n\n
    sudo virsh dumpxml ubuntu-iso | grep vnc<\/code><\/pre>\n\n\n\n
    Then, open a new SSH connection and tunnel that port of the connection. In my case, I’m running this on WSL, so I’ll tunnel over the SSH connection in a new tab. If you’re using putty, this can also be configured. In WSL use the following command:<\/p>\n\n\n\n
    ssh nilfranadmin@52.191.133.169 -L 5900:127.0.0.1:5900<\/code><\/pre>\n\n\n\n
    \"\"
    Set up a tunnel in Putty<\/figcaption><\/figure>\n\n\n\n
    Once the tunnel is up, you need to install VNC viewer<\/a>. Then you open VNC viewer and connect to 127.0.0.1:5900. This will show you the remote connection:<\/p>\n\n\n\n
    \"\"
    The Ubuntu installation wizard<\/figcaption><\/figure>\n\n\n\n
    We will use this terminal to finish creating the VM, and complete the installation.<\/p>\n\n\n\n
    \"\"
    Installing ubuntu<\/figcaption><\/figure>\n\n\n\n
    Once the installation is complete, you’ll have to manually start the VM again, using the following command:<\/p>\n\n\n\n
    sudo virsh start ubuntu-iso<\/code><\/pre>\n\n\n\n
    Finally, we will connect to the VM again using VNC and install an apache web server.<\/p>\n\n\n\n
    sudo apt-get update\nsudo apt-get install apache2 -y<\/code><\/pre>\n\n\n\n
    In the next step we will direct a public IP address on Azure to the virtual machine. <\/p>\n\n\n\n
    Mapping an Azure Public IP to the nested VM<\/h2>\n\n\n\n
    As a final step, I want to map a public IP in Azure to the VM in the nested virtual machine. I don’t want to use the VMs primary public IP address, I want to use a different address. This can be achieved by adding a secondary ip configuration to the NIC of the hypervisor.<\/p>\n\n\n\n
    \"\"
    Adding a secondary ip config with a public IP<\/figcaption><\/figure>\n\n\n\n
    Next, we’ll want to configure this IP address on the network interface. If you reboot, you don’t have to execute this command again, because Azure will send the secondary IP via DHCP as well.<\/p>\n\n\n\n
    sudo ip addr add 172.16.2.5 dev eth0<\/code><\/pre>\n\n\n\n
    And finally add a iptables route to forward any traffic coming into our VM on that IP to the IP of the VM. (to get the IP of the VM, do a ifconfig <\/code>in the VM). (in the below, 172.16.2.5 is the secondary IP of my NIC, and 192.168.122.215 is the IP of the nested VM).<\/p>\n\n\n\n
    sudo iptables -I FORWARD 1 -o virbr0 -m state -s 0.0.0.0\/0 -d 192.168.122.0\/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT\nsudo iptables -A PREROUTING -t nat -d 172.16.2.5 -p tcp --dport 1:65535 -j DNAT --to-destination 192.168.122.215:1-65535<\/code><\/pre>\n\n\n\n
    When this is complete, you should be able to connect to the public IP on the secondary IP config of the NIC:<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    And that’s it for now. We now have a nested VM running using KVM and Qemu.<\/p>\n\n\n\n
    Summary<\/h2>\n\n\n\n
    In this post, we used KVM, Qemu and libvirt to deploy a VM in a VM. We changed the networking configuration so we could expose the VM running in the VM using a public IP. <\/p>\n","protected":false},"excerpt":{"rendered":"
    I am working with a customer that has the use case of running nested virtual machines on Azure. They’re using KVM and QEMU today, and I wanted to prove out that it is possible to run VMs using KVM and QEMU on Azure. KVM is a technology that allows you to run full virtual machines […]<\/p>\n","protected":false},"author":1,"featured_media":889,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,4,5],"tags":[8,97,16],"class_list":["post-882","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-management","category-open-source","tag-azure","tag-nested-virtualization","tag-open-source"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2020\/03\/image-38.png","_links":{"self":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/882","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=882"}],"version-history":[{"count":2,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/882\/revisions"}],"predecessor-version":[{"id":895,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/882\/revisions\/895"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/889"}],"wp:attachment":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=882"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=882"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=882"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}