With the videos released last month from euroBSDcon 2012, I watched Michael Dexter’s talk on bhyve, the BSD hypervisor has come along way since I last tried it over a year ago & Michael has helped a with it’s progress by writing articles on CFT & scripts for running bhyve.
Last week I decided to get myself a server which I could use to do builds quickly & to run virtual machines for testing. Hetzner do high spec consumer hardware as servers, €59 per month get you a i7 with 32GB of RAM & 2x 3TB HDD, I ordered the server along with a 16GB USB flash drive with the plan of running SmartOS, once my login details for the server came through, I raised a support ticket for access to a IP KVM, within the hour I was given access & the installation went seamlessly. SmartOS was running on my server & it all went down hill from there.
As there is a IPv4 address shortage, hetzner charge a premium for additional addresses as a routed subnet, along with an additional fee for having the ability to request additional addressses as a “flexi pack”, a /27 would cost €47, I was not going to pay this so decided to go IPv6 only as I have connectivity at home & work. Unfortunately, though IPv6 support is there in the core of SmartOS by interitence from OpenSolaris, the additions from Joyent for KVM don’t, main culprit being vmadm(1m), after losing two days trying to get things working I came to the conclusion that A) it would be a big pain to maintain going forward as the burden would be on me to work around the shortfalls of the system B) I didn’t want to maintain my own release with third party patches which were not in yet C) I didn’t like the way I would have to extend the system to add functionality eg to set the hostname for your system persistently you have to use a script D) getting IPv6 support to guests was painful.
The majority of the work I’m doing is oriented around FreeBSD, it takes over 4 hours to do a build world & kernel on my ThinkPad X61s with a 1.6GHz Core2Duo so anything that can prolong it’s life & give me new builds quickly is good. I placed another support request for IP KVM (LARA in the world of hetzner) & once I had the login details I netbooted the server to their FreeBSD rescue environment which is a FreeBSD 8.3 based copy of mfsBSD. From there I fetched the latest FreeBSD-CURRENT usb image & wrote it to the flash drive using dd(1) & went about setting up a mirrored zpool to install FreeBSD onto.
Once the installation was complete & the system was up & running I revisited Michael’s talk, slides & scripts.
His scripts are numbered sequentially so you can easily go from creating a disk image to running & managing your virtual machines. This article covers a summary of what is involved to get a guest VM ready with FreeBSD-CURRENT built from source which are taken from his scripts & slides. As development has progressed since the talk, some things which are performed are no longer required. Essentially, you can boot a stock system from a disk image with only 2 necessary modifications to stock configuration files for dealing with the console.
There is also a vmrun.sh script which simplifies the whole process to try out (see instructions)
First build world & kernel (not necessary, you can use the precompiled binary instead if you choose)
On the host add the following to /boot/loader.conf
vmm_load="YES"
if_tap_load="YES"
bridgestp_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
Create a file which will be used as your disk, eg a 80GB one
truncate -s 80G disk.img
Create a md(4) disk with the file you just created
mdconfig disk.img
Initialise the disk to use the entire disk as a freebsd slice
fdisk -BI md0
You’ll receive the following error which can be safely ignored
******* Working on device /dev/md0 *******
fdisk: invalid fdisk partition table found
Write a standard label & boot code to slice 1
bsdlabel -wB /dev/md0s1
Write a filesystem to slice 1a
newfs -U /dev/md0s1a
Mount it to /mnt
mount /dev/md0s1a /mnt
From /usr/src, install world, kernel & distribution (contents of /etc) onto the disk image
make installworld DESTDIR=/mnt
make installkernel DESTDIR=/mnt
make distribution DESTDIR=/mnt
Setup your fstab to mount root from /dev/vtbd0s1a
echo "/dev/vtbd0s1a / ufs rw 1 1" > /mnt/etc/fstab
Configure your console
echo 'console "/usr/libexec/getty std.9600" vt100 on secure' > /mnt/etc/ttys
echo 'console="userboot"' > /mnt/boot/loader.conf
Aside from configuring /etc/rc.conf the instructions above cover the bare minimum to get a booting VM.
From Michael’s 2-install-guest.sh I’ve skipped loading the virtio drivers in /boot/loader.conf as they’re loaded by default in FreeBSD-CURRENT & the following though I’ve not given it more testing
Helps Kernel detected it’s running in a virtualised environment
smbios.bios.vendor="Bochs"
Avoid clock drift
kern.timecounter.hardware="TSC"
kern.timecounter.invariant_tsc="1"
PCI pass-through support as it caused hangs
hw.pci.enable_msix="0"
hw.pci.honor_msi_blacklist="0"
Unmount the file system
umount /mnt
Detach the file from md(4)
mdconfig -d -u 0
Assuming you’re using md0
You can get a list of configured devices with
mdconfig -l
As covered in 3-host-prep.sh you can load the required kernel modules for bhyve & guest networking by running
kldload vmm or rebooting
kldload if_tap
kldload bridgestp
kldload if_bridge
Before starting your VM, you need to create the needed interfaces, a tap(4) interfaces with a bridge(4) linked to the interface you want the VM to be able to communicate on, in my case a re(4)
ifconfig tap0 create up
ifconfig bridge0 create up
ifconfig bridge0 addm tap0 addm re0 up
Because of STP, once you have started the virtual machine, you should pause at the boot menu by pressing space & waiting 20 seconds until STP has stabilised otherwise you may find strange issues with you guest not being able to communicate properly.
If you restart a VM, it is also important to destroy the tap & bridge interfaces before starting up again or you will again experience odd behaviour e.g I was seeing traffic come in to the VM but not going out.
ifconfig tap0 destroy
ifconfig bridge0
To start a VM with less than 4GB RAM issue
sudo bhyveload -d /path/to/disk.img -m 256 vmname && sudo bhyve -c 1 -a -A -m 256 -I -H -g 0 -s 0:0,hostbridge -s 2:0,virtio-blk,/path/to/disk.img -s 1:0,virtio-net,tap0 -S 31,uart,stdio vmname
This will start a VM called vmname which uses 256MB RAM.
To start a VM which uses 4GB or more you’ll have to specify memory settings differently as you need to lead space for PCI MMIO decode below 4GB, so for example, if you wanted to use 8GB RAM, you’d issue
sudo bhyveload -d /path/to/disk.img -m 2048 -M 6144 vmname && sudo bhyve -c 1 -a -A -m 2048 -M 6144 -I -H -g 0 -s 0:0,hostbridge -s 2:0,virtio-blk,/path/to/disk.img -s 1:0,virtio-net,tap0 -S 31,uart,stdio vmname
To shutdown a VM issue
bhyvectl --vm=vmname --destroy
My next step is to now see how to use a ZFS filesystem instead of a file based disk for the VM.
