Category Archives: FreeBSD

FreeBSD on 11″ MacBook Air 5,1 (mid-2012)

This tiny machine has been with me for a few years now, It has mostly run OS X though I have tried OpenBSD on it. Besides the screen resolution I’m still really happy with it, hardware wise. Software wise, not so much. I use an external disk containing a zpool with my data on it. Among this data are several source trees. CVS on a ZFS filesystem on OS X is painfully slow. I dislike that builds running inside Terminal.app are slow at the expense of a responsive UI. The system seems fragile, at the slightest push the machine will either hang or become unresponsive. Buggy serial drivers which do not implement the break signal and cause instability are frustrating.
Last week whilst working on Rump kernel builds I introduced some new build issues in the process of fixing others, I needed to pick up new changes from CVS by updating my copy of the source tree and run builds to test if issues were still present.
I was let down on both counts, it took ages to update source and in the process of cross compiling a NetBSD/evbmips64-el release, the system locked hard. That was it, time to look what was possible elsewhere. While I have been using OS X for many years, I’m not tied to anything exclusive on it, maybe tweetbot, perhaps, but that’s it.
On the BSDnow podcast they’ve been covering changes coming in to TrueOS (formerly PC-BSD – a desktop focused distro based on FreeBSD), their experiments seemed interesting, the project now tracks FreeBSD-CURRENT, they’ve replaced rcng with OpenRC as the init system and it comes with a pre-configured desktop environment, using their own window manager (Lumina). Booting the USB flash image it made it to X11 without any issue. The dock has a widget which states the detected features, no wifi (Broadcom), sound card detected and screen resolution set to 1366×768. I planned to give it a try on the weekend. Friday, I made backups and wiped the system. TrueOS installed without issue, after a short while I had a working desktop, resuming from sleep worked out of the box. I didn’t spend long testing TrueOS, switching out NetBSD-HEAD only to realise that I really need ZFS so while I was testing things out, might as well give stock FreeBSD 11-STABLE a try (TrueOS was based on -CURRENT). Turns out sleep doesn’t work yet but sound does work out of the box and with a few invocations of pkg(8) I had xorg, dwm, firefox, CVS and virtuabox-ose installed from binary packages. VirtualBox seems to cause the system to panic (bug 219276) but I should be able to survive without my virtual machines over the next few days as I settle in. I’m considering ditching VirtualBox and converting the vdi files to raw images so that they can be written to a new zvol for use with bhyve. As my default keyboard layout is Dvorak, OS X set the EFI settings to this layout. The first time I installed FreeBSD 11-STABLE, I opted for full disk encryption but ran into this odd issue where on boot the keyboard layout was Dvorak and password was accepted, the system would boot and as it went to mount the various filesystems it would switch back to QWERTY. I tried entering my password with both layout but wasn’t able to progress any further, no bug report yet as I haven’t ruled myself out as the problem.
Thunderbolt gigabit adapter – bge(4) and DVI adapter both worked on FreeBSD though the gigabit adapter needs to be plugged in at boot to be detected. The trackpad bind to wsp(4), left, right and middle clicks are available through single, double and tripple finger tap. Sound card binds to snd_hda(4) and works out of the box.
For wifi I’m using a urtw(4) Alfa adapter which is a bit on the large side but works very reliably.
A copy of the dmesg is here.

A new addition to FreeBSD.org

Adding Sevan Janiyan as a documentation committer
After many years of tinkering with FreeBSD, I received an invite to join the FreeBSD project earlier last month. When I first started out with FreeBSD (back in v5.0), the handbook was what lead me through the start and made me realise how empowering decent documentation is. My previous experience with $LICENSEPREFIX/$SOMEKERNEL distros had mainly consisted of marathon searches on instruction how to accomplish $thing, finding instructions for another distro which I wasn’t running & going down another rabbit hole from there. I’ll be working with my mentor Benedict Reuschling as a member of the documentation team to continue the maintenance and improvement of the documentation & manual pages in FreeBSD and also cross-polinating necessary changes to the other BSDs in the family, where applicable.

As a starting point, the committers guide instructs a new committer on some preliminary commits to the doc, base and ports repositories to add necessary information such as name / email address, PGP keys and ICBM co-ordinates.

Pretty stoked to reach this mile stone as a part of a journey that started some years back and took me travelling around the world because of work to attending conferences and other events such as the doc sprints at BSDCan.
Now begins the next milestone to make the documentation even greater, again!
to the kernel source code!

FreeBSD Latest News

FreeBSD News Flash

Running FreeBSD / OpenBSD / NetBSD as a virtualised guest on Online.net

I’ve been running a mixture of FreeBSD / OpenBSD & NetBSD as guests on a dedicated server at Online.net. While getting the operating systems installed was fairly seamless, getting networking going was not.

  1. Client are not isolated in a layer 2 domain
  2. DHCPv6 config is broken

Clients not being isolated is not so much a problem itself and is typically what you’d expect if you plugged a bunch of computers into a switch with a single VLAN or unmanaged switched for example; but in a shared environment with untrusted tenants it can cause problems. Broadcast & IPv6 multicast floods aside, one is open to most of the attacks in something like THC-IPv6 due to lack of MLD snooping which would prevent a rogue IPv6 router.

Attacks via IPv6 are not so much of a problem as their use of non-RFC complaint timers settings in their DHCPv6 make it unfeasible to use the offered native IPv6 connectivity as clients will fail to renew leases. Depending on the DHCPv6 client used, the amount of time it takes fail to renew a lease will vary. dhcpcd for example now warns if detects a lease is not compliant with RFC 3315 section 22.4 “Identity Association for Non-temporary Addresses Option”.

Despite having a vast address range in IPv6 and a /48 subnet is allotted free of charge, you’ll need the equal amount of v4 address addresses as the v6 addresses you intend to use at Online.net. There is a way of using a /48 and allocating addresses yourself but it’s only possible using a version of Proxmox which they provide.

You can save yourself a lot of hassle both with configuration & trying to deal with their support  regarding IPv6 by using a Hurricane Electric tunnel. I actually found connectivity was also faster from Hurricane Electric than using the native connectivity.

For IPv4 connectivity on a guest (assuming you’re renting individual IP addresses & not a /27 prefix), you’ll need to use the default gateway IP address assigned to your host alongside the allotted IP address and a /32 prefix.

Assuming the network details are as follows
Default gateway on host: 192.0.2.1
Failover IP #1: 198.51.100.10, assigned to MAC address 00:50:56:00:01:AA
Failover IP #2: 203.0.113.11, assigned to MAC address 00:50:56:00:02:BB
Failover IP #3: 203.0.113.100, assigned to MAC address 00:50:56:00:03:CC

The MAC addresses need to be assigned to the tap(4) interface on the host.
If you’re using bhyve and your guest is using the interface tap0, this would be performed using the -s flag to configure the virtual PCI ethernet card, eg -s 1:0,virtio-net,tap0,mac=00:50:56:00:01:AA

It’s then onto configuring each OS to handle a gateway which is in a another subnet for IPv4 connectivity.

FreeBSD

In FreeBSD you need to construct a route to reach the default IP address first, before you specify the default IP address, otherwise things will not work. So assuming we’re going to use Failover IP #1, your configuration in /etc/rc.conf would be as follows

ifconfig_vtnet0="inet 198.51.100.10/32"
gateway_if="vtnet0"
gateway_ip="192.0.2.1"
static_routes="gateway default"
route_gateway="-host $gateway_ip -interface $gateway_if"
route_default="default $gateway_ip"

Note, the installer at present prevents network installs, you should use a iso image containing the distfiles, bug 206355 has more details.

NetBSD

On NetBSD, configure networking using /etc/netstart.local, entering the commands you’d enter at the console inside the file. Assuming failover IP #2 is going to be used for the NetBSD VM, the following would configure the guest to reach the outside world using 192.0.2.1, as discussed in the NetBSD Network FAQ

ifconfig vioif0 203.0.113.11/32
route add -net 192.0.2.1 -link -cloning -iface vioif0
route add default -ifa 203.0.113.11 192.0.2.1

OpenBSD

On OpenBSD, configure the networking from the ethernet interfaces configuration file hostname.if(5).

Assuming failover IP #3 is going to be used for the OpenBSD VM, the following will setup networking.

/etc/hostname.vio0

inet 203.0.113.100 255.255.255.255 NONE
!/sbin/route add -net 192.0.2.1 -netmask 255.255.255.255 -link -cloning -iface vio0
!/sbin/route add default -ifa 203.0.113.100 192.0.2.1

It’s also possible to not specify the -cloning flag but a patch is required if you’re running 5.9 release.

Issuing secure erase ATA command using camcontrol(8)

The ATA command set has a command to instruct a device to secure erase itself.
Depending on the application & level of sensitivity of the data on disk, it can be a convenient way to decommission a disk or reset an SSD to regain performance. On FreeBSD this can be issued using camcontrol(8).

The command below performs an enhanced erase with a timeout of 60 seconds for the command to be accepted by the disk, this is needed if you get timeout errors when you do not specify it.
camcontrol security ada0 -U user -s Erase -h Erase -T 60

Zvol backed bhyve guest

Things have moved forward in the world of bhyve since I last looked at it over a year ago, support for zvol backed guests where fixed long ago among other things such as the birth of vmrc by Michael Dexter.
To run a guest with a ZFS zvol as its disk is no different to running with a disk image, only thing is that my version of /usr/share/examples/bhyve/vmrun.sh (11.0-CURRENT r267869 at the time of writing) fails to start from the disk once the OS has been installed.

A typical deployment scenario would be

Create a zvol of some size, e.g. 10GB

zfs create -V 10g zroot/guest0

Start a guest which’ll boot from the FreeBSD install CD iso & install onto the zvol

# sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d /dev/zvol/zroot/guest0 -i -I FreeBSD-10.0-RELEASE-amd64-disc1.iso guest0

Use the “ZFS – Automatic Root-on-ZFS” option from the Partitioning menu

As instructed in the bhyve section of the handbook, before rebooting, drop to the shell & edit /etc/ttys & replace the console line with

console "/usr/libexec/getty std.9600" xterm on secure

Shutdown the operating system
halt -p

Kill the guest
/usr/sbin/bhyvectl --destroy --vm=guest0

Create a new guest
bhyveload -m 4G -d /dev/zvol/zroot/guest0 guest0

Boot the new guest from the zvol
bhyve -c 1 -m 4G -A -H -P -s0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,ahci-hd,/dev/zvol/zroot/guest0 -s 31,lpc -l com1,stdio guest0

These instruction skip the creation of networking which is covered in the FreeBSD handbook as linked to above.

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014
root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (3399.54-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a Stepping = 9
Features=0x8f83ab7f
Features2=0xfe9a6257
AMD Features=0x20100800
AMD Features2=0x1
Standard Extended Features=0x200
TSC: P-state invariant
real memory = 5368709120 (5120 MB)
avail memory = 4103143424 (3913 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table:
ioapic0 irqs 0-23 on motherboard
random: initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80cfa5e0, 0) error 19
kbd1 at kbdmux0
acpi0: on motherboard
acpi0: Power Button (fixed)
atrtc0: port 0x70-0x71 irq 8 on acpi0
Event timer "HPET" frequency 10000000 Hz quality 550
Event timer "HPET1" frequency 10000000 Hz quality 450
Event timer "HPET2" frequency 10000000 Hz quality 450
Event timer "HPET3" frequency 10000000 Hz quality 450
Event timer "HPET4" frequency 10000000 Hz quality 450
Event timer "HPET5" frequency 10000000 Hz quality 450
Event timer "HPET6" frequency 10000000 Hz quality 450
Event timer "HPET7" frequency 10000000 Hz quality 450
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: port 0x408-0x40b on acpi0
pcib0: port 0xcf8-0xcff on acpi0
pci0: on pcib0
virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff irq 16 at device 1.0 on pci0
vtnet0: on virtio_pci0
virtio_pci0: host features: 0x1018020
virtio_pci0: negotiated features: 0x18020
vtnet0: Ethernet address: 00:a0:98:7f:5a:41
virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff irq 17 at device 2.0 on pci0
vtblk0: on virtio_pci1
virtio_pci1: host features: 0x10000044
virtio_pci1: negotiated features: 0x10000044
vtblk0: 40960MB (83886080 512 byte sectors)
isab0: at device 31.0 on pci0
isa0: on isab0
uart0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: port 0x2f8-0x2ff irq 3 on acpi0
sc0: at flags 0x100 on isa0
sc0: MDA
vga0: at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
atkbdc0: at port 0x60,0x64 on isa0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
random: unblocking device.
Netvsc initializing... Timecounter "TSC-low" frequency 1699769676 Hz quality 1000
Trying to mount root from zfs:zroot/ROOT/default []...

Giving up on creating a port of OpenNMS

After 5 years of going back & forth, I’ve decided to give up on trying to complete the OpenNMS ports for FreeBSD and dropped maintainership of the Java dependencies in the ports tree (net/jicmp, net/jicmp6, databases/jrrd, databases/iplike)

There are some issues which are show stoppers that need addressing upstream

  1. Separation of configuration & user data from the location of application binaries, Initially I began patching the source to look for files in a different location to the default so that things would integrate with the user land correctly but it soon became apparent that the patching would be a nightmare to maintain on an ongoing basis due to the number of patches required per configuration file. It was clear that things would need to be dealt with at the source rather than patched post release, a long running discussion with developers, bug reports raised, some (minor) patches submitted, 4+ years on, still ignored due to a lack of interest.
  2. Dynamically generated filenames, inherited from Google Web Toolkit, every build attempt generates new filename which make packing impossible.
    Update OpenNMS developer Benjamin Reed points to a possible fix
  3. Unreliable build process, maven fails between 2 to 3 times minimum which would cause lots of false alarms in an automated build environment e.g. the freebsd build cluster.
    This is somewhat of an improvement from a few years back where it would not be possible to build because repositories were not available for a day or two.

As it stands, the port is a shell which make it easy to install OpenNMS on FreeBSD but has major issues when it comes to upgrades or uninstallation. It’s best install dependencies from ports & install OpenNMS manually.

System fails to boot with root on ZFS

I’d installed FreeBSD on my ThinkPad X61s when the head branch of the source tree was at 9-CURRENT, multi-booting it with Windows 7 & OpenBSD.
At the time I was not aware that it was possible to boot FreeBSD from a root file system on a ZFS volume from a disk partitioned using a MBR scheme. Instead, I opted to store /boot on a UFS filesystem.
This install existed for a couple of years, the ThinkPad got a roasting every once in a while to build a new release to install for updates. At some point support for 4K sectors in ZFS was improved, zpool status began to report degraded performance as the disk had been using 512byte sectors where in fact it could support 4K sized sectors.

Eventually, I deleted the existing slices in the FreeBSD partition & attempted to reinstall but found this time the system would not boot.
Booting from the install CD & issuing zpool import reported the new pool & old pool from the previous install.
Destroying a pool before deleting slices stopped this problem from re-occurring but the system still wouldn’t boot from a ZFS volume on a MBR partition.
The next step was to see if things would work if the whole disk was dedicated to FreeBSD, with a GPT partition scheme, things worked but switching to MBR, again, it failed to boot, hanging at a flashing cursor.
Over the next four months, many installs were attempted. On a MBR partitioned disk
FreeBSD failed to boot but PCBSD could by using GRUB.

I stopped trying any further at this point & took a break from it, one thing that had been raised at BSDCan was the possibility it could be lingering metadata, I’d thought as zpool(8) wasn’t reporting any existing pools when asked to import that this wasn’t the case. To give the benefit of a doubt, I ran dd on the disk with no difference in result.
This approach to clearing old pools seemed a little rough so over the weekend I looked into what other options are available.

The zpool(8) man page documents the labelclear option as

zpool labelclear [-f] device

Removes ZFS label information from the specified device. The device
must not be part of an active pool configuration.

-v Treat exported or foreign devices as inactive.

I still had the FreeBSD snapshot from the last attempt which I booted the X61s with, headed to the shell, deleted the existing partitions & issued
zpool labelclear -f /dev/ada0

Everything worked as intended after that.

Thanks to Allan Jude & everyone who chipped in at BSDCan.
Through the trial of getting this working Allan added the option to use a BSD partition scheme to the FreeBSD installer as well as MBR & GPT, which was previously unavailable.

Building CoovaChilli on FreeBSD 10 & newer

I’ve been working on getting CoovaChilli running on FreeBSD 10 the past few weeks. The first problem was that the source code would not build. FreeBSD 10 shipped with clang instead of GCC as the default compiler which is not as forgiving as GCC, clang flagged up lots of issues with the code base such as lack of parameters for functions, one change needed which was a shortcoming of clang itself was the lack of support for nested functions.
The other problem was that CoovaChilli was using a structure that had been marked as deprecated in Net/Free/OpenBSD/Darwin for quite some time & with the release of 10.0, FreeBSD removed this structure from their codebase. This resulted in the tun(4) interface being initialised but no IP address being assigned to it, which was interesting, dhcp responses were still going out but obviously nothing could talk back to the IP address that coova was configured to be listening on (uamlisten).

All BSDs & derivatives were separated out from Linux in dev_set_address(), Linux was set to use the pre-existing method & the BSD & derivatives were switch to use the “new” ifaliasreq structure.

With these change, it’s possible for a client connected to a router running CoovaChilli to visit sites listed in uamallowed. These are sites which CoovaChilli allows to be browsed without needing authentication through the captive portal first. Next stage is to get the captive portal running along with a RADIUS server so that the previous revision of the setup guide can be updated. With the release of FreeRADIUS 3 configuration has changed somewhat, hence the old documentation for version 2 doesn’t necessarily apply.

These changes so far only allow the code to build without any features enabled, for example, enabling OpenSSL support yielded more errors which have not been fixed yet & I suspect others will show up as more options are switched on.

The work so far can be found here

Merged 3/6/2014

Dvorak layout with ctrl & caps lock swapped round on FreeBSD

Since getting a MacBook Air with a Japanese keyboard, I have grown accustomed to having the control key in place of the caps lock, so much so that now it’s annoying not having it there if I switch to another system. Searching around to see what I’d find on the subject, it seems that a similar frustration exists for users who are used to the left control key being in the bottom left hand side instead of the new Fn key & infact I have a bios image which makes this change for my X61s.

From the brief search for a bios image brought up lots of links to the image which I already have for swapping fn with ctrl & I didn’t fancy an introduction to ThinkPad BIOS hacking with IDA so I began to look for OS specific solutions.

On FreeBSD, a variant of the UK (uk.cp850-ctrl.kbd, uk.iso-ctrl.kdb) & US standard layouts (us.pc-ctrl.kbd) exist which swap the location of the two keys round & I found this guide (in Japenese) which discusses what changes are needed to a layout file (reassigning scan codes 29 & 58) but all this was already taken care of some 12 years ago. The header of us.dvorak.kbd menions:

“There are some minor variations, but this seems like the most common layout. I personally use one with three more pairs swapped:
esc `~, clock lctrl, and =+ \| (supplied as “us.dvorakx.kbd“). ”

I just needed to declare keymap="us.dvorakx.kbd" in /etc/rc.conf & restart /etc/rc.d/syscons.

bhyve – BSD Hypervisor

With the videos released last month from euroBSDcon 2012, I watched Michael Dexter’s talk on bhyve, the BSD hypervisor has come along way since I last tried it over a year ago & Michael has helped a with it’s progress by writing articles on CFT & scripts for running bhyve.
Last week I decided to get myself a server which I could use to do builds quickly & to run virtual machines for testing. Hetzner do high spec consumer hardware as servers,  €59 per month get you a i7 with 32GB of RAM & 2x 3TB HDD, I ordered the server along with a 16GB USB flash drive with the plan of running SmartOS, once my login details for the server came through, I raised a support ticket for access to a IP KVM, within the hour I was given access & the installation went seamlessly. SmartOS was running on my server & it all went down hill from there.
As there is a IPv4 address shortage, hetzner charge a premium for additional addresses as a routed subnet, along with an additional fee for having the ability to request additional addressses as a “flexi pack”, a /27 would cost €47, I was not going to pay this so decided to go IPv6 only as I have connectivity at home & work. Unfortunately, though IPv6 support is there in the core of SmartOS by interitence from OpenSolaris, the additions from Joyent for KVM don’t, main culprit being vmadm(1m), after losing two days trying to get things working I came to the conclusion that A) it would be a big pain to maintain going forward as the burden would be on me to work around the shortfalls of the system B) I didn’t want to maintain my own release with third party patches which were not in yet C) I didn’t like the way I would have to extend the system to add functionality eg to set the hostname for your system persistently you have to use a script D) getting IPv6 support to guests was painful.

The majority of the work I’m doing is oriented around FreeBSD, it takes over 4 hours to do a build world & kernel on my ThinkPad X61s with a 1.6GHz Core2Duo so anything that can prolong it’s life & give me new builds quickly is good. I placed another support request for IP KVM (LARA in the world of hetzner) & once I had the login details I netbooted the server to  their FreeBSD rescue environment which is a FreeBSD 8.3 based copy of mfsBSD. From there I fetched the latest FreeBSD-CURRENT usb image & wrote it to the flash drive using dd(1) & went about setting up a mirrored zpool to install FreeBSD onto.

Once the installation was complete & the system was up & running I revisited Michael’s talk, slides & scripts.
His scripts are numbered sequentially so you can easily go from creating a disk image to running & managing your virtual machines. This article covers a summary of what is involved to get a guest VM ready with FreeBSD-CURRENT built from source which are taken from his scripts & slides. As development has progressed since the talk, some things which are performed are no longer required. Essentially, you can boot a stock system from a disk image with only 2 necessary modifications to stock configuration files for dealing with the console.
There is also a vmrun.sh script which simplifies the whole process to try out (see instructions)

First build world & kernel (not necessary, you can use the precompiled binary instead if you choose)

On the host add the following to /boot/loader.conf
vmm_load="YES"
if_tap_load="YES"
bridgestp_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"

Create a file which will be used as your disk, eg a 80GB one
truncate -s 80G disk.img
Create a md(4) disk with the file you just created
mdconfig disk.img
Initialise the disk to use the entire disk as a freebsd slice
fdisk -BI md0

You’ll receive the following error which can be safely ignored
******* Working on device /dev/md0 *******
fdisk: invalid fdisk partition table found

Write a standard label & boot code to slice 1
bsdlabel -wB /dev/md0s1
Write a filesystem to slice 1a
newfs -U /dev/md0s1a
Mount it to /mnt
mount /dev/md0s1a /mnt

From /usr/src, install world, kernel & distribution (contents of /etc) onto the disk image
make installworld DESTDIR=/mnt
make installkernel DESTDIR=/mnt
make distribution DESTDIR=/mnt

Setup your fstab to mount root from /dev/vtbd0s1a
echo "/dev/vtbd0s1a / ufs rw 1 1" > /mnt/etc/fstab
Configure your console
echo 'console "/usr/libexec/getty std.9600" vt100 on secure' > /mnt/etc/ttys
echo 'console="userboot"' > /mnt/boot/loader.conf

Aside from configuring /etc/rc.conf the instructions above cover the bare minimum to get a booting VM.

From Michael’s 2-install-guest.sh I’ve skipped loading the virtio drivers in /boot/loader.conf as they’re loaded by default in FreeBSD-CURRENT & the following though I’ve not given it more testing
Helps Kernel detected it’s running in a virtualised environment
smbios.bios.vendor="Bochs"
Avoid clock drift
kern.timecounter.hardware="TSC"
kern.timecounter.invariant_tsc="1"

PCI pass-through support as it caused hangs
hw.pci.enable_msix="0"
hw.pci.honor_msi_blacklist="0"

Unmount the file system
umount /mnt
Detach the file from md(4)
mdconfig -d -u 0
Assuming you’re using md0
You can get a list of configured devices with
mdconfig -l

As covered in 3-host-prep.sh you can load the required kernel modules for bhyve & guest networking by running
kldload vmm
kldload if_tap
kldload bridgestp
kldload if_bridge
or rebooting 🙂

Before starting your VM, you need to create the needed interfaces, a tap(4) interfaces with a bridge(4) linked to the interface you want the VM to be able to communicate on, in my case a re(4)
ifconfig tap0 create up
ifconfig bridge0 create up
ifconfig bridge0 addm tap0 addm re0 up

Because of STP, once you have started the virtual machine, you should pause at the boot menu by pressing space & waiting 20 seconds until STP has stabilised otherwise you may find strange issues with you guest not being able to communicate properly.
If you restart a VM, it is also important to destroy the tap & bridge interfaces before starting up again or you will again experience odd behaviour e.g I was seeing traffic come in to the VM but not going out.
ifconfig tap0 destroy
ifconfig bridge0

To start a VM with less than 4GB RAM issue
sudo bhyveload -d /path/to/disk.img -m 256 vmname && sudo bhyve -c 1 -a -A -m 256 -I -H -g 0 -s 0:0,hostbridge -s 2:0,virtio-blk,/path/to/disk.img -s 1:0,virtio-net,tap0 -S 31,uart,stdio vmname
This will start a VM called vmname which uses 256MB RAM.

To start a VM which uses 4GB or more you’ll have to specify memory settings differently as you need to lead space for PCI MMIO decode below 4GB, so for example, if you wanted to use 8GB RAM, you’d issue
sudo bhyveload -d /path/to/disk.img -m 2048 -M 6144 vmname && sudo bhyve -c 1 -a -A -m 2048 -M 6144 -I -H -g 0 -s 0:0,hostbridge -s 2:0,virtio-blk,/path/to/disk.img -s 1:0,virtio-net,tap0 -S 31,uart,stdio vmname

To shutdown a VM issue
bhyvectl --vm=vmname --destroy

My next step is to now see how to use a ZFS filesystem instead of a file based disk for the VM.

FreeBSD, 10 years on

I write this article a week after my 10th anniversary as a FreeBSD user.
I had heard of FreeBSD previously but had never tried it. The closest I had come to a flavour of BSD was unsuccessful attempts at downloading NetBSD on various modems ranging from 14.4k to 33.6 to install onto a Sun 3/60 in the late 90’s.
In the summer of 2002 I managed to obtain a DEC Alpha which I initially ran NT4 on & Redhat 7.2.
I performed a full install with Gnome & watched as the system crawled as it started X11, over the next couple of days It became more & more apparent that the system couldn’t handle it.
I was reading slashdot one night & saw FreeBSD 5.0 had just been announced & the Alpha was a supported platform so I decided to give it a try & downloaded an iso.
Installation went ok, I can’t remember if I had to restart the process because I’d said yes to test the X configuration in sysinstall or not but I do remember that managed to set my syscons font to swiss.
My background was DOS & Windows with several failed attempts at becoming a Linux users, I had some basic knowledge of the *nix user land but more dangerous than anything. Relying on search engines to find answers which in the case of Linux were either incorrect, outdated or didn’t apply to the distro I happen to be running at the time.
It quickly became apparent that this was not a problem on FreeBSD, everything pointed back to the handbook. Using the handbook with some pointers from IRC I made a lot of progress, far more than I had ever made with Linux, I was able to get GDM running, a BSD theme installed & switch window managers. The system also performed really fast, there was a clear noticeable difference between FreeBSD 5.0 & Redhat 7.2.
Using ports I was able to compile software with little effort & the clear divide of user land between base installed & user installed made it easy to track things down.
I ran the 5.0 release for a couple of weeks & was very happy with the progress I had made with configuring the system but I did run into lots of issues which I was told were bugs in FreeBSD 5.0 & it’s not really production ready so I re-installed 4.7 & stuck with the RELENG_4 branch until 4.11.
I was in love with FreeBSD, it was un-intrusive, well organised, well documented & empowering.
By the time version 5.3 was released I was hosting my first customers websites & email with it & have continued to do so for myself & other customers on many occasions since. 10 years on I am now working with many servers running FreeBSD around the world & I’m as happy with it as the first day that I installed it.

swap_pager: indefinite wait buffer:

I have a virtualbox VM of FreeBSD-CURRENT running on my work laptop which I’m using for testing & development. To bring the system up to date I started buildworld after updating src, going back to check on the build process I found my SSH session had hung and the VM console had starting showing swap_pager: indefinite wait buffer: followed by values for objects in buffer, block number & size.
A search on google brought up the following answer from UNIXguide

This means that a process is trying to page memory to disk, and the page attempt has hung trying to access the disk for more than 20 seconds. It might be caused by bad blocks on the disk drive, disk wiring, cables, or any other disk I/O-related hardware. If the drive itself is actually bad, you will also see disk errors in /var/log/messages and in the output of dmesg. Otherwise, check your cables and connections.

Increasing the amount of RAM allocated to a VM seems to resolves the issue without having to resort to checking virtual cables or connections.

Update 4/1/2013

It seems that I had forgotten to define MALLOC_PRODUCTION in /etc/make.conf as this problem was also raised on the FreeBSD/ARM mailing list

Juniper SRX & FreeBSD/mips

I didn’t realise the Juniper SRX line (at least the 100) was based on a MIPS SoC made by OCTEON.

CPU in a SRX100b
OCTEON CN5020-SCP pass 1.1, Core clock: 500 MHz, DDR clock: 266MHz (532 Mhz data rate)

dmesg from SRX100

Thinking about it now, I now understand why Juniper contributed the code back up to FreeBSD back in 2007 & as I search around for reference material to link to in this blog post the pieces are falling into place.
An announcement was made at the start of month that DTrace had been ported to FreeBSD/MIPS by Oleksandr Tymoshenko.
What this will mean is that when the code makes it back into a Junos release you will have the ability to get near realtime answers of what is going on your router or firewall for example using the network provider & it’ll be safe to run in production because DTrace is designed not to be harmful, something which Cisco doesn’t do & use of debug commands is discouraged on production systems because they are considered harmful.

If you’ve never played with DTrace & have a Mac, its available on all system running Leopard & above, see this article on getting started
Its available in Solaris (& derivatives) which is also where it originates from & on FreeBSD but system has to be rebuilt to enable support, see the wiki article for details.

Building the MSP430 openchronos firmware on FreeBSD

There are two openchronos projects, there’s the original OpenChronos project & the continuation openchronos project.
The openchronos code has a few modifications which are not upstream in poelzi’s OpenChronos repo, most importantly the changes to build under mspgcc 4, I was unable to build under mspgcc 3 as support for some versions of the MSP430 were missing, this may just be an issue specific to version currently in FreeBSD ports tree however.
To build the openchronos firmware on FreeBSD you’ll need the following ports installed:
devel/git
devel/msp430-libc
lang/python/

At the config stage of msp430-libc leave the “Use new msp430-gcc4 compiler” option left on & build.
Once everything is installed clone the repo listed on the openchronos website with git.
The config process for openchronos uses python & depends on the locale to be defined correctly, otherwise running gmake config on the shell will cause an error such as:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2503' in position 20: ordinal not in range(128)2
Defining LC_CTYPE with the appropriate UTF-8 encoding for your locale resolvers this, run locale -a for a list of supported types which you can declare.
Once that’s defined, running gmake config should show the configuration script, if you’re still receiving errors you may want to run gmake clean & try again.
You need to check the frequency setting is correct depending on the model of watch you bought.
Now save your configuration & run gmake to compile the code.
If you’re unable to compile the image successfully as the image generated is too large (see the problems section of README) either set “Metric only code” option in configure or try this patch which reduces the size of the image (Thanks to Andrey Ulanov for the pointer).
If build completes successfully, you’ll have two files in the build directory named eZChronos.elf & eZChronos.txt.
At this point I cheated & used Windows to flash the watch wirelessly.
Set the watch in rFbSL mode & run the Chronos data logger app, go to the wireless update tab, point it to the txt files & press “Update Watch”
A counter should show up to display the progress on the watch.
Once the flash is complete, all the elements on the LCD display should switch on

Building & administering jails on FreeBSD, Part 1

22/05/2014
These instructions are now part of the FreeBSD handbook since docs/189901 was committed. Please refer to the instructions in the handbook.

The FreeBSD jail(8) manpage & Chapter 15 of the FreeBSD handbook do a great job of explaining jails & helping you get on your way with creating jails, this post builds on that information, covering alternative methods for getting your jails installed & adding what’s not covered already such as maintenance of jails (patching to be specific) & version upgrades.

  • Part 1 (this post :)) will cover alternative install methods & jail maintenance
  • Part 2 (not yet published) will cover upgrading to a new version FreeBSD

Once completed the information from these posts will be submitted for inclusion in the handbook.

So lets begin, when creating a “complete” jail you have two options for the source of the userland, compile from source code or use the prebuilt binaries from install media, both the jail manpage & handbook cover building from source code, we wont go over it again here.

One thing worth mentioning though is if you want to build from source code, create a src.conf file & disable items which are not required, this should speed up the time required to build world & reduce the amount of disk space used by jails.

Here are two sample src.conf files, which disable building items such as firewalls (no use unless you’re using vimage), acpi or documentation:
Sample src.conf #1
Sample src.conf #2

To install the userland from installation media
first create the root directory for the jail, eg
mkdir -p /usr/jails/mynewjail
set the $DESTDIR variable to this location
if using sh
export DESTDIR=/usr/jails/mynewjail
if using csh/tcsh
setenv DESTDIR /usr/jails/mynewjail
mount the media (using the 8.0-RELEASE cd 1 iso in this example)
mount -t cd9660 /dev/`mdconfig -f /some/path/to/8.0-RELEASE-i386-disc1.iso` /mnt

Extract the binaries from the tar balls on the install media into your declared destination, realistically, you’ll only need to extract base, but you can do a complete install if you wish to.
To install just base:
cd /mnt/8.0-RELEASE/base; ./install.sh

You are about to extract the base distribution into /usr/jails/mynewjail – are you SURE
you want to do this over your installed system (y/n)?

To install everything but kernel:
if using sh
cd /mnt/8.0-RELEASE; for dir in base catpages dict doc games info manpages ports; do (cd $dir ; ./install.sh) ; done
if using csh/tcsh
foreach dir ( base catpages dict doc games info manpages ports )

cd /mnt/8.0-RELEASE/$dir; ./install.sh

end

All configuration steps from here on to get up and running are as specified in the jail man page & handbook.

Keeping jails up to date with patches
On a host with default settings the freebsd-update(8) tool doesn’t work as
chflags(1) is not permitted in a jail, set sysctl security.jail.chflags_allowed to 1 to allow it & freebsd-update can be used.
The other option is to patch the userland manually from the host OS. All the needs to be done is the $DESTDIR has to be passed to the make install command eg.
In section 2b of the FreeBSD-SA-10:04.jail advisory you’re told to
# make obj && make depend && make && make install
after patching, instead you would issue
# make obj && make depend && make && make install DESTDIR=/usr/jails/mynewjail

22/05/2014

Use the -b flag for freebsd-update from the host to update jails instead of taking drastic measures.

OpenNMS-dev port for FreeBSD

10/6/14 – No longer maintained

I’ve created a new FreeBSD port for installing releases from the unstable branch of OpenNMS.
This port suffers from the same issue as the stable port

You can grab the port here

9/6/10
Initial port, installs version 1.7.92

6/11/10
Update to version 1.9.2

25/4/11
I’ve setup a temporary mercurial repository with all version of the port in the repo to make moving forward easier (I say the repo is temporary as I intend to host my own instance of mercurial & to push out to git & bitbucket as well).

26/4/11
Update to version 1.9.7

17/5/11
Update to version 1.9.8
With this release, OpenNMS switched to the new JNA Pinger The JNA Pinger assumes IPv6 is enabled by default & if not doesn’t fail gracefully, this will cause problems if you’re running OpenNMS in a jail from example & you’ve not assigned the jail an IPv6 address, you can keep with the progress of this issue in NMS-4673
PR’s have been raised to update JICMP, JRRD & iplike to the latest versions in ports, see PR #’s 156785 156786 157120

11/08/11
Update to version 1.9.90

17/11/11
Update to version 1.9.93

IPlike port for FreeBSD

As part of getting OpenNMS on FreeBSD via ports I’ve created a port for the IPLIKE which is a C implementation of the iplike stored procedure that’s used by OpenNMS.
You can download a copy of the port here

If I haven’t heard any bad reports by the end of the week, I will raise a PR to have it added to ports.

ports/142581 was commited earlier today

iplike commit message on freshports.org

The port can be found at databases/iplike, please update your ports

OpenNMS port for FreeBSD

10/6/14 – No longer maintained

The port is for the current stable version, v1.6.2. It is in its very early stages, there are still some issues which need to be ironed out:

* The port will install just fine except that it complains about some files listed in the pkg-plist which are not there, well they are there but the files named are dynamically generated everytime a build is attempted (jetty-webapps & webapps cache files) so this will need to be fixed.

* As there are issues with these filenames in the pkg-plist, make package fails.

* A problems with the jicmp dependency, it fails to detect that jicmp is installed & attempts to build & install it no-matter what & obviously fails if it is.

All previous issues with the port listed above have been resolved, the port now just needs to be tested before submission for inclusion in ports.

You can grab the port here

Moved progress status to a separate text file